[MAINTENANCE] Remove some references to block style datasource #9868
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for niobium-lead-7998 canceled.
|
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## develop #9868 +/- ##
===========================================
- Coverage 78.25% 78.07% -0.18%
===========================================
Files 484 484
Lines 42394 42390 -4
===========================================
- Hits 33174 33096 -78
- Misses 9220 9294 +74
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
tyler-hoffman
changed the title
tyler-hoffman
changed the title
This reverts commit 323faa4.
tyler-hoffman
commented
May 3, 2024
| @@ -913,28 +913,6 @@ def build_sa_validator_with_data( # noqa: C901, PLR0912, PLR0913, PLR0915 | |||
| if context is None: | |||
| context = build_in_memory_runtime_context() | |||
|
|
|||
| assert context is not None, 'Instance of any child of "AbstractDataContext" class is required.' | |||
There was a problem hiding this comment.
why was this stuff here? IDK, but things pass without it 🤷
This reverts commit 9d651d1.
…block style datasources
tyler-hoffman
commented
May 3, 2024
| @@ -226,246 +218,6 @@ def test_update_project_config( | |||
| assert context.progress_bars["globally"] is True | |||
|
|
|||
|
|
|||
| @pytest.mark.unit | |||
| def test_add_datasource_with_existing_datasource( | |||
There was a problem hiding this comment.
we're going to remove datasource mgmt from context, so no use in reworking these tests.
| }, | ||
| }, | ||
| ) | ||
| # Updating "execution_engine" to insure peculiarities, incorporated herein, propagate to "ExecutionEngine" itself. # noqa: E501 |
Comment on lines
-99
to
-217
|
|
||
| @pytest.mark.unit | ||
| def test_get_datasource_cache_miss(in_memory_runtime_context) -> None: | ||
| """ | ||
| What does this test and why? | ||
|
|
||
| For all non-Cloud contexts, we should leverage the underlying store in the case | ||
| of a cache miss. | ||
| """ | ||
| context = in_memory_runtime_context | ||
|
|
||
| name = "my_fake_datasource_name" | ||
|
|
||
| # Initial GET will miss the cache, necessitating store retrieval | ||
| with mock.patch( | ||
| "great_expectations.datasource.datasource_dict.DatasourceDict.__getitem__" | ||
| ) as mock_get: | ||
| context.get_datasource(name) | ||
|
|
||
| assert mock_get.called | ||
|
|
||
| # Subsequent GET will retrieve from the cache | ||
| with mock.patch("great_expectations.data_context.store.DatasourceStore.get") as mock_get: | ||
| context.get_datasource(name) | ||
|
|
||
| assert not mock_get.called | ||
|
|
||
|
|
||
| @pytest.mark.unit | ||
| def test_BaseDataContext_add_datasource_updates_cache( | ||
| in_memory_runtime_context: EphemeralDataContext, | ||
| pandas_enabled_datasource_config: dict, | ||
| ) -> None: | ||
| """ | ||
| What does this test and why? | ||
|
|
||
| For persistence-disabled contexts, we should only update the cache upon adding a | ||
| datasource. | ||
| """ | ||
| context = in_memory_runtime_context | ||
|
|
||
| name = pandas_enabled_datasource_config["name"] | ||
|
|
||
| assert name not in context.datasources | ||
|
|
||
| context.add_datasource(**pandas_enabled_datasource_config) | ||
|
|
||
| assert name in context.datasources | ||
|
|
||
|
|
||
| @pytest.mark.unit | ||
| def test_BaseDataContext_update_datasource_updates_existing_data_source( | ||
| in_memory_runtime_context: EphemeralDataContext, | ||
| pandas_enabled_datasource_config: dict, | ||
| ) -> None: | ||
| """ | ||
| What does this test and why? | ||
|
|
||
| Updating a Data Source should update a Data Source | ||
| """ | ||
| context = in_memory_runtime_context | ||
|
|
||
| name = context.list_datasources()[0]["name"] | ||
| pandas_enabled_datasource_config["name"] = name | ||
| data_connectors = pandas_enabled_datasource_config["data_connectors"] | ||
| pandas_enabled_datasource_config.pop("class_name") | ||
| datasource = Datasource(**pandas_enabled_datasource_config) | ||
|
|
||
| assert name in context.datasources | ||
| cached_datasource = context.datasources[name] | ||
| assert cached_datasource.data_connectors.keys() != data_connectors.keys() # type: ignore[union-attr] | ||
|
|
||
| context.update_datasource(datasource) | ||
|
|
||
| retrieved_datasource = context.get_datasource(datasource_name=name) | ||
| assert retrieved_datasource.data_connectors.keys() == data_connectors.keys() # type: ignore[union-attr] | ||
|
|
||
|
|
||
| @pytest.mark.unit | ||
| def test_BaseDataContext_update_datasource_fails_when_datsource_does_not_exist( | ||
| in_memory_runtime_context: EphemeralDataContext, | ||
| pandas_enabled_datasource_config: dict, | ||
| ) -> None: | ||
| """ | ||
| What does this test and why? | ||
|
|
||
| Updating a data source that does not exist should create a new data source. | ||
| """ | ||
| context = in_memory_runtime_context | ||
|
|
||
| name = pandas_enabled_datasource_config["name"] | ||
| pandas_enabled_datasource_config.pop("class_name") | ||
| datasource = Datasource(**pandas_enabled_datasource_config) | ||
|
|
||
| assert name not in context.datasources | ||
|
|
||
| with pytest.raises(DatasourceNotFoundError): | ||
| context.update_datasource(datasource) | ||
|
|
||
|
|
There was a problem hiding this comment.
just confirming that these are being removed because we don't support these methods on the data context.
There was a problem hiding this comment.
Yup, I think there were 2 files that tested all this crud on context, but since we'll remove these methods in https://greatexpectations.atlassian.net/browse/V1-321, I don't think there's sense in taking the time to port them over to FDS
| except AssertionError as e: | ||
| result = e | ||
| result = Validator( | ||
| execution_engine=PandasExecutionEngine(), |
There was a problem hiding this comment.
interesting that this works. it's preferable to accessing a private attr on the datasource, but makes me wonder what our ideal pattern would be.
Comment on lines
+715
to
+717
| # TODO: Convert this to actually mock an exception being thrown | ||
| # graph = ValidationGraph(execution_engine=execution_engine) | ||
| # graph.build_metric_dependency_graph = mock_error # type: ignore[method-assign] |
There was a problem hiding this comment.
is this going to be follow up work? is it scoped somewhere?
There was a problem hiding this comment.
Thanks for keeping me honest! https://greatexpectations.atlassian.net/browse/V1-322
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Removes all references to block style Datasource except for in
new_datasource.py,abstract_data_context.py, and an__init__.pyfile. There are likely a near infinite number of tests that depend on the existence of the class, though.invoke lint(usesruff format+ruff check)For more information about contributing, see Contribute.
After you submit your PR, keep the page open and monitor the statuses of the various checks made by our continuous integration process at the bottom of the page. Please fix any issues that come up and reach out on Slack if you need help. Thanks for contributing!