[FEATURE] Add `DataContext.update_datasource` CRUD method #5417

cdkini · 2022-06-30T18:47:27Z

Changes proposed in this pull request:

New update_datasource method to enable updating of existing datasources from notebooks
Updated tests for DatasourceStore

Definition of Done

Please delete options that are not relevant.

My code follows the Great Expectations style guide
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have added unit tests where applicable and made sure that new and existing tests are passing.
I have run any local integration tests and made sure that nothing is broken.

Thank you for submitting!

netlify · 2022-06-30T18:47:32Z

✅ Deploy Preview for niobium-lead-7998 ready!

Name	Link
🔨 Latest commit	`b0c9e4b`
🔍 Latest deploy log	https://app.netlify.com/sites/niobium-lead-7998/deploys/62bf3387508fee000835b1c0
😎 Deploy Preview	https://deploy-preview-5417--niobium-lead-7998.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

ghost · 2022-06-30T18:48:40Z

👇 Click on the image for a new way to code review

Make big changes easier — review code in small groups of related files
Know where to start — see the whole change at a glance
Take a code tour — explore the change with an interactive tour
Make comments and review — all fully sync’ed with github

Try it now!

Legend

donaldheppner · 2022-06-30T19:34:38Z

great_expectations/data_context/data_context/base_data_context.py

@@ -1913,6 +1913,22 @@ def _instantiate_datasource_from_config(
            )
        return datasource

+    def update_datasource(


It feels like there is more to do here, and that maybe we don't want to pass a config object here but a datasource one. If someone grabs a context with a datasource, makes changes, how do they save it? Other methods work to maintain the cached datasource collection, is that same effort needed here?

I agree! I've modified the signature to take in an actual datasource.

Additionally, I've made it such that BaseDataContext doesn't persist by default and DataContext does. The cache has also been updated.

Let me know if this is closer to what you're envisioning.

billdirks

LGTM!

billdirks · 2022-06-30T19:30:40Z

great_expectations/data_context/data_context/base_data_context.py

+        Args:
+            datasource_name: The name of the Datasource to update.
+            datasource_config: The config object to persist using the DatasourceStore.
+        """


Do you want to add a Raises section here or do you not want to bake that into the API?

That's a good question! I'm not sure what is a best practice here. I've leaned towards only adding Raises if an exception is raised within the top level scope of the function body.

Any preference?

There is a small comment about this made in the Google python style guide:
https://google.github.io/styleguide/pyguide.html#doc-function-raises

My reading of it is If you want to codify that raising this exception is part of this functions API and callers should be handling it then document it, otherwise don't. I'm not sure where/how this is planning on being used.

billdirks · 2022-06-30T19:45:35Z

tests/data_context/store/test_datasource_store.py

+        datasource_name=datasource_name, datasource_config=updated_datasource_config
+    )
+
+    key: DataContextVariableKey = DataContextVariableKey(


My preference would be to remove the annotations on key and actual_config since you instantiate/cast it in the same line and mypy will infer this.

I agree! We've been leaning towards extra verbosity but this is quite obvious when we see the constructor call in the same line. Updating now.

billdirks · 2022-06-30T19:49:31Z

tests/data_context/store/test_datasource_store.py

+        resource_name=datasource_name,
+    )
+    actual_config: DatasourceConfig = cast(
+        DatasourceConfig, datasource_store_with_single_datasource.get(key=key)


Just looked and the backing store for this config is a dict and not typed 😢

Ugh yeah our typing needs some work - lots of bags of data that could use structure. We've been incrementally adding dataclasses to provide some of that type safety but it's a work in progress.

…_expectations into feature/great-888/great-1037/DataContext.update_datasource

billdirks · 2022-07-05T14:49:45Z

great_expectations/data_context/data_context/base_data_context.py

+        Args:
+            datasource_name: The name of the Datasource to update.
+            datasource_config: The config object to persist using the DatasourceStore.
+        """


There is a small comment about this made in the Google python style guide:
https://google.github.io/styleguide/pyguide.html#doc-function-raises

My reading of it is If you want to codify that raising this exception is part of this functions API and callers should be handling it then document it, otherwise don't. I'm not sure where/how this is planning on being used.

billdirks · 2022-07-05T14:54:28Z

great_expectations/data_context/data_context/base_data_context.py

+    def update_datasource(
+        self,
+        datasource_name: str,
+        datasource: Union[LegacyDatasource, BaseDatasource],


datasource has a name property so passing datasource_name now seems redundant.

billdirks · 2022-07-05T14:57:24Z

great_expectations/data_context/data_context/data_context.py

+    def update_datasource(
+        self,
+        datasource_name: str,
+        datasource: Union[LegacyDatasource, BaseDatasource],


Similar comment here about datasource_name.

feat: add new CRUD method

b09eeb1

github-actions bot added the core-team label Jun 30, 2022

cdkini self-assigned this Jun 30, 2022

chore: update docstrings

a9b43cd

cdkini requested review from anthonyburdi, donaldheppner, Shinnnyshinshin, alexsherstinsky, billdirks and NathanFarmer June 30, 2022 19:16

donaldheppner requested changes Jun 30, 2022

View reviewed changes

billdirks approved these changes Jun 30, 2022

View reviewed changes

cdkini added 7 commits July 1, 2022 12:01

Merge branch 'develop' of https://github.com/great-expectations/great…

722d693

…_expectations into feature/great-888/great-1037/DataContext.update_datasource

feat: differentiate BaseDataContext and DataContext impls

0ad82e3

chore: have update take a core obj instead of a config

4a0af3f

chore: remove unnecessary type hints

c88f15a

chore: update linter thresholds

df4b641

Merge branch 'develop' of https://github.com/great-expectations/great…

229766d

…_expectations into feature/great-888/great-1037/DataContext.update_datasource

chore: remove unused imports

b0c9e4b

cdkini requested review from donaldheppner and billdirks July 1, 2022 17:49

donaldheppner approved these changes Jul 1, 2022

View reviewed changes

cdkini enabled auto-merge (squash) July 1, 2022 18:04

cdkini merged commit 5f80232 into develop Jul 1, 2022

cdkini deleted the feature/great-888/great-1037/DataContext.update_datasource branch July 1, 2022 19:01

billdirks reviewed Jul 5, 2022

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FEATURE] Add `DataContext.update_datasource` CRUD method #5417

[FEATURE] Add `DataContext.update_datasource` CRUD method #5417

cdkini commented Jun 30, 2022 •

edited

Loading

netlify bot commented Jun 30, 2022 •

edited

Loading

ghost commented Jun 30, 2022 •

edited by ghost

Loading

donaldheppner Jun 30, 2022

cdkini Jul 1, 2022

billdirks left a comment

billdirks Jun 30, 2022

cdkini Jul 1, 2022

billdirks Jul 5, 2022

billdirks Jun 30, 2022

cdkini Jul 1, 2022

billdirks Jun 30, 2022

cdkini Jul 1, 2022

billdirks Jul 5, 2022

billdirks Jul 5, 2022

billdirks Jul 5, 2022

[FEATURE] Add DataContext.update_datasource CRUD method #5417

[FEATURE] Add DataContext.update_datasource CRUD method #5417

Conversation

cdkini commented Jun 30, 2022 • edited Loading

Definition of Done

netlify bot commented Jun 30, 2022 • edited Loading

✅ Deploy Preview for niobium-lead-7998 ready!

ghost commented Jun 30, 2022 • edited by ghost Loading

Legend

Choose a reason for hiding this comment

Choose a reason for hiding this comment

billdirks left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

[FEATURE] Add `DataContext.update_datasource` CRUD method #5417

[FEATURE] Add `DataContext.update_datasource` CRUD method #5417

cdkini commented Jun 30, 2022 •

edited

Loading

netlify bot commented Jun 30, 2022 •

edited

Loading

ghost commented Jun 30, 2022 •

edited by ghost

Loading