[FEATURE] Add support to AWS Glue Data Catalog #5123

lccasagrande · 2022-05-15T05:45:21Z

Changes proposed in this pull request:

Add ConfiguredAssetGlueCatalogDataConnector to explicitly define database and tables from AWS Glue Data Catalog
Add InferredAssetGlueCatalogDataConnector to implicitly list database and tables from AWS Glue Data Catalog
Upg moto pkg to mock glue.get_databases() function
Upd SparkDFExecutionEngine to read data from the catalog (hive or glue), using the new GlueCatalogBatchSpec class
Add a new BatchSpec for data assets from GlueCatalogDataConnectors
Upd BaseDataContext to include GlueCatalogDataConnectors
Upd DataConnectorConfigSchema to include GlueCatalogDataConnectors fields

Previous Design Review notes:

closes AWS Glue Catalog Data Connector Support #4945

Definition of Done

My code follows the Great Expectations style guide
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have added unit tests where applicable and made sure that new and existing tests are passing.

netlify · 2022-05-15T05:45:23Z

👷 Deploy request for niobium-lead-7998 pending review.

Visit the deploys page to approve it

Name	Link
🔨 Latest commit	`c7e9b44`

rdodev · 2022-05-25T19:35:18Z

Howdy @lccasagrande thanks a bunch for the contrib. I'll take a look and leave comments as necessary.

lccasagrande · 2022-06-21T19:08:53Z

@rdodev is there anything more I can do in this PR? looking forward to be able to use it.

rdodev · 2022-06-21T19:18:50Z

@rdodev is there anything more I can do in this PR? looking forward to be able to use it.

So very sorry @lccasagrande thanks for re-raising this PR. I will ask engineering to take a look.

great_expectations/datasource/data_connector/configured_asset_glue_catalog_data_connector.py

great_expectations/datasource/data_connector/inferred_asset_glue_catalog_data_connector.py

alexsherstinsky

There are many failing tests; I would like to re-review this interesting contribution after the working branch is updated to the latest and the failing tests pass. Additional questions:

Is this focused mainly on connecting the the AWS Glue Catalog as an SQL backend?
Is it also possible to make the utilization of AWS Glue as a Spark execution engine seamless?

Thank you very much for this work and looking forward to updates. /cc @talagluck

lccasagrande · 2022-07-20T19:11:41Z

I will rebase and re-run tests. Answering your questions:

Q) Is this focused mainly on connecting the the AWS Glue Catalog as an SQL backend?

Yes, it enables the discovery and usage of database tables cataloged in Glue Data Catalog without the need to use the InferredAssetS3DataConnector. Although it was not tested already, but it may allow the usage of this framework with database tables shared through the AWS Lake Formation, like in data mesh architectures.

Q) Is it also possible to make the utilization of AWS Glue as a Spark execution engine seamless? Yes it is. The Glue Data Catalog connector is the first step. The AWS Glue Execution engine will be the second one. Lets finish this PR first and, after that, we can discuss the AWS Glue Execution Engine. I did not want to make a PR with too much contributions, but it is possible and I already have an working sample that we developed as a plugin.

…into develop

…DME. (great-expectations#5541)

…reat-expectations#5548)

…reat-expectations#5552)

…ectations#5554) * ipywidgets not required for dev since it is in requirements.txt

…ta_context_id` (great-expectations#5555) * fix: use helper method for project config logic * docs: add docstring * test: write tests

Co-authored-by: github-actions <github-actions@github.com> Co-authored-by: Chetan Kini <chetan@superconductive.com>

…s implemented `_init_datasource_store` (great-expectations#5543) * feat: init commit * feat: finish initial impl * test: write integration tests * chore: remove unused imports * docs: add comments

great_expectations/execution_engine/sparkdf_execution_engine.py

alexsherstinsky · 2022-09-28T19:50:42Z

@lccasagrande Thank you for your updates and replies. I did another round of review and left a few comments. I feel that at this point, we should be maintaining the working branch that passes all Azure checks so that once this pull request is approved (meaning only minor changes, if any at all would be remaining), then we can just merge it upon approval. Thanks!

lccasagrande · 2022-10-23T22:15:43Z

requirements.txt

@@ -7,7 +7,7 @@ Ipython>=7.16.3
 ipywidgets>=7.5.1
 jinja2>=2.10
 jsonpatch>=1.22
-jsonschema>=2.5.1
+jsonschema>=2.5.1,<=4.7.2


I had to set the max jsonschema pkg version because, with latest versions the error message from validate() is not the one that is being tested here:
assert "'variable_count' is a required property" in str(e.value).

I only got this problem with this test. I see two choices:

Upd the message in the test assertion and do not set the max pkg version.

Set the max pkg version and keep it as is.

I followed the second one.

alexsherstinsky · 2022-10-24T16:35:00Z

@lccasagrande Thank you for all this great work! How about now changing the status of this pull request to be "regular" (as in "not Draft Pull Request" any more and updating the branch with the latest develop -- then you can set it on "auto-merge" so that once it is approved, it will track any future updates with the develop branch automatically and then get merged? Thank you very much!

alexsherstinsky

@lccasagrande LGTM -- thank you so much for this very valuable contribution to Great Expectations and for your diligent efforts over the many weeks and months! /cc @kyleaton @talagluck @donaldheppner

lccasagrande added 2 commits May 15, 2022 02:33

Upg moto pkg to mock glue catalog tests

1365908

Add glue catalog data connector

d8f54ed

Fix linting errors

f99237d

lccasagrande mentioned this pull request May 15, 2022

AWS Glue Catalog Data Connector Support #4945

Closed

rdodev added the community label May 25, 2022

alexsherstinsky reviewed Jul 14, 2022

View reviewed changes

great_expectations/datasource/data_connector/configured_asset_glue_catalog_data_connector.py Outdated Show resolved Hide resolved

alexsherstinsky reviewed Jul 14, 2022

View reviewed changes

great_expectations/datasource/data_connector/inferred_asset_glue_catalog_data_connector.py Outdated Show resolved Hide resolved

alexsherstinsky suggested changes Jul 14, 2022

View reviewed changes

lccasagrande and others added 17 commits July 20, 2022 18:02

Upg moto pkg to mock glue catalog tests

34f725c

Add glue catalog data connector

b00398f

Fix linting errors

fcd9aad

Merge branch 'develop' of github.com:lccasagrande/great_expectations …

4feaa73

…into develop

[MAINTENANCE] Initial Dockerfile for running tests and associated REA…

2a08e67

…DME. (great-expectations#5541)

[BUGFIX] Explicitly pass in mostly as 1 if not set in configuration. (g…

12d398b

…reat-expectations#5548)

update VolumeDataAssistant tests (great-expectations#5551)

7b57b4f

increase precision for categorical rule for fractional comparisons (g…

8f7a2ac

…reat-expectations#5552)

[MAINTENANCE] Other dialect test (great-expectations#5547)

d5f2984

[BUGFIX] Remove ipywidgets from dev package dependencies (great-exp…

3750103

…ectations#5554) * ipywidgets not required for dev since it is in requirements.txt

[BUGFIX] Update logic used in DataContext.__init__() to persist `da…

fe88aa5

…ta_context_id` (great-expectations#5555) * fix: use helper method for project config logic * docs: add docstring * test: write tests

[RELEASE] 0.15.15

49ced48

Co-authored-by: github-actions <github-actions@github.com> Co-authored-by: Chetan Kini <chetan@superconductive.com>

[MAINTENANCE] Ensure that each class in the DataContext hierachy ha…

c6c7b25

…s implemented `_init_datasource_store` (great-expectations#5543) * feat: init commit * feat: finish initial impl * test: write integration tests * chore: remove unused imports * docs: add comments

Add mostly docs for MultiColumnMapExpectation (great-expectations#5521)

c60b96c

Add titanic dataset to test temp path

a77b059

Merge branch 'develop' into develop

93470fd

Fix glue catalog sampling test assertions

2474fbf

lccasagrande and others added 5 commits September 25, 2022 20:41

Merge branch 'develop' into develop

46bee64

Refactoring glue data connectors

0ee6b39

Fix linting

7c92438

Merge branch 'develop' into develop

11e467c

Merge branch 'develop' into develop

6f9d64d

alexsherstinsky reviewed Sep 28, 2022

View reviewed changes

great_expectations/execution_engine/sparkdf_execution_engine.py Outdated Show resolved Hide resolved

alexsherstinsky reviewed Sep 28, 2022

View reviewed changes

great_expectations/execution_engine/sparkdf_execution_engine.py Outdated Show resolved Hide resolved

alexsherstinsky reviewed Sep 28, 2022

View reviewed changes

great_expectations/execution_engine/sparkdf_execution_engine.py Outdated Show resolved Hide resolved

lccasagrande and others added 8 commits October 2, 2022 15:50

Merge branch 'develop' into develop

6a0e321

Merge branch 'develop' into develop

dc08f59

Revert to develop

86230e6

Add partitions to connector

a551d62

Add integration tests for datasource

db3e66e

Upd spark reader_path var to path

2a9d6f3

Fix test must be skipped when pyspark not found

33846e4

Fix max pkg version to not break error msgs

0ad43c7

lccasagrande commented Oct 23, 2022

View reviewed changes

Merge branch 'develop' into develop

b32a159

lccasagrande marked this pull request as ready for review October 24, 2022 16:38

lccasagrande requested a review from alexsherstinsky October 24, 2022 16:44

alexsherstinsky approved these changes Oct 24, 2022

View reviewed changes

lccasagrande and others added 2 commits October 24, 2022 15:35

Merge branch 'develop' into develop

4542fbd

Merge branch 'develop' into develop

f28e35d

alexsherstinsky enabled auto-merge (squash) October 24, 2022 20:33

alexsherstinsky added 2 commits October 24, 2022 15:47

Merge branch 'develop' into develop

5918e30

Merge branch 'develop' into develop

c7e9b44

alexsherstinsky merged commit 82ccff2 into great-expectations:develop Oct 25, 2022

itaise mentioned this pull request Dec 4, 2022

Package conflict with Airflow - jsonschema #6485

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FEATURE] Add support to AWS Glue Data Catalog #5123

[FEATURE] Add support to AWS Glue Data Catalog #5123

lccasagrande commented May 15, 2022 •

edited

Loading

netlify bot commented May 15, 2022 •

edited

Loading

rdodev commented May 25, 2022

lccasagrande commented Jun 21, 2022

rdodev commented Jun 21, 2022

alexsherstinsky left a comment

lccasagrande commented Jul 20, 2022

alexsherstinsky commented Sep 28, 2022

lccasagrande Oct 23, 2022

alexsherstinsky commented Oct 24, 2022

alexsherstinsky left a comment •

edited

Loading

[FEATURE] Add support to AWS Glue Data Catalog #5123

[FEATURE] Add support to AWS Glue Data Catalog #5123

Conversation

lccasagrande commented May 15, 2022 • edited Loading

Definition of Done

netlify bot commented May 15, 2022 • edited Loading

👷 Deploy request for niobium-lead-7998 pending review.

rdodev commented May 25, 2022

lccasagrande commented Jun 21, 2022

rdodev commented Jun 21, 2022

alexsherstinsky left a comment

Choose a reason for hiding this comment

lccasagrande commented Jul 20, 2022

alexsherstinsky commented Sep 28, 2022

lccasagrande Oct 23, 2022

Choose a reason for hiding this comment

alexsherstinsky commented Oct 24, 2022

alexsherstinsky left a comment • edited Loading

Choose a reason for hiding this comment

lccasagrande commented May 15, 2022 •

edited

Loading

netlify bot commented May 15, 2022 •

edited

Loading

alexsherstinsky left a comment •

edited

Loading