Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MAINTENANCE] Remove remaining references to block style datasources #9881

Conversation

tyler-hoffman
Copy link
Contributor

@tyler-hoffman tyler-hoffman commented May 3, 2024

Removes all references to block style Datasource class. Note: This does not remove BaseDatasource; that will be removed in a followup PR.

NOTE: This raises some new DataContextErrors in places if we get a non-fluent datasource. These errors should not be hit and are just to make the type checker happy.

There are a lot of other test deletions in here that I think are reasonable. The main categories:

  • block style Datasource tests
  • tests around datasource CRUD on context, since that's been replaced by fluent
  • multi-batch tests, since we aren't supporting these in 1.0 and their setup involved block style
  • lots of stuff around non-fluent data connectors
  • Description of PR changes above includes a link to an existing GitHub issue
  • PR title is prefixed with one of: [BUGFIX], [FEATURE], [DOCS], [MAINTENANCE], [CONTRIB]
  • Code is linted - run invoke lint (uses ruff format + ruff check)
  • Appropriate tests and docs have been updated

For more information about contributing, see Contribute.

After you submit your PR, keep the page open and monitor the statuses of the various checks made by our continuous integration process at the bottom of the page. Please fix any issues that come up and reach out on Slack if you need help. Thanks for contributing!

Copy link

netlify bot commented May 3, 2024

Deploy Preview for niobium-lead-7998 canceled.

Name Link
🔨 Latest commit d3934c7
🔍 Latest deploy log https://app.netlify.com/sites/niobium-lead-7998/deploys/663bb6c2bc0c110008f155bd

@tyler-hoffman tyler-hoffman changed the title M/v1 25/remove remaining references to block style datasoures [MAINTENANCE] Remove remaining references to block style datasources May 3, 2024
Copy link

codecov bot commented May 6, 2024

Codecov Report

Attention: Patch coverage is 58.33333% with 5 lines in your changes are missing coverage. Please review.

Project coverage is 77.36%. Comparing base (86a5862) to head (d3934c7).
Report is 7 commits behind head on develop.

Files Patch % Lines
...data_context/data_context/abstract_data_context.py 40.00% 3 Missing ⚠️
great_expectations/datasource/datasource_dict.py 33.33% 2 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #9881      +/-   ##
===========================================
- Coverage    77.77%   77.36%   -0.42%     
===========================================
  Files          494      493       -1     
  Lines        42389    42360      -29     
===========================================
- Hits         32967    32770     -197     
- Misses        9422     9590     +168     
Flag Coverage Δ
3.10 63.93% <50.00%> (-0.30%) ⬇️
3.10 athena or clickhouse or openpyxl or pyarrow or project or sqlite or aws_creds ?
3.10 aws_deps ?
3.10 big ?
3.10 databricks ?
3.10 filesystem ?
3.10 mssql ?
3.10 mysql ?
3.10 postgresql ?
3.10 snowflake ?
3.10 spark ?
3.10 trino ?
3.11 63.93% <50.00%> (-0.30%) ⬇️
3.11 athena or clickhouse or openpyxl or pyarrow or project or sqlite or aws_creds 53.92% <41.66%> (-0.07%) ⬇️
3.11 aws_deps 44.75% <41.66%> (-0.14%) ⬇️
3.11 big 54.16% <41.66%> (-1.71%) ⬇️
3.11 databricks 45.95% <41.66%> (-0.12%) ⬇️
3.11 filesystem 59.29% <41.66%> (-1.54%) ⬇️
3.11 mssql 48.70% <41.66%> (-0.18%) ⬇️
3.11 mysql 48.76% <41.66%> (-0.18%) ⬇️
3.11 postgresql 52.66% <41.66%> (-0.11%) ⬇️
3.11 snowflake 46.56% <41.66%> (-0.12%) ⬇️
3.11 spark 56.16% <41.66%> (-1.12%) ⬇️
3.11 trino 50.74% <41.66%> (-0.12%) ⬇️
3.8 63.96% <50.00%> (-0.30%) ⬇️
3.8 athena or clickhouse or openpyxl or pyarrow or project or sqlite or aws_creds 53.93% <41.66%> (-0.07%) ⬇️
3.8 aws_deps 44.77% <41.66%> (-0.14%) ⬇️
3.8 big 54.17% <41.66%> (-1.71%) ⬇️
3.8 databricks 45.97% <41.66%> (-0.12%) ⬇️
3.8 filesystem 59.30% <41.66%> (-1.54%) ⬇️
3.8 mssql 48.68% <41.66%> (-0.18%) ⬇️
3.8 mysql 48.74% <41.66%> (-0.18%) ⬇️
3.8 postgresql 52.65% <41.66%> (-0.11%) ⬇️
3.8 snowflake 46.57% <41.66%> (-0.12%) ⬇️
3.8 spark 56.13% <41.66%> (-1.12%) ⬇️
3.8 trino 50.72% <41.66%> (-0.12%) ⬇️
3.9 63.95% <50.00%> (-0.30%) ⬇️
3.9 athena or clickhouse or openpyxl or pyarrow or project or sqlite or aws_creds ?
3.9 aws_deps ?
3.9 big ?
3.9 databricks ?
3.9 filesystem ?
3.9 mssql ?
3.9 mysql ?
3.9 postgresql ?
3.9 snowflake ?
3.9 spark ?
3.9 trino ?
cloud 0.00% <0.00%> (ø)
docs-basic 47.43% <50.00%> (-1.49%) ⬇️
docs-creds-needed 48.89% <50.00%> (-1.47%) ⬇️
docs-spark 47.15% <58.33%> (-1.29%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

include_pandas: bool = True,
include_spark: bool = True,
) -> AbstractDataContext:
def build_in_memory_runtime_context() -> AbstractDataContext:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not relevant to your PR but I don't know why this method is in our repo if we only really use it as a test util - same comment for a lot of the methods in here we might want to write a ticket about removing them to undo some import cycles they cause

@@ -598,7 +597,10 @@ def test_escape_all_config_variables_skip_substitution_vars(


@pytest.mark.filesystem
def test_create_data_context_and_config_vars_in_code(tmp_path_factory, monkeypatch):
# @mock.patch.object(sa, "create_engine")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit - can we remove?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done!

@tyler-hoffman tyler-hoffman added this pull request to the merge queue May 8, 2024
Copy link
Contributor

@billdirks billdirks left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't gotten too far but wanted to send what I had before more meetings. I've looked through the non-test code and some of the test files.

include_pandas: bool = True,
include_spark: bool = True,
) -> AbstractDataContext:
def build_in_memory_runtime_context() -> AbstractDataContext:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a test util in the normal code package?
Now it's building a context with no datasources, maybe we can delete this whole function? Maybe that involves deleting BaseDatasource which is happening as a followup?

@@ -894,61 +893,6 @@ def titanic_pandas_data_context_with_v013_datasource_with_checkpoints_v1_with_em
context = get_context(context_root_dir=context_path)
assert context.root_directory == context_path

datasource_config: str = f"""
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wondering if we can delete this whole fixture since there is no longer a datasource.

Merged via the queue into develop with commit a1c0c5a May 8, 2024
69 checks passed
@tyler-hoffman tyler-hoffman deleted the m/v1-25/remove-remaining-references-to-block-style-datasoures branch May 8, 2024 18:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants