Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MAINTENANCE] SQL backend integration tests #9822

Merged
merged 23 commits into from Apr 29, 2024

Conversation

tyler-hoffman
Copy link
Contributor

@tyler-hoffman tyler-hoffman commented Apr 26, 2024

Approach

This leans on the setup of existing docs tests, so it uses the test script runner. There's a bit of grossness around passing in the names of methods that end up getting called, but I don't think a larger refactor of this is worth it at the moment.

  • Description of PR changes above includes a link to an existing GitHub issue
  • PR title is prefixed with one of: [BUGFIX], [FEATURE], [DOCS], [MAINTENANCE], [CONTRIB]
  • Code is linted - run invoke lint (uses ruff format + ruff check)
  • Appropriate tests and docs have been updated

For more information about contributing, see Contribute.

After you submit your PR, keep the page open and monitor the statuses of the various checks made by our continuous integration process at the bottom of the page. Please fix any issues that come up and reach out on Slack if you need help. Thanks for contributing!

Copy link

netlify bot commented Apr 26, 2024

Deploy Preview for niobium-lead-7998 canceled.

Name Link
🔨 Latest commit 2868c9b
🔍 Latest deploy log https://app.netlify.com/sites/niobium-lead-7998/deploys/662fda085703660009132b3c

num_expected_batch_definitions=299,
num_expected_rows_in_first_batch_definition=2,
expected_column_values=self.taxi_test_data.year_month_day_batch_identifier_data(),
),
TaxiPartitioningTestCase(
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We aren't supporting these yet in 1.0

Copy link

codecov bot commented Apr 26, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 82.52%. Comparing base (f204d6b) to head (2868c9b).

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #9822      +/-   ##
===========================================
- Coverage    82.54%   82.52%   -0.03%     
===========================================
  Files          504      504              
  Lines        45097    45097              
===========================================
- Hits         37227    37215      -12     
- Misses        7870     7882      +12     
Flag Coverage Δ
3.10 65.64% <ø> (ø)
3.11 65.64% <ø> (ø)
3.11 athena or clickhouse or openpyxl or pyarrow or project or sqlite or aws_creds 53.30% <ø> (-0.01%) ⬇️
3.11 aws_deps 44.21% <ø> (ø)
3.11 big 61.07% <ø> (ø)
3.11 databricks 45.26% <ø> (ø)
3.11 filesystem 62.53% <ø> (ø)
3.11 mssql 47.97% <ø> (ø)
3.11 mysql 48.02% <ø> (ø)
3.11 postgresql 51.68% <ø> (ø)
3.11 snowflake 45.85% <ø> (-0.01%) ⬇️
3.11 spark 57.83% <ø> (ø)
3.11 trino 51.00% <ø> (ø)
3.8 65.66% <ø> (ø)
3.8 athena or clickhouse or openpyxl or pyarrow or project or sqlite or aws_creds 53.31% <ø> (-0.01%) ⬇️
3.8 aws_deps 44.22% <ø> (ø)
3.8 big 61.07% <ø> (ø)
3.8 databricks 45.27% <ø> (ø)
3.8 filesystem 62.54% <ø> (ø)
3.8 mssql 47.95% <ø> (ø)
3.8 mysql 48.01% <ø> (ø)
3.8 postgresql 51.67% <ø> (ø)
3.8 snowflake 45.86% <ø> (-0.01%) ⬇️
3.8 spark 57.78% <ø> (ø)
3.8 trino 50.99% <ø> (ø)
3.9 65.65% <ø> (ø)
cloud 0.00% <ø> (ø)
docs-basic 51.90% <ø> (-0.13%) ⬇️
docs-creds-needed 52.99% <ø> (+0.04%) ⬆️
docs-spark 51.66% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Contributor

@billdirks billdirks left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for doing this. This seems like the way to leverage the established test pattern.

tests/integration/db/taxi_data_utils.py Show resolved Hide resolved
tests/integration/db/taxi_data_utils.py Outdated Show resolved Hide resolved
tests/integration/db/taxi_data_utils.py Outdated Show resolved Hide resolved
@tyler-hoffman tyler-hoffman changed the title M/v1 291/sql backend integration tests [MAINTENANCE] SQL backend integration tests Apr 29, 2024
@tyler-hoffman tyler-hoffman marked this pull request as ready for review April 29, 2024 14:42
@@ -965,6 +967,26 @@ def get_connection_string_and_dialect(
return dialect, connection_string


def add_datasource(
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Attention reviewer! I don't like this, but I think it's the best we can do without restructuring tests more or passing method names as strings 🤮 . Open to suggestions if there's something better we can do!

Copy link
Contributor

@billdirks billdirks left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

) -> SQLDatasource:
"""Add a datasource to the context based on the dialect from config file.

Needed because context.data_sources.add_sql is prohibitted when
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Needed because context.data_sources.add_sql is prohibitted when
Needed because context.data_sources.add_sql is prohibited when

@tyler-hoffman tyler-hoffman added this pull request to the merge queue Apr 29, 2024
Merged via the queue into develop with commit 5049346 Apr 29, 2024
68 checks passed
@tyler-hoffman tyler-hoffman deleted the m/v1-291/sql-backend-integration-tests branch April 29, 2024 19:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants