Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MAINTENANCE] Integration tests around s3 batches #9846

Merged
merged 6 commits into from
May 3, 2024

Conversation

tyler-hoffman
Copy link
Contributor

  • Description of PR changes above includes a link to an existing GitHub issue
  • PR title is prefixed with one of: [BUGFIX], [FEATURE], [DOCS], [MAINTENANCE], [CONTRIB]
  • Code is linted - run invoke lint (uses ruff format + ruff check)
  • Appropriate tests and docs have been updated

For more information about contributing, see Contribute.

After you submit your PR, keep the page open and monitor the statuses of the various checks made by our continuous integration process at the bottom of the page. Please fix any issues that come up and reach out on Slack if you need help. Thanks for contributing!

Copy link

netlify bot commented Apr 30, 2024

Deploy Preview for niobium-lead-7998 canceled.

Name Link
🔨 Latest commit 4bb8cf7
🔍 Latest deploy log https://app.netlify.com/sites/niobium-lead-7998/deploys/6634e5931d808600080bc676

Copy link

codecov bot commented Apr 30, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 78.63%. Comparing base (fd560d9) to head (4bb8cf7).
Report is 17 commits behind head on develop.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #9846      +/-   ##
===========================================
- Coverage    82.51%   78.63%   -3.89%     
===========================================
  Files          504      484      -20     
  Lines        45079    42536    -2543     
===========================================
- Hits         37199    33449    -3750     
- Misses        7880     9087    +1207     
Flag Coverage Δ
3.10 64.51% <ø> (-1.13%) ⬇️
3.10 athena or clickhouse or openpyxl or pyarrow or project or sqlite or aws_creds ?
3.10 aws_deps ?
3.10 big ?
3.10 databricks ?
3.10 filesystem ?
3.10 mssql ?
3.10 mysql ?
3.10 postgresql ?
3.10 snowflake ?
3.10 spark ?
3.10 trino ?
3.11 64.51% <ø> (-1.13%) ⬇️
3.11 athena or clickhouse or openpyxl or pyarrow or project or sqlite or aws_creds 53.90% <ø> (+0.59%) ⬆️
3.11 aws_deps 44.86% <ø> (+0.63%) ⬆️
3.11 big 55.98% <ø> (-5.10%) ⬇️
3.11 databricks 46.01% <ø> (+0.74%) ⬆️
3.11 filesystem 61.20% <ø> (-1.33%) ⬇️
3.11 mssql 48.88% <ø> (+0.89%) ⬆️
3.11 mysql 48.93% <ø> (+0.89%) ⬆️
3.11 postgresql 52.79% <ø> (+1.09%) ⬆️
3.11 snowflake 46.64% <ø> (+0.77%) ⬆️
3.11 spark 57.06% <ø> (-0.79%) ⬇️
3.11 trino 50.79% <ø> (-0.23%) ⬇️
3.8 64.54% <ø> (-1.12%) ⬇️
3.8 athena or clickhouse or openpyxl or pyarrow or project or sqlite or aws_creds 53.90% <ø> (+0.59%) ⬆️
3.8 aws_deps 44.87% <ø> (+0.63%) ⬆️
3.8 big 55.99% <ø> (-5.10%) ⬇️
3.8 databricks 46.02% <ø> (+0.74%) ⬆️
3.8 filesystem 61.22% <ø> (-1.33%) ⬇️
3.8 mssql 48.86% <ø> (+0.89%) ⬆️
3.8 mysql 48.92% <ø> (+0.89%) ⬆️
3.8 postgresql 52.78% <ø> (+1.09%) ⬆️
3.8 snowflake 46.65% <ø> (+0.78%) ⬆️
3.8 spark 57.02% <ø> (-0.79%) ⬇️
3.8 trino 50.77% <ø> (-0.23%) ⬇️
3.9 64.53% <ø> (-1.12%) ⬇️
3.9 athena or clickhouse or openpyxl or pyarrow or project or sqlite or aws_creds ?
3.9 aws_deps ?
3.9 big ?
3.9 databricks ?
3.9 filesystem ?
3.9 mssql ?
3.9 mysql ?
3.9 postgresql ?
3.9 snowflake ?
3.9 spark ?
3.9 trino ?
cloud 0.00% <ø> (ø)
docs-basic 49.00% <ø> (-2.91%) ⬇️
docs-creds-needed 50.13% <ø> (-2.88%) ⬇️
docs-spark 48.22% <ø> (-3.46%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

data_asset = datasource.add_csv_asset(name=asset_name, s3_prefix=s3_prefix)

batch_definition = data_asset.add_batch_definition_path(
"monthly",
Copy link
Member

@jcampbell jcampbell May 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just for clarity, I think we should name this something different:

Suggested change
"monthly",
"path",

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call. Fixed!

@tyler-hoffman tyler-hoffman added this pull request to the merge queue May 3, 2024
Merged via the queue into develop with commit 0cc1455 May 3, 2024
68 checks passed
@tyler-hoffman tyler-hoffman deleted the m/v1-291/s3-integration-tests branch May 3, 2024 14:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants