Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Add sort_ascending to BatchDefinition fluent API #9756

Merged
merged 1 commit into from Apr 15, 2024

Conversation

joshua-stauffer
Copy link
Member

Recent changes added a sort_ascending: bool parameter to Partitioners, so this PR allows defining that from the BatchDefinition fluent API.

Copy link

netlify bot commented Apr 15, 2024

Deploy Preview for niobium-lead-7998 canceled.

Name Link
🔨 Latest commit 2f10daa
🔍 Latest deploy log https://app.netlify.com/sites/niobium-lead-7998/deploys/661d4268cc20dd0008bb878e

Copy link

codecov bot commented Apr 15, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 82.68%. Comparing base (c6e31c7) to head (2f10daa).
Report is 1 commits behind head on develop.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #9756      +/-   ##
===========================================
- Coverage    82.68%   82.68%   -0.01%     
===========================================
  Files          512      512              
  Lines        46896    46896              
===========================================
- Hits         38778    38777       -1     
- Misses        8118     8119       +1     
Flag Coverage Δ
3.10 65.03% <ø> (ø)
3.10 athena or clickhouse or openpyxl or pyarrow or project or sqlite or aws_creds ?
3.10 aws_deps ?
3.10 big ?
3.10 databricks ?
3.10 filesystem ?
3.10 mssql ?
3.10 mysql ?
3.10 postgresql ?
3.10 snowflake ?
3.10 spark ?
3.10 trino ?
3.11 65.03% <ø> (ø)
3.11 athena or clickhouse or openpyxl or pyarrow or project or sqlite or aws_creds 53.74% <ø> (ø)
3.11 aws_deps 48.84% <ø> (ø)
3.11 big 63.94% <ø> (ø)
3.11 databricks 48.03% <ø> (ø)
3.11 filesystem 63.72% <ø> (-0.01%) ⬇️
3.11 mssql 47.25% <ø> (ø)
3.11 mysql 47.30% <ø> (ø)
3.11 postgresql 54.02% <ø> (ø)
3.11 snowflake 48.55% <ø> (-0.01%) ⬇️
3.11 spark 60.42% <ø> (ø)
3.11 trino 53.66% <ø> (ø)
3.8 65.04% <ø> (ø)
3.8 athena or clickhouse or openpyxl or pyarrow or project or sqlite or aws_creds 53.75% <ø> (ø)
3.8 aws_deps 48.85% <ø> (ø)
3.8 big 63.94% <ø> (ø)
3.8 databricks 48.04% <ø> (ø)
3.8 filesystem 63.72% <ø> (-0.01%) ⬇️
3.8 mssql 47.23% <ø> (ø)
3.8 mysql 47.28% <ø> (ø)
3.8 postgresql 54.01% <ø> (ø)
3.8 snowflake 48.57% <ø> (ø)
3.8 spark 60.38% <ø> (ø)
3.8 trino 53.65% <ø> (ø)
3.9 65.04% <ø> (ø)
3.9 athena or clickhouse or openpyxl or pyarrow or project or sqlite or aws_creds ?
3.9 aws_deps ?
3.9 big ?
3.9 databricks ?
3.9 filesystem ?
3.9 mssql ?
3.9 mysql ?
3.9 postgresql ?
3.9 snowflake ?
3.9 spark ?
3.9 trino ?
cloud 0.00% <ø> (ø)
docs-basic 54.30% <ø> (ø)
docs-creds-needed 54.87% <ø> (ø)
docs-spark 54.40% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Comment on lines +167 to +169
batch_definition = asset.add_batch_definition_daily(
name=name, column=column, sort_ascending=sort_ascending
)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The output of this test doesn't change based on the new param right? Could we add a test where the order matters?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @cdkini! These methods just return a BatchDefinition, not a list of batches; that functionality is tested in test_get_batch_list_from_batch_request__sort_ascending/descending. I created a followup ticket to add integration tests around the fluent batch def api.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will add a comment in that ticket that explicitly lists testing sorters as part of the AC.

@joshua-stauffer joshua-stauffer added this pull request to the merge queue Apr 15, 2024
Merged via the queue into develop with commit f293aa2 Apr 15, 2024
69 of 70 checks passed
@joshua-stauffer joshua-stauffer deleted the f/v1-242/pass_through_sort branch April 15, 2024 19:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants