New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FEATURE] Batch definition sorting #9720
Conversation
✅ Deploy Preview for niobium-lead-7998 canceled.
|
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #9720 +/- ##
===========================================
+ Coverage 82.64% 82.69% +0.04%
===========================================
Files 512 512
Lines 46782 46813 +31
===========================================
+ Hits 38663 38712 +49
+ Misses 8119 8101 -18
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, LGTM! A couple very minor comments.
Overview
Adds
sort_batches_ascending
property toPartitioner
s, including both the (soon to be removed) generic ones, as well as the asset-specific ones.Most of the changes here are around schemas because of ^.
Important things to look for
DataAsset::sort_batches
has been updated to just look at the partitioner tuples. I might circle back on this to make the logic a bit simpler, but for now left it with minimal changes.get_batch_list_from_batch_request
What this does not do
This PR does not touch
DataAsset::add_sorters
orDataAsset::order_by
that it sets. I plan to remove that next, but this PR is already huge enough.invoke lint
(usesruff format
+ruff check
)For more information about contributing, see Contribute.
After you submit your PR, keep the page open and monitor the statuses of the various checks made by our continuous integration process at the bottom of the page. Please fix any issues that come up and reach out on Slack if you need help. Thanks for contributing!