perf(dask): avoid triggering compute for dynamic limit/offset #8747
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mfatihaktas
added
performance
mfatihaktas
force-pushed
the
perf/dask/avoid-triggering-compute-for-offset
branch
from
3630b55
to
eec1af8
Compare
mfatihaktas
changed the title
mfatihaktas
force-pushed
the
perf/dask/avoid-triggering-compute-for-offset
branch
from
eec1af8
to
544eb4a
Compare
|
With the caveat that I don't know much about Dask, the code no longer looks like it obviously triggers compute. @jcrist @ncclementi Can one of y'all review? |
jcrist
reviewed
Mar 25, 2024
There was a problem hiding this comment.
This is better than what's there currently, but we could still do better in the long run:
- For the common case of literal limits and no offset we should be able to take a more efficient path of calling
ddf.head(n, npartitions=-1). We're currently hampered by the rewrites in the pandas backend always upcasting these scalars to tables with a single element (making the literal scalar nature opaque to the dask backend). - Even for dynamic limits we should be able to rely on
ddf.head(n, npartitions=-1), but that also only works if the input is represented as scalar and not a table. - Only the case where
offsetis non-0 requires something like what we're doing here (though again ifoffsetis a scalar instead of a table there are more ergonomic ways of spelling this). IMO offset is a rare enough case that if we wanted to error at compile time for non-zerooffsetin the dask backend that would be fine.
Anyway, this should be fine to go in as is.
cpcloud
approved these changes
Mar 26, 2024
cpcloud
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of changes
Aims to close #8186.