-
Notifications
You must be signed in to change notification settings - Fork 5.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
For a dataset comprised of both empty and non-empty blocks, let the non-empty blocks determine the schema #22834
Conversation
…on-empty blocks determine the schema
I realize I should have created a new branch for a different issue. The previous commit logs make it a bit messy. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice!
Could you explain the root cause of the issue? Schema of empty block is wrong? Should we always set schema of empty block to None? |
The schema for empty block is default to pyarrow (#22673 (comment)). |
If we don't want to trust the schema of empty block even if it's set, should we just set to None in the first place? Otherwise we will see inconsistent schemas for blocks in the same dataset? |
The block is of type Union[List[T], "pyarrow.Table", "pandas.DataFrame", bytes], so even if it's empty, it already has a type and hence an implied the schema type. |
Seems tests are failing. |
Yeah. As I was working on fixing it, I realized that it is not feasible to rely on num_rows in block metadata to determine whether a block is empty. IIUC, we do not necessarily always know num_rows because we employ lazy execution/lazy data loading. |
Why are these changes needed?
There is a bug in combining the results from map_batches: if we create two dataset out of the same data, but with different num of partitions, we may get different results when run the same map_batches() on them. That is, num of partitions is affecting the map_batches() results, which should not.
Related issue number
Closes #22673
Checks
scripts/format.sh
to lint the changes in this PR.