avoid merge errors when blocks contain different type #45269
Closed
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
example:
wil get
![image](https://private-user-images.githubusercontent.com/49055103/329760908-e931af66-f266-48cb-939a-7fde74be7beb.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjM3NjgyNzQsIm5iZiI6MTcyMzc2Nzk3NCwicGF0aCI6Ii80OTA1NTEwMy8zMjk3NjA5MDgtZTkzMWFmNjYtZjI2Ni00OGNiLTkzOWEtN2ZkZTc0YmU3YmViLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MTYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODE2VDAwMjYxNFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTVmYWQxNzNhZDU2YzQwMjk1OTZiYzk0YmYzZDNmMTM3NWM5NWNkMmY3YjhhYWRjY2M0N2EzMjQ4MmRiMDhkYmEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.cI2Zbolz94DUkJmCQlKH9IeO3JHZpOH-2iWQOo3KyRg)
# <class 'pyarrow.lib.Table'>, <class 'pandas.core.frame.DataFrame'>
because in blockaccess, fallback maybe arise, like
["a", "b", np.nan]
cannot cover to arrow will raise exception cause fallback, and then return pandaswhen use this code
will finally cause
# <class 'pyarrow.lib.Table'>, <class 'pandas.core.frame.DataFrame'>
merge in to_pandas method, it will raise exception.In short, I solved the problem that merging two blocks of a dataset will throw an exception when there are two formats of pandas arrow in the blocks.
Related issue number
#45236
Checks
git commit -s
) in this PR.scripts/format.sh
to lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/
under thecorresponding
.rst
file.