[Data] Dedupe repeated schema during ParquetDatasource metadata prefetching
#44750
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Scott Lee <sjl@anyscale.com>
raulchen
reviewed
Apr 15, 2024
| # serialize it to compare equality and store in dict. | ||
| fragment_md = _SerializedFragmentMetadata(fragment_md) | ||
| md_exists = False | ||
| for unique_md, md_indices in self._unique_metadata.items(): |
There was a problem hiding this comment.
since we've already serialized the FileMetaData objects to binaries, we can simply put the binaries in a set to dedup them, no need to have this linear scan.
| for fragment_idx, fragment_md in enumerate(metadata_list): | ||
| # pyarrow.parquet.FileMetaData is not hashable, so we need to | ||
| # serialize it to compare equality and store in dict. | ||
| fragment_md = _SerializedFragmentMetadata(fragment_md) |
There was a problem hiding this comment.
It looks like that FileMetaData also contains attributes like num_rows, which is different.
Maybe we should only dedupe a few attributes that are likely to be identical (e.g. schema)
Signed-off-by: Scott Lee <sjl@anyscale.com>
Signed-off-by: Scott Lee <sjl@anyscale.com>
Signed-off-by: Scott Lee <sjl@anyscale.com>
|
For the following Parquet/metadata related release tests, checked the release test results, no regression from nightly results: https://buildkite.com/ray-project/release/builds/13878#_
|
scottjlee
requested review from
ericl,
scv119,
c21,
amogkam,
bveeramani,
stephanie-wang and
omatthew98
as code owners
scottjlee
changed the title
ParquetDatasource metadata prefetchingParquetDatasource metadata prefetching
raulchen
approved these changes
Apr 19, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
For large datasets consisting of many files and many columns, the metadata for individual Parquet file fragments can be significantly large, using up memory resources. Often times, the metadata are identical across multiple Parquet file fragments, so we can save memory usage by maintaining a set of unique MetaData objects, and referencing the common set of MetaData objects in
ParquetDatasource.Related issue number
Closes #44754
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.