Implement a parquet parser and printer
#4116
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
balavinaithirthan
force-pushed
the
topic/parquetPrinterAndParser
branch
from
de5f56b
to
bba8cea
Compare
dominiklohmann
changed the title
parquet parser and printer
balavinaithirthan
force-pushed
the
topic/parquetPrinterAndParser
branch
from
8c4e97e
to
97993b5
Compare
balavinaithirthan
force-pushed
the
topic/parquetPrinterAndParser
branch
3 times, most recently
from
7f1f94f
to
bb55300
Compare
balavinaithirthan
added
format
dominiklohmann
approved these changes
Apr 19, 2024
balavinaithirthan
force-pushed
the
topic/parquetPrinterAndParser
branch
2 times, most recently
from
ef81bba
to
ddf98f9
Compare
modify .gitignore
Improve naming conventions and remove comments Clean up naming Handle errors in parquet printer instance construction Move `contiguous_buffer_stream` into the `parquet` plugin
Fix parquet round trip errors and start test file from feather tests
modify gitignore
…omments for stream API.
Reduce filesize of feather test outputs Modify feather tests to account for batching Update parquet test with correct formatting Modify parquet tests to remove feather-only python script
Modify tests to remove local file dependence Format whitespace test.bats Modify feather and parquet tests to not rely on local user system
balavinaithirthan
force-pushed
the
topic/parquetPrinterAndParser
branch
from
ddf98f9
to
1fb3114
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We developed a plugin for Tenzir that parses and prints data in the parquet format, similar to other formats like JSON, feather, and YAML. Our parquet parser and printer enable parsing and printing through operators such as write, to, and from. Parquet requires a single schema. The parser is implemented in a blocking format and receives an upstream generator of chunk_ptr (byte representation) as input, which returns a generator of table_slice for downstream operators. Within the parser, the complete parquet input with metadata is collected and converted into record batches before being co_yielded as a table_slice.
In contrast, the printer is a streaming class that takes a generator of table slices as input and outputs a generator of chunks. Table slices are converted to record batches and written as parquet, with the parquet footer written at the end based on the metadata from the stream. To create valid metadata, we derived a new arrow::io::OutputStream that acts as a file for metadata purposes and a stream for memory efficiency. The generator of table_slice input is written to a byte buffer before being co_yielded at the end of a row.
Change log: The
parquetformat now reads and writes Arrow IPC streams and Parquet files. The format is streamable forwrite parquet.