[FLINK-28876][python][format/orc] Support writing RowData into Orc files #20505
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Vancior
force-pushed
the
feat/py_orc_bulk_write
branch
from
9783b83 to
f35f82a
Compare
Collaborator
Vancior
force-pushed
the
feat/py_orc_bulk_write
branch
3 times, most recently
from
2cd5e87 to
30e8b26
Compare
HuangXingBo
reviewed
Aug 9, 2022
Contributor
HuangXingBo
left a comment
HuangXingBo
left a comment
There was a problem hiding this comment.
@Vancior Thanks a lot for the PR. Looks good overall. I only have left two comments.
| {{< /tabs >}} | ||
|
|
||
| For PyFlink users, `OrcBulkWriters.for_row_data_vectorization` could be used to create `BulkWriterFactory` to write `Row` records to files in Orc format. | ||
| It should be noted that if the preceding operator of sink is an operator producing `RowData` records, e.g. CSV source, it needs to be converted to `Row` records before writing to sink. |
Contributor
There was a problem hiding this comment.
Maybe we need to give a some description to help pyflink users to understand RowData, which can be a doc link or something.
Contributor
Author
There was a problem hiding this comment.
I'll try to eliminate this in another PR.
| ) | ||
|
|
||
| @staticmethod | ||
| def _create_properties(conf: Configuration): |
Contributor
There was a problem hiding this comment.
this method can be moved to datastream.utils?
Vancior
force-pushed
the
feat/py_orc_bulk_write
branch
2 times, most recently
from
801ea8a to
5fdb220
Compare
Vancior
force-pushed
the
feat/py_orc_bulk_write
branch
from
5fdb220 to
42a3f34
Compare
dianfu
approved these changes
Aug 10, 2022
huangxiaofeng10047
pushed a commit
to huangxiaofeng10047/flink
that referenced
this pull request
Nov 3, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is the purpose of the change
This PR supports
OrcBulkWriters.for_row_data_vectorizationAPI to create aBulkWriterFactorythat writes rows into Orc files in a batch fashion. This branch will be rebased after #20499 merged.Verifying this change
This change added tests and can be verified as follows:
FileSinkOrcBulkWritersTestsin test_orc.pyDoes this pull request potentially affect one of the following parts:
@Public(Evolving): (yes)Documentation