-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
PARQUET-2451: Add BYTE_STREAM_SPLIT support for FIXED_LEN_BYTE_ARRAY, INT32 and INT64 #1291
Conversation
c1e1aa5
to
b3031ad
Compare
@wgtmac @etseidl @gszadovszky This is a draft PR. It works fine on the integration test but still needs unit tests. Since I'm quite new to this codebase (and to Java in general), I welcome any early comments. |
parquet-hadoop/src/test/java/org/apache/parquet/hadoop/TestInterOpReadByteStreamSplit.java
Outdated
Show resolved
Hide resolved
b3031ad
to
1f5d488
Compare
...ava/org/apache/parquet/column/values/bytestreamsplit/ByteStreamSplitValuesReaderForFLBA.java
Show resolved
Hide resolved
parquet-hadoop/src/test/java/org/apache/parquet/hadoop/TestInterOpReadByteStreamSplit.java
Outdated
Show resolved
Hide resolved
This is ready for review now. The only failing test is the interop test, since the required file is not in parquet-testing yet (I have checked it passes locally). |
...umn/src/main/java/org/apache/parquet/column/values/factory/DefaultV2ValuesWriterFactory.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I will take a look after I am back from vacation.
@gszadovszky Could you help review this, too?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You probably want to remove DO NOT MERGE:
from the title.
...ava/org/apache/parquet/column/values/bytestreamsplit/ByteStreamSplitValuesReaderForFLBA.java
Outdated
Show resolved
Hide resolved
parquet-column/src/main/java/org/apache/parquet/column/ParquetProperties.java
Show resolved
Hide resolved
.../org/apache/parquet/column/values/bytestreamsplit/ByteStreamSplitValuesReaderForInteger.java
Outdated
Show resolved
Hide resolved
...ava/org/apache/parquet/column/values/bytestreamsplit/ByteStreamSplitValuesReaderForLong.java
Outdated
Show resolved
Hide resolved
...umn/src/main/java/org/apache/parquet/column/values/factory/DefaultV2ValuesWriterFactory.java
Outdated
Show resolved
Hide resolved
...umn/src/main/java/org/apache/parquet/column/values/factory/DefaultV2ValuesWriterFactory.java
Outdated
Show resolved
Hide resolved
...a/org/apache/parquet/column/values/bytestreamsplit/ByteStreamSplitValuesReaderForDouble.java
Outdated
Show resolved
Hide resolved
.../org/apache/parquet/column/values/bytestreamsplit/ByteStreamSplitValuesReaderForInteger.java
Outdated
Show resolved
Hide resolved
.../main/java/org/apache/parquet/column/values/bytestreamsplit/ByteStreamSplitValuesWriter.java
Outdated
Show resolved
Hide resolved
.../main/java/org/apache/parquet/column/values/bytestreamsplit/ByteStreamSplitValuesReader.java
Show resolved
Hide resolved
.../main/java/org/apache/parquet/column/values/bytestreamsplit/ByteStreamSplitValuesReader.java
Show resolved
Hide resolved
c52e48a
to
76ca342
Compare
76ca342
to
f4a7ce3
Compare
For the record, I must still address some of the review comments. |
f8634b3
to
1ad366f
Compare
parquet-hadoop/src/test/java/org/apache/parquet/hadoop/TestInterOpReadByteStreamSplit.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1 on my side.
1ad366f
to
0f7a043
Compare
884cb22
to
431a025
Compare
@gszadovszky I think I addressed your comments, could you take another look? |
@pitrou Do you want to have this in the 1.14.0 release? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry for the late response, @pitrou. Thanks for fixing my comments.
Implement the format additions described in PARQUET-2414.
Jira
them in the PR title. For example, "PARQUET-1234: My Parquet PR"
the ASF 3rd Party License Policy.
Tests
Commits
from "How to write a good git commit message":
Style
mvn spotless:apply -Pvector-plugins
Documentation