ORC-1327: Exclude the proto files from the nohive jar #1334

Fokko · 2022-12-06T17:33:08Z

Why are the changes needed?

We're seeing some conflicts in the proto files when updating Iceberg in Trino: trinodb/trino#15079 (comment)

I think we should exclude the proto files from the nohive jar.

Before the change:

➜  java git:(main) ✗ rm -rf core/target/
➜  java git:(main) ✗ mvn package -DskipTests
➜  java git:(main) ✗ cd core/target
➜  target git:(main) ✗ unzip orc-core-1.9.0-SNAPSHOT-nohive.jar
➜  target git:(main) find . | grep -i "google/protobuf/"
./google/protobuf/timestamp.proto
./google/protobuf/field_mask.proto
./google/protobuf/api.proto
./google/protobuf/duration.proto
./google/protobuf/struct.proto
./google/protobuf/wrappers.proto
./google/protobuf/source_context.proto
./google/protobuf/any.proto
./google/protobuf/type.proto
./google/protobuf/empty.proto
./google/protobuf/compiler
./google/protobuf/compiler/plugin.proto
./google/protobuf/descriptor.proto

After the change:

➜  java git:(main) ✗ rm -rf core/target/
➜  java git:(main) ✗ mvn package -DskipTests
➜  java git:(main) ✗ cd core/target
➜  target git:(main) ✗ unzip orc-core-1.9.0-SNAPSHOT-nohive.jar
➜  target git:(main) ✗ find . | grep -i "google/protobuf/"
∅

Interestingly enough, the normal jar doesn't contain these proto files:

➜  java git:(main) ✗ rm -rf core/target/
➜  java git:(main) ✗ mvn package -DskipTests
➜  java git:(main) ✗ cd core/target
➜  target git:(main) unzip orc-core-1.9.0-SNAPSHOT.jar| grep -i proto
➜  target git:(main) find . | grep -i "google/protobuf/"
∅

What changes were proposed in this pull request?

Exclude the proto files

How was this patch tested?

Inspecting the produced jar

Closes #1333

Before the change: ``` ➜ java git:(main) ✗ rm -rf core/target/ ➜ java git:(main) ✗ mvn package -DskipTests ➜ java git:(main) ✗ cd core/target ➜ target git:(main) ✗ unzip orc-core-1.9.0-SNAPSHOT-nohive.jar ➜ target git:(main) find . | grep -i "google/protobuf/" ./google/protobuf/timestamp.proto ./google/protobuf/field_mask.proto ./google/protobuf/api.proto ./google/protobuf/duration.proto ./google/protobuf/struct.proto ./google/protobuf/wrappers.proto ./google/protobuf/source_context.proto ./google/protobuf/any.proto ./google/protobuf/type.proto ./google/protobuf/empty.proto ./google/protobuf/compiler ./google/protobuf/compiler/plugin.proto ./google/protobuf/descriptor.proto ``` After the change: ``` ➜ java git:(main) ✗ rm -rf core/target/ ➜ java git:(main) ✗ mvn package -DskipTests ➜ java git:(main) ✗ cd core/target ➜ target git:(main) ✗ unzip orc-core-1.9.0-SNAPSHOT-nohive.jar ➜ target git:(main) ✗ find . | grep -i "google/protobuf/" ∅ ``` Interestingly enough, the normal jar doesn't contain these proto files: ``` ➜ java git:(main) ✗ rm -rf core/target/ ➜ java git:(main) ✗ mvn package -DskipTests ➜ java git:(main) ✗ cd core/target ➜ target git:(main) unzip orc-core-1.9.0-SNAPSHOT.jar| grep -i proto ➜ target git:(main) find . | grep -i "google/protobuf/" ∅ ```

dongjoon-hyun

Thank you for making a PR, @Fokko .
According to the attached Trino PR, this will be Apache ORC 1.8.2 because Aapche Iceberg 1.1.0 is using Apache ORC 1.8.0.

Fokko · 2022-12-06T18:07:54Z

Thanks @dongjoon-hyun that would be great! 🙏🏻

dongjoon-hyun · 2022-12-06T18:13:49Z

BTW, @Fokko . We recently released Apache ORC 1.8.1. So, 1.8.2 is scheduled on March 03, 2023 (https://github.com/apache/orc/milestone/15). Is it okay?

dongjoon-hyun

+1, LGTM.

### Why are the changes needed? We're seeing some conflicts in the proto files when updating Iceberg in Trino: trinodb/trino#15079 (comment) I think we should exclude the proto files from the `nohive` jar. Before the change: ``` ➜ java git:(main) ✗ rm -rf core/target/ ➜ java git:(main) ✗ mvn package -DskipTests ➜ java git:(main) ✗ cd core/target ➜ target git:(main) ✗ unzip orc-core-1.9.0-SNAPSHOT-nohive.jar ➜ target git:(main) find . | grep -i "google/protobuf/" ./google/protobuf/timestamp.proto ./google/protobuf/field_mask.proto ./google/protobuf/api.proto ./google/protobuf/duration.proto ./google/protobuf/struct.proto ./google/protobuf/wrappers.proto ./google/protobuf/source_context.proto ./google/protobuf/any.proto ./google/protobuf/type.proto ./google/protobuf/empty.proto ./google/protobuf/compiler ./google/protobuf/compiler/plugin.proto ./google/protobuf/descriptor.proto ``` After the change: ``` ➜ java git:(main) ✗ rm -rf core/target/ ➜ java git:(main) ✗ mvn package -DskipTests ➜ java git:(main) ✗ cd core/target ➜ target git:(main) ✗ unzip orc-core-1.9.0-SNAPSHOT-nohive.jar ➜ target git:(main) ✗ find . | grep -i "google/protobuf/" ∅ ``` Interestingly enough, the normal jar doesn't contain these proto files: ``` ➜ java git:(main) ✗ rm -rf core/target/ ➜ java git:(main) ✗ mvn package -DskipTests ➜ java git:(main) ✗ cd core/target ➜ target git:(main) unzip orc-core-1.9.0-SNAPSHOT.jar| grep -i proto ➜ target git:(main) find . | grep -i "google/protobuf/" ∅ ``` ### What changes were proposed in this pull request? Exclude the proto files ### How was this patch tested? Inspecting the produced jar Closes #1333 Closes #1334 from Fokko/fd-exclude-proto-files. Authored-by: Fokko Driesprong <fokko@tabular.io> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit f6fee07) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

Fokko · 2022-12-06T18:16:19Z

@dongjoon-hyun Thanks. It depends, if Trino is fine with excluding the proto files then March next year would be perfectly fine. Otherwise, it would be great to fast-track it if possible.

dongjoon-hyun · 2022-12-06T18:16:30Z

Oh, @Fokko , what is ORC-1333? It's not Apache ORC JIRA ID.

Fokko · 2022-12-06T18:17:40Z

@dongjoon-hyun it is the ID of the GitHub Issue: #1333

dongjoon-hyun · 2022-12-06T18:18:14Z

My bad. Let me revert this. Unfortunately, it's not Apache ORC way, @Fokko .

dongjoon-hyun · 2022-12-06T18:19:09Z

It's reverted from main/branch-1.8.

dongjoon-hyun · 2022-12-06T18:22:40Z

I'm working on ORC JIRA and recommitting, @Fokko . There is nothing to do from your side~

We're seeing some conflicts in the proto files when updating Iceberg in Trino: trinodb/trino#15079 (comment) I think we should exclude the proto files from the `nohive` jar. Before the change: ``` ➜ java git:(main) ✗ rm -rf core/target/ ➜ java git:(main) ✗ mvn package -DskipTests ➜ java git:(main) ✗ cd core/target ➜ target git:(main) ✗ unzip orc-core-1.9.0-SNAPSHOT-nohive.jar ➜ target git:(main) find . | grep -i "google/protobuf/" ./google/protobuf/timestamp.proto ./google/protobuf/field_mask.proto ./google/protobuf/api.proto ./google/protobuf/duration.proto ./google/protobuf/struct.proto ./google/protobuf/wrappers.proto ./google/protobuf/source_context.proto ./google/protobuf/any.proto ./google/protobuf/type.proto ./google/protobuf/empty.proto ./google/protobuf/compiler ./google/protobuf/compiler/plugin.proto ./google/protobuf/descriptor.proto ``` After the change: ``` ➜ java git:(main) ✗ rm -rf core/target/ ➜ java git:(main) ✗ mvn package -DskipTests ➜ java git:(main) ✗ cd core/target ➜ target git:(main) ✗ unzip orc-core-1.9.0-SNAPSHOT-nohive.jar ➜ target git:(main) ✗ find . | grep -i "google/protobuf/" ∅ ``` Interestingly enough, the normal jar doesn't contain these proto files: ``` ➜ java git:(main) ✗ rm -rf core/target/ ➜ java git:(main) ✗ mvn package -DskipTests ➜ java git:(main) ✗ cd core/target ➜ target git:(main) unzip orc-core-1.9.0-SNAPSHOT.jar| grep -i proto ➜ target git:(main) find . | grep -i "google/protobuf/" ∅ ``` Exclude the proto files Inspecting the produced jar Closes #1333 Closes #1334 from Fokko/fd-exclude-proto-files. Authored-by: Fokko Driesprong <fokko@tabular.io> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit f6fee07) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

We're seeing some conflicts in the proto files when updating Iceberg in Trino: trinodb/trino#15079 (comment) I think we should exclude the proto files from the `nohive` jar. Before the change: ``` ➜ java git:(main) ✗ rm -rf core/target/ ➜ java git:(main) ✗ mvn package -DskipTests ➜ java git:(main) ✗ cd core/target ➜ target git:(main) ✗ unzip orc-core-1.9.0-SNAPSHOT-nohive.jar ➜ target git:(main) find . | grep -i "google/protobuf/" ./google/protobuf/timestamp.proto ./google/protobuf/field_mask.proto ./google/protobuf/api.proto ./google/protobuf/duration.proto ./google/protobuf/struct.proto ./google/protobuf/wrappers.proto ./google/protobuf/source_context.proto ./google/protobuf/any.proto ./google/protobuf/type.proto ./google/protobuf/empty.proto ./google/protobuf/compiler ./google/protobuf/compiler/plugin.proto ./google/protobuf/descriptor.proto ``` After the change: ``` ➜ java git:(main) ✗ rm -rf core/target/ ➜ java git:(main) ✗ mvn package -DskipTests ➜ java git:(main) ✗ cd core/target ➜ target git:(main) ✗ unzip orc-core-1.9.0-SNAPSHOT-nohive.jar ➜ target git:(main) ✗ find . | grep -i "google/protobuf/" ∅ ``` Interestingly enough, the normal jar doesn't contain these proto files: ``` ➜ java git:(main) ✗ rm -rf core/target/ ➜ java git:(main) ✗ mvn package -DskipTests ➜ java git:(main) ✗ cd core/target ➜ target git:(main) unzip orc-core-1.9.0-SNAPSHOT.jar| grep -i proto ➜ target git:(main) find . | grep -i "google/protobuf/" ∅ ``` Exclude the proto files Inspecting the produced jar Closes #1333 Closes #1334 from Fokko/fd-exclude-proto-files. Authored-by: Fokko Driesprong <fokko@tabular.io> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit f6fee07) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 5ea8c06) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

Fokko · 2022-12-06T18:27:13Z

@dongjoon-hyun Ah, race condition. I created a new PR: #1335

dongjoon-hyun · 2022-12-06T18:29:07Z

Oh, @Fokko . I was handling it by myself as I mentioned #1334 (comment)

5ea8c06 (master)
7bfdaaf (branch-1.8)

### Why are the changes needed? It looks like the shaded-protobuf.jar has the same issue that was fixed in #1334 ### What changes were proposed in this pull request? Exclude the proto files ### How was this patch tested? Test manually. Closes #1336 from nastra/orc-1327. Authored-by: Eduard Tudenhoefner <etudenhoefner@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

### Why are the changes needed? It looks like the shaded-protobuf.jar has the same issue that was fixed in #1334 ### What changes were proposed in this pull request? Exclude the proto files ### How was this patch tested? Test manually. Closes #1336 from nastra/orc-1327. Authored-by: Eduard Tudenhoefner <etudenhoefner@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 72deb94) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

### Why are the changes needed? We're seeing some conflicts in the proto files when updating Iceberg in Trino: trinodb/trino#15079 (comment) I think we should exclude the proto files from the `nohive` jar. Before the change: ``` ➜ java git:(main) ✗ rm -rf core/target/ ➜ java git:(main) ✗ mvn package -DskipTests ➜ java git:(main) ✗ cd core/target ➜ target git:(main) ✗ unzip orc-core-1.9.0-SNAPSHOT-nohive.jar ➜ target git:(main) find . | grep -i "google/protobuf/" ./google/protobuf/timestamp.proto ./google/protobuf/field_mask.proto ./google/protobuf/api.proto ./google/protobuf/duration.proto ./google/protobuf/struct.proto ./google/protobuf/wrappers.proto ./google/protobuf/source_context.proto ./google/protobuf/any.proto ./google/protobuf/type.proto ./google/protobuf/empty.proto ./google/protobuf/compiler ./google/protobuf/compiler/plugin.proto ./google/protobuf/descriptor.proto ``` After the change: ``` ➜ java git:(main) ✗ rm -rf core/target/ ➜ java git:(main) ✗ mvn package -DskipTests ➜ java git:(main) ✗ cd core/target ➜ target git:(main) ✗ unzip orc-core-1.9.0-SNAPSHOT-nohive.jar ➜ target git:(main) ✗ find . | grep -i "google/protobuf/" ∅ ``` Interestingly enough, the normal jar doesn't contain these proto files: ``` ➜ java git:(main) ✗ rm -rf core/target/ ➜ java git:(main) ✗ mvn package -DskipTests ➜ java git:(main) ✗ cd core/target ➜ target git:(main) unzip orc-core-1.9.0-SNAPSHOT.jar| grep -i proto ➜ target git:(main) find . | grep -i "google/protobuf/" ∅ ``` ### What changes were proposed in this pull request? Exclude the proto files ### How was this patch tested? Inspecting the produced jar Closes apache#1333 Closes apache#1334 from Fokko/fd-exclude-proto-files. Authored-by: Fokko Driesprong <fokko@tabular.io> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

We're seeing some conflicts in the proto files when updating Iceberg in Trino: trinodb/trino#15079 (comment) I think we should exclude the proto files from the `nohive` jar. Before the change: ``` ➜ java git:(main) ✗ rm -rf core/target/ ➜ java git:(main) ✗ mvn package -DskipTests ➜ java git:(main) ✗ cd core/target ➜ target git:(main) ✗ unzip orc-core-1.9.0-SNAPSHOT-nohive.jar ➜ target git:(main) find . | grep -i "google/protobuf/" ./google/protobuf/timestamp.proto ./google/protobuf/field_mask.proto ./google/protobuf/api.proto ./google/protobuf/duration.proto ./google/protobuf/struct.proto ./google/protobuf/wrappers.proto ./google/protobuf/source_context.proto ./google/protobuf/any.proto ./google/protobuf/type.proto ./google/protobuf/empty.proto ./google/protobuf/compiler ./google/protobuf/compiler/plugin.proto ./google/protobuf/descriptor.proto ``` After the change: ``` ➜ java git:(main) ✗ rm -rf core/target/ ➜ java git:(main) ✗ mvn package -DskipTests ➜ java git:(main) ✗ cd core/target ➜ target git:(main) ✗ unzip orc-core-1.9.0-SNAPSHOT-nohive.jar ➜ target git:(main) ✗ find . | grep -i "google/protobuf/" ∅ ``` Interestingly enough, the normal jar doesn't contain these proto files: ``` ➜ java git:(main) ✗ rm -rf core/target/ ➜ java git:(main) ✗ mvn package -DskipTests ➜ java git:(main) ✗ cd core/target ➜ target git:(main) unzip orc-core-1.9.0-SNAPSHOT.jar| grep -i proto ➜ target git:(main) find . | grep -i "google/protobuf/" ∅ ``` Exclude the proto files Inspecting the produced jar Closes apache#1333 Closes apache#1334 from Fokko/fd-exclude-proto-files. Authored-by: Fokko Driesprong <fokko@tabular.io> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit f6fee07) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

### Why are the changes needed? It looks like the shaded-protobuf.jar has the same issue that was fixed in apache#1334 ### What changes were proposed in this pull request? Exclude the proto files ### How was this patch tested? Test manually. Closes apache#1336 from nastra/orc-1327. Authored-by: Eduard Tudenhoefner <etudenhoefner@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

github-actions bot added BUILD JAVA labels Dec 6, 2022

dongjoon-hyun added this to the 1.8.2 milestone Dec 6, 2022

dongjoon-hyun reviewed Dec 6, 2022

View reviewed changes

Fokko mentioned this pull request Dec 6, 2022

Update to Iceberg 1.1.0 trinodb/trino#15079

Merged

dongjoon-hyun approved these changes Dec 6, 2022

View reviewed changes

dongjoon-hyun closed this in f6fee07 Dec 6, 2022

Fokko deleted the fd-exclude-proto-files branch December 6, 2022 18:15

dongjoon-hyun changed the title ~~ORC-1333: Exclude the proto files from the nohive jar~~ ORC-1327: Exclude the proto files from the nohive jar Dec 6, 2022

nastra mentioned this pull request Dec 7, 2022

ORC-1328: Exclude the proto files from the shaded protobuf jar #1336

Closed

Fokko mentioned this pull request Dec 12, 2022

Remove redundant protobuf exclusions trinodb/trino#15371

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ORC-1327: Exclude the proto files from the nohive jar #1334

ORC-1327: Exclude the proto files from the nohive jar #1334

Fokko commented Dec 6, 2022

dongjoon-hyun left a comment •

edited

Fokko commented Dec 6, 2022

dongjoon-hyun commented Dec 6, 2022

dongjoon-hyun left a comment

Fokko commented Dec 6, 2022

dongjoon-hyun commented Dec 6, 2022

Fokko commented Dec 6, 2022

dongjoon-hyun commented Dec 6, 2022

dongjoon-hyun commented Dec 6, 2022

dongjoon-hyun commented Dec 6, 2022

Fokko commented Dec 6, 2022

dongjoon-hyun commented Dec 6, 2022

ORC-1327: Exclude the proto files from the nohive jar #1334

ORC-1327: Exclude the proto files from the nohive jar #1334

Conversation

Fokko commented Dec 6, 2022

Why are the changes needed?

What changes were proposed in this pull request?

How was this patch tested?

dongjoon-hyun left a comment • edited

Choose a reason for hiding this comment

Fokko commented Dec 6, 2022

dongjoon-hyun commented Dec 6, 2022

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Fokko commented Dec 6, 2022

dongjoon-hyun commented Dec 6, 2022

Fokko commented Dec 6, 2022

dongjoon-hyun commented Dec 6, 2022

dongjoon-hyun commented Dec 6, 2022

dongjoon-hyun commented Dec 6, 2022

Fokko commented Dec 6, 2022

dongjoon-hyun commented Dec 6, 2022

dongjoon-hyun left a comment •

edited