HIVE-25996: Backport HIVE-25098 to branch-2.3 #3066

wangyum · 2022-03-02T07:04:19Z

What changes were proposed in this pull request?

Backport HIVE-25098 to branch-2.3.

Why are the changes needed?

Make the downstreams easy to upgrade their Thrift version.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Local test.

dongjoon-hyun

Just a question: What about branch-3.1 and branch-3.0? Are these patches released officially in any Apache Hive artifacts after vote?

github-actions · 2022-05-12T00:21:03Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the dev@hive.apache.org list if the patch is in need of reviews.

wangyum · 2022-06-17T14:04:59Z

cc @sunchao

pan3793 · 2022-06-17T15:51:24Z

Thanks to the work done by @wangyum, Hive 2.3 is widely used in the industry and adopted by many downstream projects such as Apache Spark. A security patch release is greatly appreciated.

bgmarsh · 2022-08-03T14:58:28Z

Are there still plans to get this into the Hive 2.3 branch?

pan3793 · 2023-07-16T14:36:02Z

cc @sunchao this is the first step of upgrading thrift to get rid of CVE (0.14.0+), as we are planning Hive 2.3.10, please reopen this PR.

sunchao · 2023-07-16T15:22:36Z

@wangyum could you rebase the PR?

wangyum · 2023-07-17T05:46:57Z

metastore/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php

This file removed by: 1945e2f

wangyum · 2023-07-17T05:47:15Z

metastore/src/gen/thrift/gen-php/metastore/Types.php

This file removed by: 1945e2f

pan3793 · 2023-07-20T13:37:29Z

There are around 30~40 tests failure in the latest branch-2.3, some of them are flaky.
36, 28, 34 test failures were reported by the latest three commits.

Jenkins reports 41 test failures for this PR, which seems reasonable.

pan3793

Verified thrift-generated files locally with

# install thrift compiler 0.14.1 at /home/chengpan/bin/thrift-0.14.1/compiler/cpp/cmake-build
# provide share/fb303/if/fb303.thrift
mvn -pl metastore -Pthriftif -Dthrift.home=/home/chengpan/bin/thrift-0.14.1/compiler/cpp/cmake-build compile -am

Without fb303.thrift the compile command failed with the following error message, is it expected?

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (generate-thrift-sources) on project hive-metastore: An Ant BuildException has occured: The following error occurred while executing this line:
[ERROR] /home/chengpan/Projects/hive-2.3/metastore/target/antrun/build-main.xml:15: exec returned: 1
[ERROR] around Ant part ...<for param="thrift.file">... @ 9:28 in /home/chengpan/Projects/hive-2.3/metastore/target/antrun/build-main.xml
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :hive-metastore

wangyum · 2023-07-24T13:05:30Z

@pan3793 It should throw Could not find include file share/fb303/if/fb303.thrift:

root@e8c54f097cdf:/opensource/HIVE-25996# mvn -pl metastore -Pthriftif -DskipTests=true package -Dthrift.home=/usr/local
[INFO] Scanning for projects...
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Hive Metastore 2.3.10-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @ hive-metastore ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-banned-dependencies) @ hive-metastore ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-property) @ hive-metastore ---
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (generate-thrift-sources) @ hive-metastore ---
[INFO] Executing tasks

main:
   [delete] Deleting directory /opensource/HIVE-25996/metastore/src/gen/thrift
    [mkdir] Created dir: /opensource/HIVE-25996/metastore/src/gen/thrift
     [echo] Generating Thrift code for /opensource/HIVE-25996/metastore/if/hive_metastore.thrift
     [exec] [ERROR:/opensource/HIVE-25996/metastore/if/hive_metastore.thrift:1033] (last token was 'fb303.FacebookService')
     [exec] Service "fb303.FacebookService" has not been defined.
     [exec] [WARNING:/opensource/HIVE-25996/metastore/if/hive_metastore.thrift:25] Could not find include file share/fb303/if/fb303.thrift
     [exec] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.199 s
[INFO] Finished at: 2023-07-24T12:59:31+00:00
[INFO] Final Memory: 33M/362M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (generate-thrift-sources) on project hive-metastore: An Ant BuildException has occured: The following error occurred while executing this line:
[ERROR] /opensource/HIVE-25996/metastore/target/antrun/build-main.xml:15: exec returned: 1
[ERROR] around Ant part ...<for param="thrift.file">... @ 9:28 in /opensource/HIVE-25996/metastore/target/antrun/build-main.xml
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

It works if add fb303.thrift to share/fb303/if/:

root@e8c54f097cdf:/opensource/HIVE-25996# mvn -pl metastore -Pthriftif -DskipTests=true package -Dthrift.home=/usr/local
[INFO] Scanning for projects...
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Hive Metastore 2.3.10-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @ hive-metastore ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-banned-dependencies) @ hive-metastore ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-property) @ hive-metastore ---
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (generate-thrift-sources) @ hive-metastore ---
[INFO] Executing tasks

main:
   [delete] Deleting directory /opensource/HIVE-25996/metastore/src/gen/thrift
    [mkdir] Created dir: /opensource/HIVE-25996/metastore/src/gen/thrift
     [echo] Generating Thrift code for /opensource/HIVE-25996/metastore/if/hive_metastore.thrift
[INFO] Executed tasks
[INFO] 
...

pan3793 · 2023-07-24T13:07:37Z

Yes, I fixed it by adding share/fb303/if/fb303.thrift, so why share/fb303/if/fb303.thrift is not included in Hive source?

sunchao

LGTM.

I manually verified the Thrift changes, and the CI results. There is one new test failure after this but it is unrelated.

This PR is important to fix a CVE issue in branch-2.3

sunchao · 2023-07-28T04:37:36Z

@wangyum could you resolve the conflict? I'll merge this after it.

wangyum · 2023-07-28T06:12:52Z

Thank you @sunchao. Conflict resolved.

sunchao · 2023-07-28T23:22:10Z

Thanks! merged to branch-2.3

This reverts commit 8be7466.

kgyrtkirk added the tests pending label Mar 2, 2022

wangyum mentioned this pull request Mar 2, 2022

[SPARK-37090][BUILD][3.2] Upgrade libthrift to 0.16.0 to avoid security vulnerabilities apache/spark#35646

Closed

kgyrtkirk added tests failed and removed tests pending labels Mar 2, 2022

dongjoon-hyun reviewed Mar 2, 2022

View reviewed changes

wangyum changed the title ~~HIVE-25996: Backport HIVE-21498 and HIVE-25098 to branch-2.3 to fix CVE-2020-13949~~ HIVE-25996: Backport HIVE-25098 to branch-2.3 Mar 12, 2022

github-actions bot added the stale label May 12, 2022

github-actions bot closed this May 19, 2022

sunchao reopened this Jul 16, 2023

kgyrtkirk added tests pending and removed tests failed labels Jul 16, 2023

kgyrtkirk added tests failed and removed tests pending labels Jul 16, 2023

github-actions bot removed the stale label Jul 17, 2023

wangyum force-pushed the HIVE-25996 branch from 96ac047 to 9e476a5 Compare July 17, 2023 04:38

kgyrtkirk added tests pending and removed tests failed labels Jul 17, 2023

wangyum commented Jul 17, 2023

View reviewed changes

metastore/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php Outdated

Copy link

Member Author

wangyum Jul 17, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file removed by: 1945e2f

wangyum commented Jul 17, 2023

View reviewed changes

metastore/src/gen/thrift/gen-php/metastore/Types.php Outdated

Copy link

Member Author

wangyum Jul 17, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file removed by: 1945e2f

kgyrtkirk added tests failed and removed tests pending labels Jul 17, 2023

pan3793 approved these changes Jul 24, 2023

View reviewed changes

sunchao approved these changes Jul 26, 2023

View reviewed changes

HIVE-25996: Backport HIVE-21498 and HIVE-25098 to fix CVE-2020-13949

5a3a492

wangyum force-pushed the HIVE-25996 branch from 9e476a5 to db4e38d Compare July 28, 2023 05:39

kgyrtkirk added tests pending tests failed and removed tests failed tests pending labels Jul 28, 2023

Rebase

157bbe4

wangyum force-pushed the HIVE-25996 branch from db4e38d to 157bbe4 Compare July 28, 2023 05:45

kgyrtkirk added tests pending tests failed and removed tests failed tests pending labels Jul 28, 2023

sunchao merged commit 8be7466 into apache:branch-2.3 Jul 28, 2023

wangyum deleted the HIVE-25996 branch July 29, 2023 01:02

charlso96 added a commit to charlso96/hive that referenced this pull request Sep 7, 2023

Revert "HIVE-25996: Backport HIVE-25098 to branch-2.3 (apache#3066)"

91d51af

This reverts commit 8be7466.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HIVE-25996: Backport HIVE-25098 to branch-2.3 #3066

HIVE-25996: Backport HIVE-25098 to branch-2.3 #3066

wangyum commented Mar 2, 2022 •

edited

Loading

dongjoon-hyun left a comment

github-actions bot commented May 12, 2022

wangyum commented Jun 17, 2022

pan3793 commented Jun 17, 2022

bgmarsh commented Aug 3, 2022

pan3793 commented Jul 16, 2023

sunchao commented Jul 16, 2023

wangyum Jul 17, 2023

wangyum Jul 17, 2023

pan3793 commented Jul 20, 2023 •

edited

Loading

pan3793 left a comment

wangyum commented Jul 24, 2023

pan3793 commented Jul 24, 2023 •

edited

Loading

sunchao left a comment •

edited

Loading

sunchao commented Jul 28, 2023

wangyum commented Jul 28, 2023

sunchao commented Jul 28, 2023

HIVE-25996: Backport HIVE-25098 to branch-2.3 #3066

HIVE-25996: Backport HIVE-25098 to branch-2.3 #3066

Conversation

wangyum commented Mar 2, 2022 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

dongjoon-hyun left a comment

Choose a reason for hiding this comment

github-actions bot commented May 12, 2022

wangyum commented Jun 17, 2022

pan3793 commented Jun 17, 2022

bgmarsh commented Aug 3, 2022

pan3793 commented Jul 16, 2023

sunchao commented Jul 16, 2023

wangyum Jul 17, 2023

Choose a reason for hiding this comment

wangyum Jul 17, 2023

Choose a reason for hiding this comment

pan3793 commented Jul 20, 2023 • edited Loading

pan3793 left a comment

Choose a reason for hiding this comment

wangyum commented Jul 24, 2023

pan3793 commented Jul 24, 2023 • edited Loading

sunchao left a comment • edited Loading

Choose a reason for hiding this comment

sunchao commented Jul 28, 2023

wangyum commented Jul 28, 2023

sunchao commented Jul 28, 2023

wangyum commented Mar 2, 2022 •

edited

Loading

pan3793 commented Jul 20, 2023 •

edited

Loading

pan3793 commented Jul 24, 2023 •

edited

Loading

sunchao left a comment •

edited

Loading