[HUDI-1721] run_sync_tool support hive3#2719
[HUDI-1721] run_sync_tool support hive3#2719MyLanPangzi wants to merge 1 commit intoapache:masterfrom MyLanPangzi:master
Conversation
run_sync_tool support hive3
| fi | ||
| HIVE_LIBFB303=`ls ${HIVE_HOME}/lib/libfb303-*.jar | tr '\n' ':'` | ||
| if [ -n "$HIVE_LIBFB303" ]; then | ||
| HIVE_JARS=$HIVE_JARS:$HIVE_LIBFB303 |
There was a problem hiding this comment.
Can you explain why we need these 2 kinds of jars ?
There was a problem hiding this comment.
I had the same problem, hive3 need these jars to be connected. The dependency packages of hive2 and hive3 on the connection have been changed.
There was a problem hiding this comment.
Is it possible to distinguish between Hive2 and Hive3 and make different library dependency ?
|
Sorry,I forgot comment the issue |
vinothchandar
left a comment
vinothchandar
left a comment
There was a problem hiding this comment.
@MyLanPangzi just want to confirm that this works on hive2 as well. We can then land if other have no concerns
I'll spend some time testing Hive2 with these vars at this week. |
|
@vinothchandar I test with hive2.3.8. it's not working. can we break down the one shell to hive2 or hive3 ? ls: cannot access /opt/module/apache-hive-2.3.8-bin/lib/calcite-*.jar: No such file or directory |
Maybe we can add a cmd config option to switch hive2 and hive3 jars loading. |
|
+1 can we make the script understand Hive 2 and Hive 3, but just have one script? |
|
i test with hive 2.3.8 and add follow shell. i got a runtime exception seems abnormal. HIVE_VERSION=$(hive --version|head -n 1|awk '{print $2}') if [[ $HIVE_VERSION =~ ^2 ]]; then Required table missing : "DBS" in Catalog "" Schema "". DataNucleus requires this table to perform its persistence operations. Either your MetaData is incorrect, or you need to enable "datanucleus.schema.autoCreateTables" 2021-05-02 06:39:10,597 ERROR [main] metastore.RetryingHMSHandler (RetryingHMSHandler.java:(82)) - HMSHandler Fatal error: MetaException(message:Version information not found in metastore. ) 2021-05-02 06:39:10,597 WARN [main] metadata.Hive (Hive.java:registerAllFunctionsOnce(234)) - Failed to register all functions. |
|
any reason why you closed this @MyLanPangzi ? |
|
@vinothchandar i think the hive dependencies too many to control... ALTER TABLE dwd_sale_sale_detail_rt ADD |
5 participants
run_sync_tool support hive3
Tips
What is the purpose of the pull request
run_sync_tool support hive3
Brief change log
Verify this pull request
(Please pick either of the following options)
This pull request is a trivial rework / code cleanup without any test coverage.
This change added tests and can be verified as follows:
(example:)
Committer checklist
Has a corresponding JIRA in PR title & commit
Commit message is descriptive of the change
CI is green
Necessary doc changes done or have another open PR
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.