Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HIVE-26614: Fix adding custom jars in Job Classpath. #3652

Merged
merged 1 commit into from
Oct 11, 2022
Merged

HIVE-26614: Fix adding custom jars in Job Classpath. #3652

merged 1 commit into from
Oct 11, 2022

Conversation

ayushtkn
Copy link
Member

@ayushtkn ayushtkn commented Oct 8, 2022

What changes were proposed in this pull request?

Make the Jar path fully qualified, so as to prevent resolution to fs.defaultFs in any application

Why are the changes needed?

Some applications don't explicitly assume the path to LFS, rather resolve to fs.defaultFs

Does this PR introduce any user-facing change?

Nopes.

How was this patch tested?

Build distribution & tried the same command. Doesn't happen in Tests due to

hive/iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java

Lines 249 to 252 in 9db3110

if (!jobConf.getBoolean(HiveConf.ConfVars.HIVE_IN_TEST_IDE.varname, false)) {
// For running unit test this won't work as maven surefire CP is different than what we have on a cluster:
// it places the current projects' classes and test-classes to top instead of jars made from these...
Utilities.addDependencyJars(jobConf, HiveIcebergStorageHandler.class);

May be specific to query engines as well. Tez I suppose alway expects the Path to be in LFS

@sonarcloud
Copy link

sonarcloud bot commented Oct 8, 2022

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug C 6 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot E 1 Security Hotspot
Code Smell A 68 Code Smells

No Coverage information No Coverage information
No Duplication information No Duplication information

deniskuzZ
deniskuzZ approved these changes Oct 11, 2022
View reviewed changes
Copy link
Member

@deniskuzZ deniskuzZ left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM +1

@deniskuzZ deniskuzZ merged commit 57d1ad6 into apache:master Oct 11, 2022
DongWei-4 pushed a commit to DongWei-4/hive that referenced this pull request Oct 28, 2022
HIVE-26103: Port Iceberg fixes to the iceberg module (apache#3164)
6e25a01 
* Source Iceberg PR - Core: Remove deprecated APIs up to 0.13.0

* Revert "HIVE-25563: Iceberg table operations hang a long time if metadata is missing/corrupted (Adam Szita, reviewed by Marton Bod)" - applying instead  Hive: Limit number of retries when metadata file is missing (apache#3379)

This reverts commit 7b600fe.

* Source Iceberg PR - Hive: Limit number of retries when metadata file is missing (apache#3379)

* Source Iceberg PR - Hive: Fix RetryingMetaStoreClient for Hive 2.1 (apache#3403)

* Source Iceberg PR - Switch from new HashMap to Maps.newHashMap (apache#3648)

* Source Iceberg PR - Hive: HiveCatalog should remove HMS stats for certain engines based on config (apache#3652) - Use the Iceberg config property

* Source Iceberg PR - Core: If status check fails, commit should be unknown (apache#3717)

* Source Iceberg PR - Build: Add checkstyle rule for instantiating HashMap, HashSet, ArrayList (apache#3689)

* Source Iceberg PR - Test: Make sure to delete temp folders (apache#3790)

* Source Iceberg PR - API: Register existing tables in Iceberg HiveCatalog (apache#3851)

* Source Iceberg PR - Hive: Make Iceberg table filter optional in HiveCatalog (apache#3908)

* Source Iceberg PR - Core: Add reserved UUID Table Property and Expose in HMS. (apache#3914)

* Source Iceberg PR - Hive: Known exception should not become CommitStateUnknownException (apache#4261)

* Source Iceberg PR - Build: Add missing @OverRide annotations (apache#3654)
DongWei-4 pushed a commit to DongWei-4/hive that referenced this pull request Oct 28, 2022
@DongWei-4
HIVE-26103: Port Iceberg fixes to the iceberg module (apache#3164)
fd62acc 
* Source Iceberg PR - Core: Remove deprecated APIs up to 0.13.0

* Revert "HIVE-25563: Iceberg table operations hang a long time if metadata is missing/corrupted (Adam Szita, reviewed by Marton Bod)" - applying instead  Hive: Limit number of retries when metadata file is missing (apache#3379)

This reverts commit 7b600fe.

* Source Iceberg PR - Hive: Limit number of retries when metadata file is missing (apache#3379)

* Source Iceberg PR - Hive: Fix RetryingMetaStoreClient for Hive 2.1 (apache#3403)

* Source Iceberg PR - Switch from new HashMap to Maps.newHashMap (apache#3648)

* Source Iceberg PR - Hive: HiveCatalog should remove HMS stats for certain engines based on config (apache#3652) - Use the Iceberg config property

* Source Iceberg PR - Core: If status check fails, commit should be unknown (apache#3717)

* Source Iceberg PR - Build: Add checkstyle rule for instantiating HashMap, HashSet, ArrayList (apache#3689)

* Source Iceberg PR - Test: Make sure to delete temp folders (apache#3790)

* Source Iceberg PR - API: Register existing tables in Iceberg HiveCatalog (apache#3851)

* Source Iceberg PR - Hive: Make Iceberg table filter optional in HiveCatalog (apache#3908)

* Source Iceberg PR - Core: Add reserved UUID Table Property and Expose in HMS. (apache#3914)

* Source Iceberg PR - Hive: Known exception should not become CommitStateUnknownException (apache#4261)

* Source Iceberg PR - Build: Add missing @OverRide annotations (apache#3654)
DongWei-4 pushed a commit to DongWei-4/hive that referenced this pull request Oct 28, 2022
@ayushtkn @DongWei-4
HIVE-26614: Fix adding custom jars in Job classpath (Ayush Saxena , r…
8021e4a 
…eviewed by Denys Kuzmenko)

Closes apache#3652
dengzhhu653 pushed a commit to dengzhhu653/hive that referenced this pull request Dec 15, 2022
@dengzhhu653
HIVE-26103: Port Iceberg fixes to the iceberg module (apache#3164)
56558c1 
* Source Iceberg PR - Core: Remove deprecated APIs up to 0.13.0

* Revert "HIVE-25563: Iceberg table operations hang a long time if metadata is missing/corrupted (Adam Szita, reviewed by Marton Bod)" - applying instead  Hive: Limit number of retries when metadata file is missing (apache#3379)

This reverts commit 7b600fe.

* Source Iceberg PR - Hive: Limit number of retries when metadata file is missing (apache#3379)

* Source Iceberg PR - Hive: Fix RetryingMetaStoreClient for Hive 2.1 (apache#3403)

* Source Iceberg PR - Switch from new HashMap to Maps.newHashMap (apache#3648)

* Source Iceberg PR - Hive: HiveCatalog should remove HMS stats for certain engines based on config (apache#3652) - Use the Iceberg config property

* Source Iceberg PR - Core: If status check fails, commit should be unknown (apache#3717)

* Source Iceberg PR - Build: Add checkstyle rule for instantiating HashMap, HashSet, ArrayList (apache#3689)

* Source Iceberg PR - Test: Make sure to delete temp folders (apache#3790)

* Source Iceberg PR - API: Register existing tables in Iceberg HiveCatalog (apache#3851)

* Source Iceberg PR - Hive: Make Iceberg table filter optional in HiveCatalog (apache#3908)

* Source Iceberg PR - Core: Add reserved UUID Table Property and Expose in HMS. (apache#3914)

* Source Iceberg PR - Hive: Known exception should not become CommitStateUnknownException (apache#4261)

* Source Iceberg PR - Build: Add missing @OverRide annotations (apache#3654)
dengzhhu653 pushed a commit to dengzhhu653/hive that referenced this pull request Dec 15, 2022
@ayushtkn @dengzhhu653
HIVE-26614: Fix adding custom jars in Job classpath (Ayush Saxena , r…
fcfae5a 
…eviewed by Denys Kuzmenko)

Closes apache#3652
yeahyung pushed a commit to yeahyung/hive that referenced this pull request Jul 20, 2023
@ayushtkn @yeahyung
HIVE-26614: Fix adding custom jars in Job classpath (Ayush Saxena , r…
f03920a 
…eviewed by Denys Kuzmenko)

Closes apache#3652
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@deniskuzZ deniskuzZ deniskuzZ approved these changes

Assignees
No one assigned
Labels
tests passed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@ayushtkn @deniskuzZ @kgyrtkirk