Upgrade Hive from 2.3.7 to 3.1.2 #126
Conversation
This commit produces a version of HiveRunner that works with Hive 3.0.0 All tests run and pass, with the sole exception of PreV200HiveShellHiveCliEmulationTest. This commit is a starting point for 3.0.0 support, but it takes a pretty blunt approach in updating HiveRunner. I've only made those changes strictly necessary to compile and pass tests on the updated Hive version, and I did so without tons of context on Hive 3 itself. There might be changes needed, or new Hive3 features that should be tested, or... Changes made include: - update pom dependencies for newer Hive/Hadoop versions - alter packaging to create a shaded uber-jar (Hadoop2 and its transitive deps are still widely used, and lag behind Hadoop3's transitive deps quite a bit. Shading our use of Hadoop3 and some deps eases integration with other projects. - alter 'HiveConf' initialization to account for Metastore configuration changes in Hive 3
* Fix ClassCastExceptions for Date and Timestamp in Hive3 * Set hive.in.test=true by default to avoid warmup sql exceptions in hive 3; use log rate limiting for datanucleus * Delete outdated Hive CLI test Co-authored-by: Marton Bod <mbod@cloudera.com>
|
Fixes #120 |
| conf.setVar(var, newFolder(basedir, folder).toAbsolutePath().toString()); | ||
| } | ||
|
|
||
| protected void configureMrExecutionEngine(HiveConf conf) { |
There was a problem hiding this comment.
this might not be necessary anymore since Hive3 dropped M/R support. But not a blocker happy to merge this in for now.
There was a problem hiding this comment.
Yeah, good point, I think we leave it for now, we could have a follow on PR that goes through the code and removes everything to do with MR as there's probably more than just this.
src/test/java/com/klarna/hiverunner/MultipleExecutionEnginesTest.java
Outdated
Show resolved
Hide resolved
| <plugin> | ||
| <groupId>org.apache.maven.plugins</groupId> | ||
| <artifactId>maven-shade-plugin</artifactId> | ||
| <version>3.1.1</version> |
There was a problem hiding this comment.
The most recent version is 3.2.4, any specific reason to use 3.1.1?
There was a problem hiding this comment.
This was probably just the most recent version when the original change was made. There are quite a few other plugins that probably have older versions. I'd rather leave it like this as we know it works, we can always have a separate PR to update various versions later.
Co-authored-by: Nicola Vitucci <nicola.vitucci@gmail.com>
|
I've merged this into master and will hopefully get a release out in the next few days. Thanks for all your contributions! |
This PR moves HiveRunner to use Hive 3.1.2 as the default supported Hive version. Hive 2 can still be used with HiveRunner 5.2.1. Going forward if we needed to do maintenance releases for Hive 2.x I'd suggest releasing these off a branch from 5.2.1 and keeping the version numbers in the 5.x range with version numbers >=6.0.0 for Hive 3.