Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DRILL-6349: Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM #1446

Closed
wants to merge 1 commit into from

Conversation

oleg-zinovev
Copy link
Contributor

PR allows both build and run with JDK8, JDK 10 (and, likely, JDK9).
All tests, except HBase, Hive, Kafka Storage Plugin tests, works on JDK10:

  • HBase cannot start master: HMaster ctor fails with message "Unexpected version format: 10.0.2"
  • Hive cannot create HiveMetaStoreClient: ctor fails with "java.base/[Ljava.lang.Object; cannot be cast to java.base/[Ljava.net.URI;"
  • Kafka: KafkaFilterPushdownTest fails with errors "java.lang.NoSuchMethodError: sun.nio.ch.DirectBuffer.cleaner()Lsun/misc/Cleaner;"

Changes:

  • Added a DrillPlatformDependent class, which tries to read maxDirectMemory from jdk.internal.misc.VM, otherwise fallbacks to netty PlatformDependent
  • asm dependency updated to 6.2.1, and ReplacingInterpreter fixed
  • fixed List.toArray() call in FileSystemPartitionDescriptor (in JDK10 this method return Object[] and cast fails)
  • surefire plugin updated to 2.21.0
  • surefire configuration changed:
    1. added -XX:+IgnoreUnrecognizedVMOptions
    2. added java.se module (mostly, for java.sql module)
    3. added -Djdk.attach.allowAttachSelf=true, required by jmockit
    4. added locale and country settings, because format tests fails with my system locale
  • compiler plugin updated to 3.8.0
  • JarBuilder fixed for JDK10 (target 1.5 and source 1.5 not supported by javac 10)
  • Drill2489CallsAfterCloseThrowExceptionsTest.ThrowsClosedBulkChecker skips new methods in JDK10 (JDK9 ?) jdbc api
  • added jaxb-api and javax.activation dependencies, because javax.xml.bind and javax.activation modules will be removed in JDK11 (javax.activation used by jersey)
  • drill-config.sh and sqlline.bat changed:
    1. added -XX:+IgnoreUnrecognizedVMOptions
    2. added --add-modules java.se (mostly, for java.sql module)
    3. added --add-opens java.base/jdk.internal.misc=ALL-UNNAMED (allows access to jdk.internal.misc.VM)

P.S. I am sorry for possible mistakes because of my bad English

@arina-ielchiieva
Copy link
Member

arina-ielchiieva commented Aug 27, 2018

@oleg-zinovev can you verify for 1.9 as well? Why failed tests cannot be fixed?
Also please provide proper PR and commit messages.

@@ -69,6 +69,8 @@ public String build(String jarName, String buildDirectory, String includeFiles,
List<String> params = new LinkedList<>();
params.add("clean");
params.add("package");
params.add("-Dmaven.compiler.source=1.6");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use 1.8

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, by default was a 1.5

}


private static long maxDirectMemory0() {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why method is named with 0 in the end?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a copy-paste from netty PlatformDependent.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@oleg-zinovev

  1. Please add javadoc for this class explaining its purpose.
  2. I still suggest to rename the method.
  3. Maybe add trace for the ignored exception, just in case we might need to find out what has happened.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is copy/paste necessary (why not to use netty PlatformDependent directly)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

netty PlatformDependent do not try to access jdk.internal.misc.VM.
By "copy-paste" i mean only a method name

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If netty PlatformDependent uses wrong max direct memory, will not netty memory allocation be affected that may cause OOM in Drill?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like it can cause OOM:
In Drill version of PooledByteBufAllocatorL exists a reflective access to PooledByteBufAllocator.directArenas field.
It's size depends on
DEFAULT_NUM_DIRECT_ARENA = Math.max(0, SystemPropertyUtil.getInt( "io.netty.allocator.numDirectArenas", (int) Math.min( defaultMinNumArena, PlatformDependent.maxDirectMemory() / defaultChunkSize / 2 / 3)));

Should i use a netty PlatformDependent, or "override" it (as PooledByteBufAllocatorL) ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Drill delegates to netty all large memory allocation and netty internally tracks how much direct memory is available. If netty incorrectly determines max direct memory size or Drill and netty max direct memory sizes are different, it is easy to get into a situation where netty will fail allocation when memory is available or it will try to allocate when memory is not available.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

netty PlatformDependent and jdk.internal.misc.VM returns same value for maxDirectMemory, so i remove my custom class

@oleg-zinovev
Copy link
Contributor Author

Can you verify for 1.9 as well?
1.9 not supported since March 2018 (http://www.oracle.com/technetwork/java/javase/eol-135779.html). I can try to verify on my machine, but it takes about a hour.

Why failed tests cannot be fixed?
I don't know version of kafka, hbase and hive that supports java 9+. I can just try to update those dependencies.

@arina-ielchiieva
Copy link
Member

@oleg-zinovev did you verify for open-jdk?

@oleg-zinovev
Copy link
Contributor Author

@oleg-zinovev did you verify for open-jdk?
I verify only for OpenJdk.

@oleg-zinovev oleg-zinovev changed the title DRILL-6349 DRILL-6349: Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM Aug 27, 2018
@oleg-zinovev
Copy link
Contributor Author

oleg-zinovev commented Aug 27, 2018

At least Kafka and HBase tests can not be fixed under Java 9+
HBase: https://hbase.apache.org/book.html#java - Java 9+ not supported yet
Kafka: https://kafka.apache.org/documentation/#java - only Java 8 is mentioned

@oleg-zinovev
Copy link
Contributor Author

oleg-zinovev commented Aug 28, 2018

build works with Java 9, except hive and kafka tests (HBase tests surprisely works)

@arina-ielchiieva
Copy link
Member

@oleg-zinovev besides comment in the PR I suggest a couple more things:

  1. File a Jira for failed tests (indicating what and why fails - basically copy info from the PR description + indicate affected JDKs).
  2. As I understand the purpose of this PR is allow build on JDK 9 and 10. Maybe it's better to rename commit message to highlight this. Description of the changes (which was really useful during the review, thanks for that!) can be included the commit message as well.

@@ -215,6 +215,12 @@ SET JAVA_CMD=%JAVA_HOME%\bin\%JAVA_EXE%
if "%JAVA_HOME%" == "" (set JAVA_CMD=%JAVA_EXE%)
set ERROR_CODE=0

for /f tokens^=2-5^ delims^=.-_^" %%j in ('"%JAVA_CMD%" -fullversion 2^>^&1') do set "version=%%j%%k"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LF/CR?

@@ -69,6 +69,8 @@ public String build(String jarName, String buildDirectory, String includeFiles,
List<String> params = new LinkedList<>();
params.add("clean");
params.add("package");
params.add("-Dmaven.compiler.source=1.8");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use pom.xml

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

@@ -179,7 +179,7 @@ protected void createPartitionSublists() {

// build a list of DFSDirPartitionLocation.
for (final List<String> dirs : dirToFileMap.keySet()) {
locations.add( new DFSDirPartitionLocation((String [])dirs.toArray(), dirToFileMap.get(dirs)));
locations.add( new DFSDirPartitionLocation(dirs.toArray(new String [0]), dirToFileMap.get(dirs)));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use dirs.size() instead of 0.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the link. There is a difference in behavior of ArrayList and Array.ArrayList that is used in this method. In any case, if you touch this method, you will need to fix it, as there is no reason to create ArrayList, so I'd suggest to file a separate JIRA and fix it in a separate PR.

pom.xml Outdated
-XX:+IgnoreUnrecognizedVMOptions
"--add-modules java.se"
-Djdk.attach.allowAttachSelf=true
-Duser.language=en
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this necessary?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-Djdk.attach.allowAttachSelf=true - for jmockit (more info: https://bugs.java.com/view_bug.do?bug_id=8180425)
-Duser.language=en - format tests fails on non-english system locale
-XX:+IgnoreUnrecognizedVMOptions "--add-modules java.se" - revalidate it tomorrow

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

http://openjdk.java.net/jeps/261 - java.se module available by default for unnamed modules, so -XX:+IgnoreUnrecognizedVMOptions "--add-modules java.se" can be removed

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@oleg-zinovev, since this is another bug, please log a Jira and remove changes connected with locale from this PR.

@oleg-zinovev oleg-zinovev force-pushed the DRILL-6349 branch 4 times, most recently from ecdf283 to 147e125 Compare August 30, 2018 08:58
<!-- <scope>test</scope> -->
</dependency>
<dependency>
<groupId>org.ow2.asm</groupId>
<artifactId>asm-util</artifactId>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see any new dependencies being introduced as part of the PR, why is this necessary? The same for all other newly introduced dependencies.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New asm version supports jdk 10, but there asm-debug-all jar was removed (see https://gitlab.ow2.org/asm/asm/commit/4c449bea1e3fb980238c61d60103fe1be76e77e5), so new dependencies, where classes used by Drill are available were introduced.

If I understand correctly, jaxb-api and activation dependencies were added due to http://openjdk.java.net/jeps/320.
@oleg-zinovev, Is it make sence to move these two dependencies to the root pom?

Also, please remove commented out lines introduced in previous commits and move version for asm dependencies to the property in the root pom.

@@ -180,7 +180,7 @@ protected void createPartitionSublists() {

// build a list of DFSDirPartitionLocation.
for (final List<String> dirs : dirToFileMap.keySet()) {
locations.add( new DFSDirPartitionLocation((String [])dirs.toArray(), dirToFileMap.get(dirs)));
locations.add( new DFSDirPartitionLocation(dirs.toArray(new String[dirs.size()]), dirToFileMap.get(dirs)));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this change necessary? As I already mentioned, I don't see why is it necessary to go back and forth between an array and a list in this method, so I would prefer not to introduce an unrelated change into the PR and handle it in a separate PR/JIRA.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://bugs.java.com/view_bug.do?bug_id=6260652
Result type of Arrays.ArrayList::toArray() was changed from T[] to Object[] in JDK 9+.
So, FileSystemPartitionDescriptor::createPartitionSublists() doesn't work if Drill running on JDK9+.

@@ -375,6 +375,11 @@ else if (NullPointerException.class == cause.getClass()

public void testAllMethods() {
for (Method method : jdbcIntf.getMethods()) {
if (method.isDefault()) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider moving to isOkayNonthrowingMethod(). What new default methods were introduced in java 10 that require special handling?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some method was added in JDBC 4.3 API
Additional information:
https://jcp.org/aboutJava/communityprocess/maintenance/jsr221/JDBC4.3MR-January2017.pdf
6. java.sql.Connection changes
8. java.sql.DatabaseMetaData changes
12. java.sql.Statement changes

pom.xml Outdated
@@ -734,6 +735,9 @@
-Djava.net.preferIPv4Stack=true
-Djava.awt.headless=true
-XX:+CMSClassUnloadingEnabled -ea
-Djdk.attach.allowAttachSelf=true
-Duser.language=en
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fixes unit tests but opens Drill for bugs on system where language or country are different from en/US.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This tests fails on non-english locale. Main reason is a decimal and thousand separator char.
https://drill.apache.org/docs/compiling-drill-from-source/ - I don't see any requirements about locale in documentaion. Maybe it must be an another Bug+PR ?

TestDurationFormat.testCompactSecMillis:58->validateDurationFormat:43 expected:<4[,]545s> but was:<4[.]545s>
TestDurationFormat.testCompactTwoDigitMilliSec:48->validateDurationFormat:43 expected:<0[,]045s> but was:<0[.]045s>
TestFunctionsQuery.testToCharFunction:534 » After matching 0 records, did not...
TestSelectivity.testFilterSelectivityOptions » UserRemote PARSE ERROR: Encount...
TestVarDecimalFunctions.testCastDecimalDouble:680->BaseTestQuery.testRunAndReturn:341 » Rpc
TestVarDecimalFunctions.testCastDoubleDecimal:650->BaseTestQuery.testRunAndReturn:341 » Rpc
TestVarDecimalFunctions.testDecimalToChar:731 » at position 0 column 's1' m...
TestTopNSchemaChanges.testMissingColumn:192 » org.apache.drill.common.excepti...
TestTopNSchemaChanges.testNumericTypes:82 » org.apache.drill.common.exception...
TestTopNSchemaChanges.testUnionTypes:162 » org.apache.drill.common.exceptions...
TestMergeJoinWithSchemaChanges.testNumericStringTypes:192->BaseTestQuery.testRunAndReturn:341 » Rpc
TestMergeJoinWithSchemaChanges.testNumericTypes:114->BaseTestQuery.testRunAndReturn:341 » Rpc
TestMergeJoinWithSchemaChanges.testOneSideSchemaChanges:348->BaseTestQuery.testRunAndReturn:341 » Rpc
TestExternalSort.testNumericTypesLegacy:49->testNumericTypes:113 » org.apache...
TestExternalSort.testNumericTypesManaged:44->testNumericTypes:113 » org.apach...
TestImageRecordReader.testAviImage:101->createAndQuery:50 » at position 0 col...
TestImageRecordReader.testBmpImage:56->createAndQuery:50 » at position 0 colu...
TestImageRecordReader.testEpsImage:121->createAndQuery:50 » at position 0 col...
TestImageRecordReader.testJpegImage:71->createAndQuery:50 » at position 0 col...
TestImageRecordReader.testMovImage:111->createAndQuery:50 » at position 0 col...
TestImageRecordReader.testPngImage:81->createAndQuery:50 » at position 0 colu...
TestImageRecordReader.testPsdImage:86->createAndQuery:50 » at position 0 colu...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@oleg-zinovev, yes, let's fix it in another PR. But currently to avoid unit tests failures you may change the locale on your machine.

@oleg-zinovev oleg-zinovev force-pushed the DRILL-6349 branch 3 times, most recently from 94c099e to 8faa57b Compare September 3, 2018 08:39
@dprutean
Copy link

I got the same issue with java 10. Please also make the jdbc-driver-all available to download without downloading the complete tar.gz. Would save much time.

java.lang.NoClassDefFoundError: Could not initialize class oadd.org.apache.drill.common.config.DrillConfig
at org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:159)
at org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:72)
at org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:68)
at oadd.org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)
at org.apache.drill.jdbc.Driver.connect(Driver.java:72)

Copy link
Member

@vvysotskyi vvysotskyi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@oleg-zinovev, could you please rebase onto the master and resolve conflicts?

pom.xml Outdated
@@ -734,6 +735,9 @@
-Djava.net.preferIPv4Stack=true
-Djava.awt.headless=true
-XX:+CMSClassUnloadingEnabled -ea
-Djdk.attach.allowAttachSelf=true
-Duser.language=en
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@oleg-zinovev, yes, let's fix it in another PR. But currently to avoid unit tests failures you may change the locale on your machine.

<!-- <scope>test</scope> -->
</dependency>
<dependency>
<groupId>org.ow2.asm</groupId>
<artifactId>asm-util</artifactId>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New asm version supports jdk 10, but there asm-debug-all jar was removed (see https://gitlab.ow2.org/asm/asm/commit/4c449bea1e3fb980238c61d60103fe1be76e77e5), so new dependencies, where classes used by Drill are available were introduced.

If I understand correctly, jaxb-api and activation dependencies were added due to http://openjdk.java.net/jeps/320.
@oleg-zinovev, Is it make sence to move these two dependencies to the root pom?

Also, please remove commented out lines introduced in previous commits and move version for asm dependencies to the property in the root pom.

@vvysotskyi
Copy link
Member

@dprutean, yes, it looks like the same issue, the stack trace looks similar, but I don't see sun/misc/VM entry there.

jdbc-driver-all is published to the maven central repository (https://mvnrepository.com/artifact/org.apache.drill.exec/drill-jdbc-all/1.14.0), so you can specify it in the pom file, or download using the link http://central.maven.org/maven2/org/apache/drill/exec/drill-jdbc-all/1.14.0/drill-jdbc-all-1.14.0.jar.

@oleg-zinovev
Copy link
Contributor Author

oleg-zinovev commented Nov 9, 2018

@vvysotskyi
Sorry, I cannot reproduce this issue on Windows...

  1. This method not calling during Drill startup
  2. Even if I try to add a storage plugin, it works just fine.

P.S. Time limit error for last build...

@oleg-zinovev
Copy link
Contributor Author

oleg-zinovev commented Nov 12, 2018

@vvysotskyi
I finally managed to reproduce the error. The winutils.exe (Hadoop WinUtils) requires a visual c ++ redistributable (msvcr100.dll).
There is 2 options:

  1. Require visual c ++ redistributable to be installed by user (via documentation?)
  2. Add msvcr100.dll for Windows x64 into winutils/bin

Copy link
Member

@vvysotskyi vvysotskyi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@oleg-zinovev, looks like the issue with winutils is caused by the difference in hadoop and hadoop-winutils versions.

Also, could you please hide warnings which are displayed after starting Drill in embedded mode for JDK 9+ for both Linux and Windows?

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil (file:/tmp/drill/distribution/target/apache-drill-1.15.0-SNAPSHOT/apache-drill-1.15.0-SNAPSHOT/jars/3rdparty/avatica-1.12.0.jar) to field java.nio.Buffer.address
WARNING: Please consider reporting this to the maintainers of org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

:end
endlocal
exit /B %ERROR_CODE%
@REM
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please revert changes here connected with replacing CRLF by LF.


:end
endlocal
exit /B %ERROR_CODE%
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please leave a new line here.

@oleg-zinovev
Copy link
Contributor Author

oleg-zinovev commented Nov 12, 2018

@vvysotskyi, 2.7.1 is the latest version of hadoop-winutils (in central.maven.org)

Also, could you please hide warnings which are displayed after starting Drill in embedded mode for JDK 9+ for both Linux and Windows?

According to http://mail.openjdk.java.net/pipermail/jigsaw-dev/2017-May/012673.html there is no official way to remove this warnings.

@oleg-zinovev oleg-zinovev force-pushed the DRILL-6349 branch 2 times, most recently from 67c055b to b0e996c Compare November 12, 2018 12:29
@oleg-zinovev
Copy link
Contributor Author

oleg-zinovev commented Nov 12, 2018

@vvysotskyi
I was able to remove warning on startup by adding --add-opens for all accessed modules. But there is not guarantee that this warning will not be shown later.

What about Visual C++ Runtime library?

@vvysotskyi
Copy link
Member

@oleg-zinovev, thanks for removing those warnings.

I'm seeing bug similar to the HBASE-18867:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:1.3.1:enforce (validate_java_and_maven_version) on project drill-root: Execution validate_java_and_maven_version of goal org.apache.maven.plugins:maven-enforcer-plugin:1.3.1:enforce failed: An API incompatibility was encountered while executing org.apache.maven.plugins:maven-enforcer-plugin:1.3.1:enforce: java.lang.ExceptionInInitializerError: null

And as it was described in that Jira it is fixed after updating maven-enforcer-plugin to 3.0.0-M1. Could you please update it in the PR?

@oleg-zinovev oleg-zinovev force-pushed the DRILL-6349 branch 2 times, most recently from d3b54f5 to d4fff1d Compare November 16, 2018 11:49
@oleg-zinovev
Copy link
Contributor Author

@vvysotskyi , done.

@vvysotskyi
Copy link
Member

@oleg-zinovev, thanks for making changes. Please resolve a problem with CRLF/LF in sqlline.bat

We will wait until hadoop-winutils 2.7.4 is published to the maven central, and then update its version before merging the PR.

@oleg-zinovev
Copy link
Contributor Author

@oleg-zinovev, thanks for making changes. Please resolve a problem with CRLF/LF in sqlline.bat

We will wait until hadoop-winutils 2.7.4 is published to the maven central, and then update its version before merging the PR.

@vvysotskyi
Why we can't just add a Visual C++ runtime? This fixes an issue.

According to https://github.com/apache/hadoop/tree/master/hadoop-common-project/hadoop-common/src/main/winutils, winutils in hadoop was last modified in 2015.
According to https://github.com/steveloughran/winutils, there is no another version for 2.7.* hadoop.

P.S. I will try to reproduce it with fresh Windows 10 and Drill 1.14.

@vvysotskyi
Copy link
Member

https://github.com/steveloughran/winutils contains built binaries, and according to the comment from the docs, sources are taken from the git commit ID used for the ASF release.

It will be built and pushed to the one of the repositories used by Drill.

Could you please clarify, what do you mean under adding a Visual C++ runtime?

@vvysotskyi
Copy link
Member

It's strange that for JDK 8 winutils.exe works correctly, but with JDK 11 it fails with the error specified above.

@vvysotskyi
Copy link
Member

Perhaps this issue woun't be fixed after winutils update. I have checked with 2.8.1, and it still fails. For JDK 8 msvcr100.dll was available in jre path, but for JDK 11 not, therefore it failed. And since winutils were built with JDK 8 (as it was pointed in https://github.com/steveloughran/winutils) binary files assume that msvcr100.dll will be accessible.

So I agree with you that we should document that for JDK 11 should be installed visual c ++ redistributable

Copy link
Member

@vvysotskyi vvysotskyi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@oleg-zinovev, please resolve CR comments in sqlline.bat and maven-failsafe-plugin, and we will merge this PR if there wouldn't be any additional issues.

@@ -104,7 +104,7 @@
<!-- Because the JDBC tests are somewhat heavyweight, we only run them in the 'verify' phase -->
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.18.1</version>
<version>2.22.0</version>
<configuration>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please also set <useSystemClassLoader>false</useSystemClassLoader> for maven-failsafe-plugin, since it also fails.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vvysotskyi , done

@oleg-zinovev oleg-zinovev force-pushed the DRILL-6349 branch 2 times, most recently from 43e64cf to dc0cd86 Compare November 22, 2018 06:18
Copy link
Member

@vvysotskyi vvysotskyi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@asfgit asfgit closed this in 639be6c Nov 26, 2018
mattpollack pushed a commit to mattpollack/drill that referenced this pull request Feb 25, 2019
lushuifeng pushed a commit to lushuifeng/drill that referenced this pull request Jun 21, 2019
xiangt920 pushed a commit to xiangt920/drill that referenced this pull request Dec 26, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants