Compatibility with Java 16 #241

ceresek · 2021-04-18T16:36:46Z

Just a quick list of benchmarks failing on Java 16 (using the 0.11.0 build), since I was running it recently:

als
java.lang.reflect.InaccessibleObjectException: Unable to make field transient java.lang.Object[] java.util.ArrayList.elementData accessible: module java.base does not "opens java.util" to unnamed module @681a8b4e
chi-square
java.lang.reflect.InaccessibleObjectException: Unable to make field transient java.lang.Object[] java.util.ArrayList.elementData accessible: module java.base does not "opens java.util" to unnamed module @681a8b4e
db-shootout
java.lang.IllegalAccessError: class net.openhft.chronicle.core.OS (in unnamed module @0x5082d622) cannot access class sun.nio.ch.FileChannelImpl (in module java.base) because module java.base does not export sun.nio.ch to unnamed module @0x5082d622
dec-tree
Caused by: java.lang.StringIndexOutOfBoundsException: begin 0, end 3, length 2
gauss-mix
java.lang.reflect.InaccessibleObjectException: Unable to make field transient java.lang.Object[] java.util.ArrayList.elementData accessible: module java.base does not "opens java.util" to unnamed module @681a8b4e
log-regression
java.lang.reflect.InaccessibleObjectException: Unable to make field transient java.lang.Object[] java.util.ArrayList.elementData accessible: module java.base does not "opens java.util" to unnamed module @4bff64c2
movie-lens
java.lang.reflect.InaccessibleObjectException: Unable to make field transient java.lang.Object[] java.util.ArrayList.elementData accessible: module java.base does not "opens java.util" to unnamed module @4c2cc639
naive-bayes
java.lang.reflect.InaccessibleObjectException: Unable to make field transient java.lang.Object[] java.util.ArrayList.elementData accessible: module java.base does not "opens java.util" to unnamed module @681a8b4e
neo4j-analytics
Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make field private java.lang.String java.lang.Throwable.detailMessage accessible: module java.base does not "opens java.lang" to unnamed module @4e31276e
page-rank
Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make field final byte[] java.nio.ByteBuffer.hb accessible: module java.base does not "opens java.nio" to unnamed module @5c08c46a

The text was updated successfully, but these errors were encountered:

farquet · 2021-04-27T14:17:57Z

With a small fix, we can make 7 of them work pretty quickly.

Illegal reflective accesses used to generate warnings, but as of JDK16, they are forbidden unless the given modules are explicitly requested on the command line (or in the MANIFEST with Add-Opens: <module>/<package> <module>/<package> ...).

I've experimented locally with a renaissance build that includes the scala upgrades from #242 to see which --add-opens where required to make the benchmarks run on JDK16:

als, chi-square, gauss-mix, log-regression, naive-bayes: --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.nio=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED
movie-lens: same as above plus --add-opens java.base/java.lang.invoke=ALL-UNNAMED

Benchmarks who don't work out of the box with the above --add-opens:

dec-tree deserves some more investigation:

org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 1.0 failed 1 times, most recent failure: Lost task 1.0 in stage 1.0 (TID 4, 192.168.1.114, executor driver): java.lang.NoClassDefFoundError: Could not initialize class com.esotericsoftware.kryo.serializers.ClosureSerializer
	at com.twitter.chill.java.Java8ClosureRegistrar.apply(Java8ClosureRegistrar.java:20)
	at com.twitter.chill.AllScalaRegistrar_0_9_2.apply(ScalaKryoInstantiator.scala:207)
	at com.twitter.chill.AllScalaRegistrar_0_10_0.apply(ScalaKryoInstantiator.scala:214)
	at com.twitter.chill.AllScalaRegistrar.apply(ScalaKryoInstantiator.scala:229)
	at org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:179)
	at org.apache.spark.serializer.KryoSerializer$$anon$1.create(KryoSerializer.scala:102)
	at com.esotericsoftware.kryo.pool.KryoPoolQueueImpl.borrow(KryoPoolQueueImpl.java:48)
	at org.apache.spark.serializer.KryoSerializer$PoolWrapper.borrow(KryoSerializer.scala:109)
	at org.apache.spark.serializer.KryoSerializerInstance.borrowKryo(KryoSerializer.scala:336)
	at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:375)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
	at org.apache.spark.sql.execution.aggregate.ComplexTypedAggregateExpression.serialize(TypedAggregateExpression.scala:263)
	at org.apache.spark.sql.catalyst.expressions.aggregate.TypedImperativeAggregate.serializeAggregateBufferInPlace(interfaces.scala:591)
	at org.apache.spark.sql.execution.aggregate.AggregationIterator.$anonfun$generateResultProjection$6(AggregationIterator.scala:279)
	at org.apache.spark.sql.execution.aggregate.ObjectAggregationIterator.next(ObjectAggregationIterator.scala:86)
	at org.apache.spark.sql.execution.aggregate.ObjectAggregationIterator.next(ObjectAggregationIterator.scala:33)
	at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
	at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:156)
	at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
	at org.apache.spark.scheduler.Task.run(Task.scala:127)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
	at java.base/java.lang.Thread.run(Thread.java:831)

neo4j-analytics is not expected to work since the given neo4j version only supports JDK11.
db-shootout still uses a chronicle version 2.17.2 which is calling setAccessible on a reflective access. We should probably try a more recent version of chronicle to see if this is gone.

java.lang.AssertionError: java.lang.NoSuchFieldException: override
	at net.openhft.chronicle.core.Jvm.setAccessible(Jvm.java:326)
	at net.openhft.chronicle.core.Jvm.getMethod0(Jvm.java:302)
	at net.openhft.chronicle.core.Jvm.getMethod(Jvm.java:296)
	at net.openhft.chronicle.core.OS.<clinit>(OS.java:78)
	at net.openhft.chronicle.map.ChronicleMapBuilder.<init>(ChronicleMapBuilder.java:206)
	at net.openhft.chronicle.map.ChronicleMapBuilder.of(ChronicleMapBuilder.java:280)
	at net.openhft.chronicle.map.ChronicleMap.of(ChronicleMap.java:71)
	at org.lmdbjava.bench.Chronicle$CommonChronicleMap.setup(Chronicle.java:118)
	at org.lmdbjava.bench.Chronicle$Reader.setup(Chronicle.java:204)
	at org.renaissance.database.DbShootout.setUpBeforeAll(DbShootout.scala:71)
	at org.renaissance.harness.ExecutionDriver.executeBenchmark(ExecutionDriver.java:57)
	at org.renaissance.harness.RenaissanceSuite$.$anonfun$runBenchmarks$1(RenaissanceSuite.scala:133)
	at org.renaissance.harness.RenaissanceSuite$.$anonfun$runBenchmarks$1$adapted(RenaissanceSuite.scala:127)
	at scala.collection.immutable.List.foreach(List.scala:333)
	at org.renaissance.harness.RenaissanceSuite$.runBenchmarks(RenaissanceSuite.scala:127)
	at org.renaissance.harness.RenaissanceSuite$.main(RenaissanceSuite.scala:104)
	at org.renaissance.harness.RenaissanceSuite.main(RenaissanceSuite.scala)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:78)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:567)
	at org.renaissance.core.Launcher.loadAndInvokeHarnessClass(Launcher.java:113)
	at org.renaissance.core.Launcher.launchHarnessClass(Launcher.java:72)
	at org.renaissance.core.Launcher.main(Launcher.java:36)

farquet · 2021-04-29T17:05:03Z

Remaining work for this issue:

Investigate dec-tree problem.
Bump chronicle version in db-shootout to see if it has been fixed in the meantime. Note that db-shootout (and chronicle in particular) is causing us several problems (db-shootout crashes with jdk-12 #163, db-shootout does not work on linux ppc64le / s390x #153). Based on the discussion in db-shootout does not work on linux ppc64le / s390x #153, we should try to replace chronicle with Ignite: db-shootout does not work on linux ppc64le / s390x #153 (comment)

vhotspur · 2021-07-21T11:01:04Z

Re-tested on a418bdb. All benchmarks are working, only als-ml fails, none of the --add-opens mentioned above worked for me.

openjdk version "16.0.1" 2021-04-20
OpenJDK Runtime Environment 21.3 (build 16.0.1+9)
OpenJDK 64-Bit Server VM 21.3 (build 16.0.1+9, mixed mode, sharing)

[2021-07-21T10:54:02.980+0000] org.renaissance.core (org.renaissance.core.Launcher main)
SEVERE: Harness failed with java.lang.RuntimeException: failed to load benchmark als-ml
	at org.renaissance.core.BenchmarkSuite.createBenchmark(BenchmarkSuite.java:100)
	at org.renaissance.harness.ExecutionDriver.create(ExecutionDriver.java:179)
	at org.renaissance.harness.RenaissanceSuite$.$anonfun$runBenchmarks$1(RenaissanceSuite.scala:137)
	at org.renaissance.harness.RenaissanceSuite$.$anonfun$runBenchmarks$1$adapted(RenaissanceSuite.scala:136)
	at scala.collection.immutable.List.foreach(List.scala:333)
	at org.renaissance.harness.RenaissanceSuite$.runBenchmarks(RenaissanceSuite.scala:136)
	at org.renaissance.harness.RenaissanceSuite$.main(RenaissanceSuite.scala:117)
	at org.renaissance.harness.RenaissanceSuite.main(RenaissanceSuite.scala)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:78)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:567)
	at org.renaissance.core.Launcher.loadAndInvokeHarnessClass(Launcher.java:114)
	at org.renaissance.core.Launcher.launchHarnessClass(Launcher.java:73)
	at org.renaissance.core.Launcher.main(Launcher.java:37)
Caused by: java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:78)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
	at org.renaissance.core.BenchmarkSuite.createBenchmark(BenchmarkSuite.java:97)
	... 14 more
Caused by: java.lang.AbstractMethodError: Receiver class org.renaissance.apache.spark.AlsMl does not define or inherit an implementation of the resolved method 'abstract void org$renaissance$apache$spark$SparkUtil$_setter_$dumpResultsBeforeTearDown_$eq(boolean)' of interface org.renaissance.apache.spark.SparkUtil.
	at org.renaissance.apache.spark.SparkUtil.$init$(SparkUtil.scala:72)
	at org.renaissance.apache.spark.AlsMl.<init>(AlsMl.scala:44)
	... 20 more

[2021-07-21T10:54:02.980+0000] org.renaissance.core (org.renaissance.core.Launcher main)
SEVERE: Harness failed with java.lang.RuntimeException: failed to load benchmark als-ml
	at org.renaissance.core.BenchmarkSuite.createBenchmark(BenchmarkSuite.java:100)
	at org.renaissance.harness.ExecutionDriver.create(ExecutionDriver.java:179)
	at org.renaissance.harness.RenaissanceSuite$.$anonfun$runBenchmarks$1(RenaissanceSuite.scala:137)
	at org.renaissance.harness.RenaissanceSuite$.$anonfun$runBenchmarks$1$adapted(RenaissanceSuite.scala:136)
	at scala.collection.immutable.List.foreach(List.scala:333)
	at org.renaissance.harness.RenaissanceSuite$.runBenchmarks(RenaissanceSuite.scala:136)
	at org.renaissance.harness.RenaissanceSuite$.main(RenaissanceSuite.scala:117)
	at org.renaissance.harness.RenaissanceSuite.main(RenaissanceSuite.scala)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:78)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:567)
	at org.renaissance.core.Launcher.loadAndInvokeHarnessClass(Launcher.java:114)
	at org.renaissance.core.Launcher.launchHarnessClass(Launcher.java:73)
	at org.renaissance.core.Launcher.main(Launcher.java:37)
Caused by: java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:78)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
	at org.renaissance.core.BenchmarkSuite.createBenchmark(BenchmarkSuite.java:97)
	... 14 more
Caused by: java.lang.AbstractMethodError: Receiver class org.renaissance.apache.spark.AlsMl does not define or inherit an implementation of the resolved method 'abstract void org$renaissance$apache$spark$SparkUtil$_setter_$dumpResultsBeforeTearDown_$eq(boolean)' of interface org.renaissance.apache.spark.SparkUtil.
	at org.renaissance.apache.spark.SparkUtil.$init$(SparkUtil.scala:72)
	at org.renaissance.apache.spark.AlsMl.<init>(AlsMl.scala:44)
	... 20 more

vhotspur · 2021-07-21T15:05:41Z

All benchmarks are working, only als-ml fails, none of the --add-opens mentioned above worked for me.

Sorry about the confusion, sbt clean have not cleaned everything (hence the als-ml still being present). als and all other benchmarks work, but JMH run fails as can be seen in this GHA log.

lbulej · 2021-07-22T08:47:48Z

Hmm, this is probably because JMH launches its own JVM and does not pass the "opens" to the newly launched JVM...

vhotspur · 2021-07-22T08:52:46Z

Hmm, this is probably because JMH launches its own JVM and does not pass the "opens" to the newly launched JVM...

I think there is something else happening as in the above mentioned log, the normal run does not append any --add-opens and it still passes the test.

I have the needed changes in this branch 65329f0.

lbulej · 2021-07-22T09:01:17Z

For the normal (non-JMH runs), the --add-opens equivalent is in MANIFEST.MF:

Add-Opens: java.base/java.lang java.base/java.lang.invoke java.base/ja
 va.util java.base/java.nio java.base/sun.nio.ch

JMH has no way of knowing to pass this on to the JVM it launches, which is why it needs the extra arguments. Which is to say that runs using the normal (non-JMH) JAR should not need workaround arguments (even for JVM 16).

vhotspur · 2021-07-22T09:06:15Z

For the normal (non-JMH runs), the --add-opens equivalent is in MANIFEST.MF:

Ahaaa, I have not noticed that.

JAR should not need workaround arguments (even for JVM 16).

Thanks! I will fix that.

Added more environments to be tested on GitHub Actions CI. It now runs on * Linux: all JDKs from 8 to 16 * Linux with OpenJ9: 8, 11, 16 * MacOS: 8, 11, 13, 15 * Windows: 8, 11, 13, 15 Also added --add-opens to JMH runs for JDK 16 (see #241).

JohnTortugo · 2021-09-15T21:49:49Z

Thank you for fixing this!

ceresek added the compatibility Relates to platform or system compatibility label Apr 18, 2021

farquet self-assigned this Apr 27, 2021

farquet mentioned this issue Apr 27, 2021

Add out of the box support for spark benchmarks on JDK16 #244

Merged

farquet added this to the 1.0.0 milestone May 3, 2021

This was referenced Jul 22, 2021

CI: test on more JDKs #291

Merged

JDK 9+ support #127

Closed

farquet closed this as completed Sep 15, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Compatibility with Java 16 #241

Compatibility with Java 16 #241

ceresek commented Apr 18, 2021

farquet commented Apr 27, 2021

farquet commented Apr 29, 2021

vhotspur commented Jul 21, 2021

vhotspur commented Jul 21, 2021

lbulej commented Jul 22, 2021

vhotspur commented Jul 22, 2021

lbulej commented Jul 22, 2021 •

edited

Loading

vhotspur commented Jul 22, 2021

JohnTortugo commented Sep 15, 2021

Compatibility with Java 16 #241

Compatibility with Java 16 #241

Comments

ceresek commented Apr 18, 2021

farquet commented Apr 27, 2021

farquet commented Apr 29, 2021

vhotspur commented Jul 21, 2021

vhotspur commented Jul 21, 2021

lbulej commented Jul 22, 2021

vhotspur commented Jul 22, 2021

lbulej commented Jul 22, 2021 • edited Loading

vhotspur commented Jul 22, 2021

JohnTortugo commented Sep 15, 2021

lbulej commented Jul 22, 2021 •

edited

Loading