JMH fails after some changes in 28th of May

[paplorinc]

It seems the nightly build from 2019 05 28 contained some changes that prohibit us from running JMH tests locally
It applies to the release branch as well
There are two snapshots for that day, the second one broke it:
• gradle-5.5-20190528232623+0000-bin.zip - bad
• gradle-5.5-20190528142438+0000-bin.zip - good

You can reproduce it on master currently if you run ./gradlew jmh from the root it will fail with:

<forked VM failed with exit code 1>
<stdout last='20 lines'>
</stdout>
<stderr last='20 lines'>
Error: Could not find or load main class org.openjdk.jmh.runner.ForkedMain
Caused by: java.lang.ClassNotFoundException: org.openjdk.jmh.runner.ForkedMain
</stderr>

[jjohannes]

I looked up the commits the snapshots were built from. This is the range: 8aa3142...ae078a5

[big-guy]

@paplorinc Good news and bad news.

Good news, I have some fixes pushed to melix/jmh-gradle-plugin#153 to fix the JMH plugin to handle the changes in 5.5

@melix could you take a look at melix/jmh-gradle-plugin#153 and merge? I'm happy to chat about it if you need more context.

Bad news, I don't think this completely fixes the problem.

Before Gradle 5.5, the classpath used by JMH had the entire Gradle runtime. I think that means it masked problems in the gradle/gradle build or the JMH plugin. When running with the changes in the PR above, JMH starts running the benchmarks and fails in a different way:
https://e.grdev.net/s/kxbfbpvf6eo24/console-log#L1923

java.lang.NoClassDefFoundError: org/bouncycastle/jcajce/provider/digest/MD5$Digest
--
at org.gradle.internal.reflect.generated.HashingAlgorithmsBenchmark_measure_jmhTest._jmh_tryInit_f_hashingalgorithmsbenchmark0_G(HashingAlgorithmsBenchmark_measure_jmhTest.java:497)
at org.gradle.internal.reflect.generated.HashingAlgorithmsBenchmark_measure_jmhTest.measure_Throughput(HashingAlgorithmsBenchmark_measure_jmhTest.java:71)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:453)
at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:437)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.ClassNotFoundException: org.bouncycastle.jcajce.provider.digest.MD5$Digest
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
... 14 more

I think this may have only worked before because the Gradle runtime was leaking into the worker and into the JMH benchmark. Now that we're not doing that, we're missing dependencies from the benchmark. I think this is a problem in the JMH Gradle plugin still, but I don't fully grok the relationship between jmh, jmhImplementation and jmhRuntime. It seems like this should be part of the classpath for the benchmark tests, but I don't see how they're used by the JMHTask right now. WDYT @melix ?

[paplorinc]

Thanks @big-guy!

The benchmark jar doesn't appear to be up-to-date safe

Could very well be, I have to run them every time like:
./gradlew --stop && ./gradlew clean jmh --no-daemon
otherwise my benchmark related changes aren't always taken into consideration

[paplorinc]

@big-guy, do you think other plugins or users will face the same? Are we safe to release 5.5 this way?

[big-guy]

@paplorinc I doubt other workers are doing what the JMH library does to derive a classpath. It was looking at the java.class.path system property and assuming that the benchmark would have the same classpath as that.

[paplorinc]

@lptr seems to be affected by this, @big-guy, @melix, any chance of fixing this?

[melix]

I have released 0.5.0-rc-1 with @big-guy 's changes.