Compiler exception running coverage: ClassNotFoundException: org.jacoco.agent.rt.internal_b0d6a23.Offline

[robfig]

"bazel coverage" fails at 0.14.1 and 4001ddd with a compiler exception. It happens with and without --experimental_java_coverage, and it seems to be triggered by instrumentation of an error-prone check.

Here's the scenario:

# //src/com/corp/util/i18n/BUILD
java_library(
    name = "i18n",
    srcs = glob(["*.java"]),
    exported_plugins = [
        "//src/com/corp/util/i18n/bugpatterns",
    ],
   ...  

and

# //src/com/corp/util/i18n/bugpatterns/BUILD
java_plugin(
    name = "bugpatterns",
    srcs = glob(["*.java"]),
   ...  

Here's the full error:

$ ../bazel/bazel-bin/src/bazel coverage test/com/corp/util/...

ERROR: /Users/robfig/alpha/src/com/corp/util/callable/BUILD:3:1: Building src/com/corp/util/callable/libcallable.jar (1 source file) failed (Exit 1): java failed: error executing command
  (cd /private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/execroot/corp && \
  exec env - \
    LC_CTYPE=en_US.UTF-8 \
  external/local_jdk/bin/java -Xbootclasspath/p:external/bazel_tools/third_party/java/jdk/langtools/javac-9+181-r4173-1.jar -jar external/bazel_tools/tools/jdk/JavaBuilder_deploy.jar @bazel-out/darwin-fastbuild/bin/src/com/corp/util/callable/libcallable.jar-2.params)
An exception has occurred in the compiler ((version info not available)). Please file a bug against the Java compiler via the Java bug reporting page (http://bugreport.java.com) after checking the Bug Database (http://bugs.java.com) for duplicates. Include your program and the following diagnostic in your report. Thank you.
java.util.ServiceConfigurationError: com.google.errorprone.bugpatterns.BugChecker: Provider com.corp.util.i18n.bugpatterns.I18nNonConstant could not be instantiated
	at java.util.ServiceLoader.fail(ServiceLoader.java:232)
	at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
	at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
	at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
	at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
	at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:47)
	at com.google.errorprone.scanner.ScannerSupplier.fromBugCheckerClasses(ScannerSupplier.java:71)
	at com.google.errorprone.ErrorPronePlugins.loadPlugins(ErrorPronePlugins.java:47)
	at com.google.errorprone.ErrorProneAnalyzer.lambda$scansPlugins$0(ErrorProneAnalyzer.java:76)
	at com.google.common.base.Suppliers$NonSerializableMemoizingSupplier.get(Suppliers.java:164)
	at com.google.errorprone.ErrorProneAnalyzer.finished(ErrorProneAnalyzer.java:152)
	at com.google.devtools.build.buildjar.javac.plugins.errorprone.ErrorPronePlugin.postFlow(ErrorPronePlugin.java:110)
	at com.google.devtools.build.buildjar.javac.BlazeJavaCompiler.flow(BlazeJavaCompiler.java:112)
	at com.sun.tools.javac.main.JavaCompiler.flow(JavaCompiler.java:1363)
	at com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:959)
	at com.sun.tools.javac.api.JavacTaskImpl.lambda$doCall$0(JavacTaskImpl.java:100)
	at com.sun.tools.javac.api.JavacTaskImpl.handleExceptions(JavacTaskImpl.java:142)
	at com.sun.tools.javac.api.JavacTaskImpl.doCall(JavacTaskImpl.java:96)
	at com.sun.tools.javac.api.JavacTaskImpl.call(JavacTaskImpl.java:90)
	at com.google.devtools.build.buildjar.javac.BlazeJavacMain.compile(BlazeJavacMain.java:110)
	at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder$2.invokeJavac(SimpleJavaLibraryBuilder.java:121)
	at com.google.devtools.build.buildjar.ReducedClasspathJavaLibraryBuilder.compileSources(ReducedClasspathJavaLibraryBuilder.java:54)
	at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.compileJavaLibrary(SimpleJavaLibraryBuilder.java:124)
	at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.run(SimpleJavaLibraryBuilder.java:132)
	at com.google.devtools.build.buildjar.BazelJavaBuilder.processRequest(BazelJavaBuilder.java:105)
	at com.google.devtools.build.buildjar.BazelJavaBuilder.runPersistentWorker(BazelJavaBuilder.java:67)
	at com.google.devtools.build.buildjar.BazelJavaBuilder.main(BazelJavaBuilder.java:45)
Caused by: java.lang.NoClassDefFoundError: org/jacoco/agent/rt/internal_b0d6a23/Offline
	at com.corp.util.i18n.bugpatterns.I18nNonConstant.$jacocoInit(I18nNonConstant.java)
	at com.corp.util.i18n.bugpatterns.I18nNonConstant.<clinit>(I18nNonConstant.java)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at java.lang.Class.newInstance(Class.java:442)
	at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
	... 24 more
Caused by: java.lang.ClassNotFoundException: org.jacoco.agent.rt.internal_b0d6a23.Offline
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 32 more
INFO: Elapsed time: 18.145s, Critical Path: 8.44s
INFO: 14 processes: 10 darwin-sandbox, 1 local, 3 worker.
FAILED: Build did NOT complete successfully

It seems like coverage instrumentation should ignore java_plugin targets.

This is a follow-up of #4398

[iirina]

@cushon I'm a bit puzzled by what happens here, do you have any idea what is causing this error?

[cushon]

When JavaBuilder loads plugins it uses a non-standard classloader that prevents classes from JavaBuilder 'leaking' on to the plugin classpath:

bazel/src/java_tools/buildjar/java/com/google/devtools/build/buildjar/javac/BlazeJavacMain.java

Line 236 in 6c8166e

private static class ClassloaderMaskingFileManager extends JavacFileManager {

Maybe those are interfering with jacoco?

It seems suspicious that the plugins are being instrumented at all, though. Do you expect instrumentation to be applied to host deps (like java_library.exported_plugins)?

[iirina]

It seems suspicious that the plugins are being instrumented at all, though. Do you expect instrumentation to be applied to host deps (like java_library.exported_plugins)?

IMO they should not be instrumented, but java_plugin has almost the same implementation as java_library. It looks like they are instrumented and I was wondering how we got away with it until now. My guess is the plugins are usually filtered away since they are in different packages. @robfig are you using --instrumentation_filter? You can also use this flag to exclude targets. Maybe the filter will temporarily fix this until we get a proper fix in.

[robfig]

Yep, we can avoid running coverage on the plugin as a workaround. Thanks

[iirina]

@robfig Can you tell me more about the setup? I couldn't replicate yet. Do you have a java_test that depends on the java_library //src/com/corp/util/i18n:i18n? Does it use the exported plugin? How does //src/com/corp/util/callable:callable come into play in this scenario?

[robfig]

I think the key thing is that a java_plugin is included in the bazel coverage command. In this case, our i18n package exports a plugin to identify common errors in using it.

So:
src/com/corp/util/i18n:i18n is a java library
src/com/corp/util/i18n/bugpatterns:bugpatterns is a java_plugin

bazel coverage src/com/corp/util/... fails because "bazel coverage src/com/corpo/util/i18n/bugpatterns" fails.

Assuming I am correct, the proposed solution is to ignore java_plugin targets when instrumenting code, or filter them out entirely when running coverage.

[robfig]

(If that doesn't seem right or you can't immediately reproduce, I'm happy to put one together. I thought it would it would reproduce quickly/easily)

[perezd]

I just ran into this as well, plugins seem to break things. (Bazel v0.21)
My sense is that the java agent isn't available in the context of the annotation processors that are running...is there a way to exclude annotation processors from being instrumented?

[perezd]

Any thoughts on this? I'm still blocked on this, coverage fails for me in the identical way described here.

[softprops]

seeing this as well with

bazel version
Build label: 0.29.1
Build target: bazel-out/darwin-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Tue Sep 10 13:47:42 2019 (1568123262)
Build timestamp: 1568123262
Build timestamp as int: 1568123262

[softprops]

still seeing this with 1.0

bazel version                                             
Build label: 1.0.0
Build target: bazel-out/darwin-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Thu Oct 10 10:19:27 2019 (1570702767)
Build timestamp: 1570702767
Build timestamp as int: 1570702767

[sgammon]

still seeing this with 2.1.0

Bazelisk version: development
Build label: 2.1.0
Build target: bazel-out/darwin-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Fri Feb 7 13:04:00 2020 (1581080640)
Build timestamp: 1581080640
Build timestamp as int: 1581080640

[ulfjack]

I can't reproduce either. I think this will require a minimal reproduction from you for the Bazel team to be able to fix it.

"Normal" Javac plugins are clearly compiled in the host config, without instrumentation. Maybe there's something wrong about how errorprone loads plugins that causes it to pick up classes from the compilation classpath? However, that would require an additional non-host dependency on the plugin, i.e., bazel query --nohost_deps "somepath(//test/target, //plugin/target)" would be non-empty.

[sgammon]

@ulfjack i'm haven't been able to track down what's breaking it yet, but i have made progress.

here's a working lcov example with Java: https://github.com/sgammon/bazel-coverage

i'm going to build on that, adding dependencies from my app until something breaks it. currently, the behaviors i'm observing are:

1. some kind of incompatibility between elemental2 and coverage, which yields the error named in this issue

2. some issue (unrelated to elemental2) wherein the coverage.dat file is written, but ends up empty, even for a valid Java test of a similar, simple nature. when bazel tries to merge this empty file into a unified coverage report, i get the following error:

Feb 17, 2020 12:56:36 PM com.google.devtools.coverageoutputgenerator.Main getTracefiles
INFO: Found 1 tracefiles.
Feb 17, 2020 12:56:36 PM com.google.devtools.coverageoutputgenerator.Main parseFiles
SEVERE: Parsing file bazel-out/darwin-dbg/testlogs/javatests/core/DualStackTest/coverage.dat
Feb 17, 2020 12:56:36 PM com.google.devtools.coverageoutputgenerator.Main getGcovInfoFiles
INFO: No gcov info file found.
Feb 17, 2020 12:56:36 PM com.google.devtools.coverageoutputgenerator.Main getProfdataFileOrNull
INFO: No .profdata file found.
Feb 17, 2020 12:56:36 PM com.google.devtools.coverageoutputgenerator.Main main
WARNING: There was no coverage found.
ERROR: output '_coverage/_coverage_report.dat' was not created
ERROR: not all outputs were created or valid
Target //javatests/core:DualStackTest up-to-date:
  dist/bin/javatests/core/DualStackTest.jar
  dist/bin/javatests/core/DualStackTest
INFO: Elapsed time: 1.290s, Critical Path: 0.63s
INFO: 2 processes: 1 remote cache hit, 1 local.
FAILED: Build did NOT complete successfully

i am working on (1) by overlaying dependencies until my sample breaks, and working on (2) the opposite direction, by removing complexity until it works. just posting this to see if it might spur anyone into a deeper understanding than i've come across yet.

[asiekkowa]

@sgammon any progress on this? (I use bazel 2.2.0)

[sgammon]

@asiekkowa unfortunately, no, and now on bazel 3.7.0 it's still broken :(

[sgammon]

@asiekkowa // cc @iirina, @lberki

okay, i was able to work around this issue by switching toolchains:

old .bazel.rc:

build --java_toolchain=@bazel_tools//tools/jdk:toolchain_vanilla
build --host_java_toolchain=@bazel_tools//tools/jdk:toolchain_vanilla

fixed .bazel.rc:

build --java_toolchain=@bazel_tools//tools/jdk:toolchain_java11
build --host_java_toolchain=@bazel_tools//tools/jdk:toolchain_java11

the error is somewhere in between these states, at least for me. having built my codebase against toolchain_vanilla, it took some work to get it building under errorprone, but once it was building there, coverage resumed working as expected.

[gergelyfabian]

I think this is caused by using an incompatible Jacoco build included in Bazel (or java_tools).
I wrote more about this in #11674 (comment).

The issue is that Bazel seems to be dependent on the Jacoco build's package name (that is b0d6a23 in this ticket). I had this when I was trying to upgrade Jacoco for my Bazel project (with a custom Jacoco version). Fixed that by changing Jacoco's package name (in a pom.xml) to the one expected by Bazel. I have no idea though where Bazel or the tools have hard-coded the package name.

I only know that Jacoco hard codes the package name to the jar (even into class files I guess), so if you give Bazel a Jacoco version that has the wrong package name then Bazel will blow up with this error.

[gergelyfabian]

If you go and try replace jacoco in Bazel's third_party/java/jacoco (from an officially released jacoco zip), you'll probably get the same error.

[sgammon]

@gergelyfabian this problem doesn't always fail with the titled exception. it is worth noting that i still get an exception looking like this unless i pin my project to Jacoco 0.8.3

[sgammon]

well, it definitely seems to me like this is related to plugins. in my local project, i have the following 3 kinds of tests:

1: calling Java from Java (JUnit 5)
2: calling Kotlin from Java (JUnit 5)
3: calling Kotlin from Java (JUnit 5, but with Micronaut plugins)

i've now been able to determine that with the changes at rules_kotlin needed to get coverage working, cases 1 and 2 above work just fine.

for kotlin or java sources touched by plugins, though (in my case, facilitating DI), coverage just vanishes.

sources i would expect to see in coverage are listed in *.instrumented_files, as they should be - they are definitely executed in the test - but they don't show up in the .dat file.

[sgammon]

okay, i've got a small code sample and the kicker is, i wasn't able to repro (yet). so the following cases work fine

1: calling Java from Java (JUnit 5)
2: calling Kotlin from Java (JUnit 5)
3: calling Java from Java with plugins (Junit 5, Micronaut plugins)

still not working:

1: calling Kotlin from Java (JUnit 5, but with micronaut plugins)

obviously this leads me to believe it might simply be some condition related to Kotlin and how that layer handles plugins. since Micronaut can generate code, perhaps that step is running after jacoco, and thus, instrumentation calls are dropped?

... but that wouldn't make sense, right, because jacoco is agent-based? i'm so confused.

@iirina is any of this making sense?

[ulfjack]

@sgammon, can you post a repro for the calling Kotlin from Java issue you're seeing? Do you see the ClassNotFoundException that this bug was originally filed for?

[sgammon]

@ulfjack https://github.com/sgammon/coverage-repro

that's a bit old, on Bazel 3.7.2, but it demonstrates each of those points above. make test builds, assembles a coverage report, and starts up a local server to view it.

at one point early on, yes, this exception was involved. i don't believe i had seen it for some time though.

[github-actions]

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 14 days unless any other activity occurs or one of the following labels is added: "not stale", "awaiting-bazeler". Please reach out to the triage team (@bazelbuild/triage) if you think this issue is still relevant or you are interested in getting the issue resolved.

[github-actions]

This issue has been automatically closed due to inactivity. If you're still interested in pursuing this, please reach out to the triage team (@bazelbuild/triage). Thanks!