genhtml can't find Java source files when generating a coverage report

[hanwen]

Please provide the following information. The more we know about your system and use case, the more easily and likely we can help.

Description of the problem / feature request / question:

$ git clone --recursive https://gerrit.googlesource.com/gerrit

$ cd gerrit

$ bazel version
Build label: 0.4.4
Build target: bazel-out/local-fastbuild/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Wed Feb 1 18:54:21 2017 (1485975261)
Build timestamp: 1485975261
Build timestamp as int: 1485975261

$ bazel coverage --test_output=streamed //gerrit-common:auto_value_tests
.........
WARNING: Streamed test output requested so all tests will be run locally, without sharding, one at a time.
INFO: Using default value for --instrumentation_filter: "//gerrit-common".
INFO: Override the above default with --instrumentation_filter
INFO: Found 1 test target...
JUnit4 Test Runner
.
Time: 0.027

OK (1 test)

BazelTestRunner exiting with a return value of 0
JVM shutdown hooks (if any) will run now.
The JVM will exit once they complete.

-- JVM shutdown starting at 2017-02-13 15:38:40 --

Exception in thread "main" java.lang.UnsupportedClassVersionError: com/google/devtools/lcovmerger/Main : Unsupported major.minor version 52.0
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:803)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:442)
at java.net.URLClassLoader.access$100(URLClassLoader.java:64)
at java.net.URLClassLoader$1.run(URLClassLoader.java:354)
at java.net.URLClassLoader$1.run(URLClassLoader.java:348)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:347)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482)
FAIL: //gerrit-common:auto_value_tests (see /home/hanwen/.cache/bazel/_bazel_hanwen/9590700e58e6cc9bff046cc6a346bc64/execroot/gerrit/bazel-out/local-fastbuild/testlogs/gerrit-common/auto_value_tests/test.log).
Target //gerrit-common:auto_value_tests up-to-date:
bazel-bin/gerrit-common/auto_value_tests.jar
bazel-bin/gerrit-common/auto_value_tests

If asking a question or requesting a feature, also tell us about the underlying problem you're trying to solve.

If possible, provide a minimal example to reproduce the problem:

Environment info

• Operating System:

• Bazel version (output of bazel info release):

• If bazel info release returns "development version" or "(@non-git)", please tell us what source tree you compiled Bazel from; git commit hash is appreciated (git rev-parse HEAD):

Have you found anything relevant by searching the web? (e.g. GitHub issues, email threads in the bazel-discuss@googlegroups.com archive)

Anything else, information or logs or outputs that would be helpful?

(If they are large, please upload as attachment or provide link).

[hanwen]

@hermione521

[hanwen]

this is on ubuntu (google laptop version)

[hanwen]

on my workstation

$ bazel coverage --test_strategy=standalone --test_output=streamed //gerrit-common:auto_value_tests
WARNING: Streamed test output requested so all tests will be run locally, without sharding, one at a time.
INFO: Using default value for --instrumentation_filter: "//gerrit-common".
INFO: Override the above default with --instrumentation_filter
INFO: Found 1 test target...
JUnit4 Test Runner
.
Time: 0.031

OK (1 test)

BazelTestRunner exiting with a return value of 0
JVM shutdown hooks (if any) will run now.
The JVM will exit once they complete.

-- JVM shutdown starting at 2017-02-13 15:43:31 --

Feb 13, 2017 3:43:31 PM com.google.devtools.lcovmerger.LcovMerger merge
SEVERE: No lcov file found.
FAIL: //gerrit-common:auto_value_tests (see /usr/local/google/home/hanwen/.cache/bazel/_bazel_hanwen/b05cca6a859569ffdc923a4aef4e793e/execroot/gerrit/bazel-out/local-fastbuild/testlogs/gerrit-common/auto_value_tests/test.log).
Target //gerrit-common:auto_value_tests up-to-date:
bazel-bin/gerrit-common/auto_value_tests.jar
bazel-bin/gerrit-common/auto_value_tests
INFO: Elapsed time: 0.837s, Critical Path: 0.51s
//gerrit-common:auto_value_tests FAILED in 1 out of 2 in 0.5s
/usr/local/google/home/hanwen/.cache/bazel/_bazel_hanwen/b05cca6a859569ffdc923a4aef4e793e/execroot/gerrit/bazel-out/local-fastbuild/testlogs/gerrit-common/auto_value_tests/test.log

[hanwen]

in the latter case, there is a coverage.dat file, but it is empty. Do we need to tweak the test runner somehow to generate the coverage file?

[hanwen]

(i'm sorry, these are really two bugs. I'm personally more interested in the latter problem.)

[hermione521]

quick update: it's not supported now

bazel/src/main/java/com/google/devtools/build/lib/rules/java/JavaCompilationHelper.java

Line 291 in 67f0f4b

// TODO(bazel-team): What about source jars?

I'll check how to make it happen.

[hanwen]

for now, I'd be satisfied if we could just skip the srcjar. They're only used for autovalue, so they shouldn't affect the coverage much.

[hanwen]

ping? could you provide a time estimate and/or a workaround?

[hermione521]

Sorry, I'm focused on benchmark now. But I will do it before I leave! (which means next month)

[ulfjack]

The auto_value_tests doesn't have any code for which Bazel could collect coverage. I'm not sure why we report an error if there's no coverage, I think in Blaze we're only reporting a warning.

In any case, I looked for a test that does have sources:

$ bazel coverage //gerrit-sshd:sshd_tests

This gives more interesting output. The test is marked as passed, but there's a stack trace in the log:

Exception in thread "Thread-0" java.lang.VerifyError: Bad type on operand stack
Exception Details:
Location:
com/google/testing/coverage/BranchDetailAnalyzer.mapProbes(Lorg/objectweb/asm/ClassReader;)Ljava/util/Map; @21: invokevirtual
Reason:
Type 'org/jacoco/core/internal/flow/ClassProbesAdapter' (current frame, stack[1]) is not assignable to 'org/objectweb/asm/ClassVisitor'
Current Frame:
bci: @21
flags: { }
locals: { 'com/google/testing/coverage/BranchDetailAnalyzer', 'org/objectweb/asm/ClassReader', 'com/google/testing/coverage/ClassProbesMapper', 'org/jacoco/core/internal/flow/ClassProbesAdapter' }
stack: { 'org/objectweb/asm/ClassReader', 'org/jacoco/core/internal/flow/ClassProbesAdapter', integer }
Bytecode:
0x0000000: bb00 2459 b700 254d bb00 2659 2c03 b700
0x0000010: 274e 2b2d 03b6 0028 2cb6 0029 b0

at com.google.testing.coverage.JacocoCoverageRunner.analyzeBranch(JacocoCoverageRunner.java:138)
at com.google.testing.coverage.JacocoCoverageRunner.create(JacocoCoverageRunner.java:84)
at com.google.testing.coverage.JacocoCoverageRunner$2.run(JacocoCoverageRunner.java:258)

That's certainly a bug, but I have no idea where that might be coming from.

[ulfjack]

Somehow, Bazel built from HEAD ends up with a broken JacocoCoverage_deploy.jar. Dunno why, but I get better results with a JacocoCoverage_deploy.jar built from head.

JUnit4 Test Runner
.SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
.
Time: 0.069

OK (2 tests)

BazelTestRunner exiting with a return value of 0
JVM shutdown hooks (if any) will run now.
The JVM will exit once they complete.

-- JVM shutdown starting at 2017-03-10 12:03:32 --

Mar 10, 2017 12:03:33 PM com.google.devtools.lcovmerger.LcovMerger merge
SEVERE: Only one lcov file supported now, but found 2

[ulfjack]

Ah, my bad. I didn't delete the old lcov files. I got coverage results now.

[ulfjack]

I filed #2663 for the deploy jar issue. With that fixed, I get coverage results for //gerrit-sshd:sshd_tests.

[ulfjack]

I can get an HTML coverage report with:

genhtml -o report/ --no-source -p "$(readlink -f bazel-gerrit)" bazel-testlogs/gerrit-sshd/sshd_tests/coverage.dat

Source doesn't work because the file names are all wrong. It would be good if we could fix that.

[hanwen]

Ulf, this is still broken for me in 0.4.5, which is newer than your update.

hanwen@han-wen:/vc/gerrit$ bazel clean
INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.
hanwen@han-wen:/vc/gerrit$ bazel coverage --test_output=streamed //gerrit-sshd:sshd_tests
WARNING: Streamed test output requested. All tests will be run locally, without sharding, one at a time.
INFO: Using default value for --instrumentation_filter: "//gerrit-sshd".
INFO: Override the above default with --instrumentation_filter
INFO: Found 1 test target...
JUnit4 Test Runner
.SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
.
Time: 0.056

OK (2 tests)

BazelTestRunner exiting with a return value of 0
JVM shutdown hooks (if any) will run now.
The JVM will exit once they complete.

-- JVM shutdown starting at 2017-04-27 12:55:53 --

Exception in thread "Thread-0" java.lang.VerifyError: Bad type on operand stack
Exception Details:
Location:
com/google/testing/coverage/BranchDetailAnalyzer.mapProbes(Lorg/objectweb/asm/ClassReader;)Ljava/util/Map; @21: invokevirtual
Reason:
Type 'org/jacoco/core/internal/flow/ClassProbesAdapter' (current frame, stack[1]) is not assignable to 'org/objectweb/asm/ClassVisitor'
Current Frame:
bci: @21
flags: { }
locals: { 'com/google/testing/coverage/BranchDetailAnalyzer', 'org/objectweb/asm/ClassReader', 'com/google/testing/coverage/ClassProbesMapper', 'org/jacoco/core/internal/flow/ClassProbesAdapter' }
stack: { 'org/objectweb/asm/ClassReader', 'org/jacoco/core/internal/flow/ClassProbesAdapter', integer }
Bytecode:
0x0000000: bb00 2459 b700 254d bb00 2659 2c03 b700
0x0000010: 274e 2b2d 03b6 0028 2cb6 0029 b0

at com.google.testing.coverage.JacocoCoverageRunner.analyzeBranch(JacocoCoverageRunner.java:138)
at com.google.testing.coverage.JacocoCoverageRunner.create(JacocoCoverageRunner.java:84)
at com.google.testing.coverage.JacocoCoverageRunner$2.run(JacocoCoverageRunner.java:258)

Target //gerrit-sshd:sshd_tests up-to-date:
bazel-bin/gerrit-sshd/sshd_tests.jar
bazel-bin/gerrit-sshd/sshd_tests
INFO: Elapsed time: 26.408s, Critical Path: 22.76s
//gerrit-sshd:sshd_tests PASSED in 0.7s

Executed 1 out of 1 test: 1 test passes.
There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are.
hanwen@han-wen:~/vc/gerrit$ bazel version
Build label: 0.4.5
Build target: bazel-out/local-fastbuild/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Thu Mar 16 12:19:38 2017 (1489666778)
Build timestamp: 1489666778
Build timestamp as int: 1489666778

[ulfjack]

You need the fix for #2663, which is 3a5d3db, which unfortunately didn't make it into 0.4.5, but will be in 0.5.0.

[hanwen]

I can get some output using 0.5.0rc3. Is there a blessed way to get genhtml to find source code, or do I have to piece together a source tree by myself?

$ genhtml -o report/ -p "$(readlink -f bazel-gerrit)" bazel-testlogs/gerrit-sshd/sshd_tests/coverage.dat
Reading data file bazel-testlogs/gerrit-sshd/sshd_tests/coverage.dat
Found 93 entries.
Using user-specified filename prefix "/usr/local/google/home/hanwen/.cache/bazel/_bazel_hanwen/3dd1ab8ef3a0892fac2e8f719daba681/execroot/gerrit"
Writing .css and .png files.
Generating output.
Processing file com/google/gerrit/sshd/CommandProvider.java
genhtml: ERROR: cannot read com/google/gerrit/sshd/CommandProvider.java

[ulfjack]

Progress! Unfortunately, Javac doesn't put the full relative path into the class files, which is not ideal, and on top of that, genhtml doesn't support looking for files under a custom prefix (or a list of prefixes), and, even if it did, we wouldn't know which prefixes.

Ideally, we'd hack Javac to output the relative path, although it's technically in violation of the spec.

@cushon Suggestions?

[hanwen]

genhtml is a Perl script (shudder), and reads files serially (sigh.). Do you know of any newer tools to process this data? Is there maybe documentation for the file format of the .dat files so I can whip up my own?

[hanwen]

I think it would be easiest to get all srcjars and then look inside them for the filenames. That also sidesteps the problem of how to deal with generated source code.

[ulfjack]

The .dat files are in gcov format, I think, which is not difficult to parse. There are dozens of tools, but I'm not aware of any good tools to generate a report. We've had other reports about genhtml that make it sound like we may have to open source the tool we're using internally or write a new one from scratch.

Note that we currently never collect coverage for generated files.

[cushon]

@ulfjack - JVMS 4.7.10 makes it clear that's deliberate, so if it's possible to handle another way that'd be better.

We're already writing a lib{name}.jar_manifest_proto output for Java compilations that includes this information, e.g.:

  compilation_unit: {
    path     : "third_party/java_src/google_common/current/java/com/google/common/base/Optional.java"
    pkg      : "com.google.common.base"
    generated_by_annotation_processor: false
    top_level: [ "Optional" ]
  }

[hanwen]

see https://gerrit-review.googlesource.com/c/106471/ . From my perspective we can close this now

[ulfjack]

I've changed the title and will reuse this bug for the problem that genhtml can't find the Java source files with the default configuration. It's great that you found a workaround, but it's not ideal in the medium term.

[ulfjack]

@cushon we need to know the relative path to the source file for each class file, ideally even if there are multiple classes with the same name. Ideally, the coverage tool will output the correct relative path, which means we need to pass this information to the coverage tool itself.

It would seem cleaner to put this information into the class file itself. If we can't use SourceName, that's fine, as long as we can put it somewhere - maybe SourceDebugExtension? Or maybe we could add a custom tag?

[dhalperi]

After recent work, this command will generate a beautiful combined coverage

bazel coverage //... \                                                  
--combined_report=lcov \
--coverage_report_generator=@bazel_tools//tools/test/LcovMerger/java/com/google/devtools/lcovmerger:Main

But if suffers from all the issues here. genhtml will, when ignoring errors, produce a useful file (though it flattens the java package hierarchy for some reason?).

genhtml -o coverage bazel-out/_coverage/_coverage_report.dat --no-source
and
genhtml --ignore-errors source -o coverage bazel-out/_coverage/_coverage_report.dat both produce useful file-level coverage but do not, of course, have the ability to show source.

Is there a now-recommended way to visualize the coverage output?

[iirina]

Running coverage with --experimental_java_coverage should fix this issue. I was able to verify this with one of bazel's java tests[1], but I would be grateful if someone else could also confirm.

[1]:

bazel coverage --experimental_java_coverage src/test/java/com/google/devtools/common/options:options_test
genhtml -o coverage-report bazel-out/k8-fastbuild/testlogs/src/test/java/com/google/devtools/common/options/options_test/coverage.dat

Generating output.
Processing file options/OptionDocumentationCategory.java
Processing file options/OptionsParsingException.java
Processing file options/TriState.java
Processing file options/ShellQuotedParamsFilePreProcessor.java
Processing file options/InvocationPolicyEnforcer.java
Processing file options/Options.java
Processing file options/InvocationPolicyParser.java
Processing file options/CommandNameCache.java
Processing file options/OptionsBase.java
Processing file options/BoolOrEnumConverter.java
Processing file options/GenericTypeHelper.java
Processing file options/OptionValueDescription.java
Processing file options/OptionFilterDescriptions.java
Processing file options/ParamsFilePreProcessor.java
Processing file options/UnquotedParamsFilePreProcessor.java
Processing file options/IsolatedOptionsData.java
Processing file options/OptionPriority.java
Processing file options/OptionEffectTag.java
Processing file options/EnumConverter.java
Processing file options/OptionsParser.java
Processing file options/OptionInstanceOrigin.java
Processing file options/OptionsData.java
Processing file options/DuplicateOptionDeclarationException.java
Processing file options/OptionMetadataTag.java
Processing file options/UsesOnlyCoreTypes.java
Processing file options/OptionsUsage.java
Processing file options/OpaqueOptionsData.java
Processing file options/ParsedOptionDescription.java
Processing file options/OptionsParserImpl.java
Processing file options/OptionDefinition.java
Processing file options/Converters.java
Processing file options/OptionsClassProvider.java
Writing directory view page.
Overall coverage rate:
  lines......: 81.7% (1589 of 1945 lines)
  functions..: 80.0% (320 of 400 functions)

[dhalperi]

For me:

bazel coverage //projects/... \                             
--combined_report=lcov \
--coverage_report_generator=@bazel_tools//tools/test/LcovMerger/java/com/google/devtools/lcovmerger:Main \
--experimental_java_coverage

followed by

genhtml -o coverage bazel-out/_coverage/_coverage_report.dat

followed by

open coverage/index.html

Seems to have worked great (@sha 041f4de). Thanks!

[iirina]

That's great to hear @dhalperi!

I will close this issue. If anyone is still having troubles with genhtml and Bazel's lcov report (using --experimental_java_coverage), feel free to reopen.