`lcov -l` (or `lcov --list`) output showing mistaken rate values in LCOV >=2.0?

[hartwork]

Hi, thanks for making and maintaining LCOV! 🙏

I'm using LCOV in Expat upstream CI and it came to my attention that lcov -l (or lcov --list) broke at some point in the past:

• LCOV 1.15 (as found in Ubuntu 22.04.5) works fine in current CI
• LCOV 2.0 shows wrongly small rate values
• LCOV 2.2 shows 99.9% for all rate values and # for all hit values

I should note that command genhtml of the same release produces more healthy result for all three releases.

PS: The issue started out at mozilla/grcov#1239 but turned out to be about LCOV more than grcov when I saw the same issue with coverage reporting on Expat…

Below are the precise outputs from each version for comparison:

1.15 — healthy

# lcov -q -l coverage__R_2_6_4-14-ga0dc7d5e/coverage.info
                           |Lines       |Functions  |Branches    
Filename                   |Rate     Num|Rate    Num|Rate     Num
=================================================================
lib/siphash.h              |98.8%     80| 100%    16|    -      0
lib/xmlparse.c             |94.8%   4706|97.8%   186|    -      0
lib/xmlrole.c              |99.8%    645| 100%    54|    -      0
lib/xmltok.c               |97.3%    515|97.7%    44|    -      0
lib/xmltok_impl.c          |99.2%   1057| 100%    84|    -      0
lib/xmltok_ns.c            | 100%     34| 100%    14|    -      0
xmlwf/codepage.c           | 0.0%     24| 0.0%     2|    -      0
xmlwf/readfilemap.c        |53.7%     41| 100%     1|    -      0
xmlwf/xmlfile.c            |48.6%    111|75.0%     8|    -      0
xmlwf/xmlwf.c              |37.6%    758|32.7%    49|    -      0
=================================================================
                     Total:|89.4%   7971|90.8%   458|    -      0

2.0 — rate values way off (too small)

# lcov -q -l coverage__R_2_6_4-14-ga0dc7d5e/coverage.info
                           |Lines       |Functions  |Branches    
Filename                   |Rate     Num|Rate    Num|Rate     Num
=================================================================
lib/siphash.h              |20.5%     78| 0.0%    16|    -      0
lib/xmlparse.c             | 3.8%   4433| 0.0%   163|    -      0
lib/xmlrole.c              | 8.2%    644| 0.0%    53|    -      0
lib/xmltok.c               | 8.8%    433| 0.0%    32|    -      0
lib/xmltok_impl.c          | 8.0%   1052| 0.0%    84|    -      0
lib/xmltok_ns.c            |43.8%     32| 0.0%    12|    -      0
xmlwf/codepage.c           |    -      0|    -     0|    -      0
xmlwf/readfilemap.c        | 4.5%     22| 0.0%     1|    -      0
xmlwf/xmlfile.c            |14.6%     48| 0.0%     5|    -      0
xmlwf/xmlwf.c              |17.1%    280| 0.0%    15|    -      0
=================================================================
                     Total:| 6.1%   7022| 0.0%   381|    -      0

2.2 — rate values all 99.9%

# lcov -q -l coverage__R_2_6_4-14-ga0dc7d5e/coverage.info
                           |Lines       |Functions  |Branches    
Filename                   |Rate     Num|Rate    Num|Rate     Num
=================================================================
lib/siphash.h              |99.9%      #|99.9%     #|99.9%      #
lib/xmlparse.c             |99.9%      #|99.9%     #|99.9%      #
lib/xmlrole.c              |99.9%      #|99.9%     #|99.9%      #
lib/xmltok.c               |99.9%      #|99.9%     #|99.9%      #
lib/xmltok_impl.c          |99.9%      #|99.9%     #|99.9%      #
lib/xmltok_ns.c            |99.9%      #|99.9%     #|99.9%      #
xmlwf/codepage.c           |99.9%      #|99.9%     #|99.9%      #
xmlwf/readfilemap.c        |99.9%      #|99.9%     #|99.9%      #
xmlwf/xmlfile.c            |99.9%      #|99.9%     #|99.9%      #
xmlwf/xmlwf.c              |99.9%      #|99.9%     #|99.9%      #
=================================================================
                     Total:|99.9%      #|99.9%     #|99.9%      #
Message summary:
  no messages were reported

[henry2cox]

Oops.
Thanks for finding and reporting this bug.
Will push the fix shortly.

The issue turns out to be that I had refactored the underlying DB in a way that broke lcov --list - but the regression tests were not sensitive.
This appears to be a very thinly used lcov feature.

[hartwork]

@henry2cox thanks for your reply! A fix would be great to have, thank you! 🙏

[henry2cox]

I think that this issue is fixed - so closing it now.

If there is still a bug: please either reopen this issue or file a new one.
Please include a detailed description of the problem, and ideally, include a testcase which illustrates the bug.

[hartwork]

@henry2cox looks way better now, thank you!

# lcov --version
lcov: LCOV version 2.2-23.g201467d

# lcov -q -l coverage__R_2_6_4-19-g8cb7d567/coverage.info
                           |Lines       |Functions  
Filename                   |Rate     Num|Rate    Num
====================================================
lib/siphash.h              |98.7%     79| 100%    16
lib/xmlparse.c             |95.1%   4665|97.6%   167
lib/xmlrole.c              |99.8%    645| 100%    53
lib/xmltok.c               |84.4%    513|84.2%    38
lib/xmltok_impl.c          |99.2%   1060| 100%    84
lib/xmltok_ns.c            |94.1%     34|85.7%    14
xmlwf/codepage.c           | 0.0%      4| 0.0%     2
xmlwf/readfilemap.c        |53.7%     41| 100%     1
xmlwf/xmlfile.c            |46.2%    104|71.4%     7
xmlwf/xmlwf.c              |37.3%    750|31.2%    48
====================================================
                     Total:|89.0%   7895|88.6%   430
Message summary:
  no messages were reported

I'll create a dedicate issue for -q no longer working as expected.

[hartwork]

@henry2cox PS: Is there a chance to release a version with this fix as LCOV 2.3?

[henry2cox]

Did you happen to check why the numbers are different for the 1.15 vs current report?
If they were using the same input data and no suppressions or filtering was applied, I would have expected identical numbers.
If you used a different compiler or compiler version: yeah. All bets off.

One way to see what is different is to execute genhtml -o diff_rpt current.info --baseline-file 1.15.info
This will give you a colorized HTML report which will categorize all the coverpoints.

If the input data was identical: then there is definitely a bug somewhere.

With respect to an updated release: possibly later this week, after a few other issues are resolved.

[hartwork]

Hi @henry2cox,

Did you happen to check why the numbers are different for the 1.15 vs current report? If they were using the same input data and no suppressions or filtering was applied, I would have expected identical numbers. If you used a different compiler or compiler version: yeah. All bets off.

the last run was a slightly different commit, and also without the MinGW parts that the 1.15 from CI included (but not the others). So some different in numbers is expected. Let me re-run it for the original commit so we can compare with the 2.0 output. I'll report back in 20 minutes or so.

One way to see what is different is to execute genhtml -o diff_rpt current.info --baseline-file 1.15.info This will give you a colorized HTML report which will categorize all the coverpoints.

That's great to keep in mind for the future.

If the input data was identical: then there is definitely a bug somewhere.

I understand.

With respect to an updated release: possibly later this week, after a few other issues are resolved.

Cool!

[hartwork]

Let me re-run it for the original commit so we can compare with the 2.0 output. I'll report back in 20 minutes or so.

@henry2cox the hit numbers are slightly bigger with Git HEAD than with 2.0 above — same Expat commit, same machine, both without MinGW just Linux, different version of LCOV:

# lcov --version
lcov: LCOV version 2.2-23.g201467d

# lcov -q -l coverage__R_2_6_4-14-ga0dc7d5e/coverage.info
                           |Lines       |Functions  
Filename                   |Rate     Num|Rate    Num
====================================================
lib/siphash.h              |98.7%     79| 100%    16
lib/xmlparse.c             |95.0%   4667|97.6%   167
lib/xmlrole.c              |99.8%    645| 100%    53
lib/xmltok.c               |84.4%    513|84.2%    38
lib/xmltok_impl.c          |99.2%   1060| 100%    84
lib/xmltok_ns.c            |94.1%     34|85.7%    14
xmlwf/codepage.c           | 0.0%      4| 0.0%     2
xmlwf/readfilemap.c        |53.7%     41| 100%     1
xmlwf/xmlfile.c            |46.2%    104|71.4%     7
xmlwf/xmlwf.c              |37.3%    750|31.2%    48
====================================================
                     Total:|88.9%   7897|88.6%   430

[henry2cox]

Numbers should be the same, given the same input data.

Two questions:

• can you share the captured data - file coverage__R_2_6_4-14-ga0dc7d5e/coverage.info
• if you can't share: can you run grep -c -E '^DA:' coverage__R_2_6_4-14-ga0dc7d5e/coverage.info
  The number we see should match the total shown in the lcov --list output.

[hartwork]

@henry2cox the number does match the table output, yes:

# grep -c -E '^DA:' coverage__R_2_6_4-14-ga0dc7d5e/coverage.info
7897

[henry2cox]

Thanks ....

Unless I have missed the plot here (always possible), I think that we think that the current TOT is doing the right thing with this example...so no bug/nothing to see here folks.

Does that match your understanding? Or do we still think something is fishy?

[hartwork]

@henry2cox nothing more fishy of the kind of fish that my nose was trained to recognize as fish 😄