Stabilize fn coverage by creating a clean room

[ryoqun]

Problem

Coverage report is lying (partly). So, increasing coverage isn't fun. ;)

Summary of Changes

Casual quick inspection and my hunch says, this might fix the issue. :p I'll add the findings as a comment later. First, let the CI spin. :)

Found in the middle of #7546

[ryoqun]

After quick googling, I bet -delete and -newer is available on macOS. Dunno why the current code uses print0-and-xargs trick needlessly.

[ryoqun]

As a matter of fact, this can't be replaced with the trick because we want to recursively remove directories which might get empty while doing search & destroy. The trick is effectively a BFS, and this find is effectively a DFS.

[t-nelson]

Confirmed macOS find has -delete and -newer

[mvines]

Cool I hope this helps! 🤞

[ryoqun]

The new logic is simpler: Remove 'em all files and directories written after the build including any coverage data and report file. Just all of them.

[codecov]

Codecov Report

Merging #7576 into master will increase coverage by 9.8%.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           master   #7576     +/-   ##
========================================
+ Coverage    71.8%   81.7%   +9.8%     
========================================
  Files         244     244             
  Lines       56626   49807   -6819     
========================================
+ Hits        40680   40706     +26     
+ Misses      15946    9101   -6845

[t-nelson]

This has to be better than the full cargo clean I usually do when coverage looks weird 😅

[ryoqun]

After painful trials, I've found pruning doesn't work.

The somewhat-complete detailed investigation report is here. :)

The excerpt for the pinned-down culprit of our diverged coverage reports:

Good:

Current view: 	top level - merkle-tree/src - merkle_tree.rs (source / functions) 			Hit 	Total 	Coverage
Test: 	lcov.info 		Lines: 	139 	153 	90.8 %
Date: 	2019-12-18 14:49:55 		Functions: 	34 	***57*** 	59.6 %
Legend: 	Lines: hit not hit 	

Bad:

Current view: 	top level - merkle-tree/src - merkle_tree.rs (source / functions) 			Hit 	Total 	Coverage
Test: 	lcov.info 		Lines: 	139 	153 	90.8 %
Date: 	2019-12-18 10:48:20 		Functions: 	34 	***114*** 	29.8 %
Legend: 	Lines: hit not hit 	

As shown above, the bad coverage report double-counts the total number of functions. This seems to happen for many files, not alone this.

This is caused by the existence of stale .gcno files, which accumulates over time in the build dir and which isn't removed by coverage.sh.

My guess is that multiple stale gcno files cause grcov to mistakenly double-counts fn definitions. No wonder for the doubled count if it can't see these are latest and stale versions and sees different object files.

Simply removing them doesn't help as well. Removing would cause grcov to fail. Because it won't be created by cargo test --no-run later. And that file is mandatory to generate a coverage report. Also, randomly removing .gcno files (like was with pruning) would make our shared? build dir a bad state, which should be expected to be consumed at various points of the git history...

Ideally, grcov should ignore old gcno files. But it don't, as it would require it to tap into cargo's build artifact file's arcane caching logic.

So, drawing from these tearful and restraining findings, as the last resort, let's stash updated and relevant coverage data files into a newly-created empty dir...

(Sorry bad explanation; I'm getting tired... :p)

[ryoqun]

I guess realpath isn't available on macOS...

[mvines]

yeah realpath is a GNU program that requires an extra install for macOS. We can require it if absolutely needed but that's not ideal (another barrier for entry)

[mvines]

I suggest just removing the data_dir variable and directly embedding target/cov/tmp everywhere data_dir is used. The two ln command aren't generic enough to handle any modifications to data_dir due to the ../../.. so the variable doesn't really help much (and in fact could hurt a future person who attempts to modify this file).

[ryoqun]

I think you have a nice point!: 690a7b3

[mvines]

I'm curious why you didn't go for a simple for loop:

for gcda_file in $(find target/cov -name \*.gcda -newer "$timing_file"); do
  declare gcno_file="${gcda_file%.gcda}.gcno"
  ln -s "../../../$gcda_file" "$data_dir/$(basename "$gcda_file")"
  ln -s "../../../$gcno_file" "$data_dir/$(basename "$gcno_file")"
done

[ryoqun]

No problem. :) That's a natural question. In short, I wanted to avoid another shellcheck exception. (SC2044). And among various alternative loop constructions, I ended to choose this particular one based on existing other uses of find in this script after some enduring time for compat. thinking. ;)

As a bonus for this round of commenting, I've made it more robust: 97fe9d9.

Ideally, I wanted to write like this (but I can't due to the restriction of bash 3+) based on this:

ryoqun@ubuqun:~/work/solana/solana$ git diff
diff --git a/scripts/coverage.sh b/scripts/coverage.sh
index 7a0df653a..3b985712f 100755
--- a/scripts/coverage.sh
+++ b/scripts/coverage.sh
@@ -51,12 +51,14 @@ mkdir -p target/cov/tmp
 
 # Can't use a simpler construct under the condition of SC2044 and bash 3
 # (macOS's default). See: https://github.com/koalaman/shellcheck/wiki/SC2044
-find target/cov -name \*.gcda -newer "$timing_file" -print0 |
-  (while IFS= read -r -d '' gcda_file; do
+(
+  shopt -s globstar nullglob
+  for gcda_file in target/cov/**/*.gcda; do
     gcno_file="${gcda_file%.gcda}.gcno"
     ln -s "../../../$gcda_file" "target/cov/tmp/$(basename "$gcda_file")"
     ln -s "../../../$gcno_file" "target/cov/tmp/$(basename "$gcno_file")"
-  done)
+  done
+)
 
 _ grcov target/cov/tmp > target/cov/lcov-full.info

[mvines]

Thank for chasing this down!

Pull request has been modified.

[ryoqun]

Let's utilize our new findings: #7576 (comment)