Upload CI generated fuzz corpus coverage to codecov #4153

Anyitechs · 2025-10-10T12:39:12Z

Following the work (#3718 and #3925) that introduced uploading coverage from no-corpus fuzzing runs into codecov in CI. This PR focuses on uploading the CI-generated fuzz corpus coverage into codecov in CI.

Closes #3926

ldk-reviews-bot · 2025-10-10T12:39:15Z

👋 Thanks for assigning @TheBlueMatt as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

codecov · 2025-10-10T13:41:43Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.17%. Comparing base (fdc8731) to head (6cd3f8f).
⚠️ Report is 6 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4153      +/-   ##
==========================================
+ Coverage   88.82%   89.17%   +0.34%     
==========================================
  Files         180      180              
  Lines      137278   137271       -7     
  Branches   137278   137271       -7     
==========================================
+ Hits       121944   122406     +462     
+ Misses      12522    12257     -265     
+ Partials     2812     2608     -204

Flag	Coverage Δ
fuzzing	`31.17% <ø> (+9.68%)`	⬆️
tests	`88.64% <ø> (-0.03%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

TheBlueMatt · 2025-10-14T13:03:38Z

contrib/generate_fuzz_coverage.sh

+        for target_dir in hfuzz_workspace/*; do
+            [ -d "$target_dir" ] || continue
+            src_name="$(basename "$target_dir")"
+            for dest in "$src_name" "${src_name%_target}"; do


I don't think you need to copy into $src_name.

TheBlueMatt · 2025-10-14T13:04:38Z

contrib/generate_fuzz_coverage.sh

+                mkdir -p "test_cases/$dest"
+                # Copy corpus files into the test_cases directory
+                find "$target_dir" -maxdepth 2 -type f \
+                \( -path "$target_dir/CORPUS/*" -o -path "$target_dir/INPUT/*" -o -path "$target_dir/NEW/*" -o -path "$target_dir/input/*" \) \


Because we're just looking in hfuzz_workspace, I believe we only need to look in input, not CORPUS, INPUT, or NEW.

TheBlueMatt · 2025-10-14T13:09:51Z

.github/workflows/build.yml

          cargo clean
      - name: Run fuzzers
        run: cd fuzz && ./ci-fuzz.sh && cd ..
+      - name: Upload honggfuzz corpus


Rather than only uploading, is there a way to make this directory persistent so that we can keep it between fuzz jobs?

I'm not sure if we really need to persist the directory here. My understanding is that the fuzz job runs on the latest code changes on every PR, so the generated corpus is tailored to the code changes on that PR. If we persist the corpus from a previous run and use that on a new run, won't that produce incorrect/misleading coverage data?

I don't think the point of the fuzz job is only to generate coverage data, but rather test the code :). Having a bit more coverage data from fuzzing than we "deserve" is okay, at least now that we split the coverage data out so that codecov shows fuzzing separately, and having persistent fuzzing corpus means our fuzzing is much more likely to catch issues.

Right, how long do you think we can have this directory persisted? The upload-artifact action have a retention-days input that can be used to persist the artifact for a while. The default is 90 days but can be adjusted (https://github.com/actions/upload-artifact?tab=readme-ov-file#retention-period).

I believe the simple "upload-artifact" task just stores data for this CI run. What I was thinking is some kind of persistent directory that's shared across jobs so that each CI fuzz task picks up the latest directory, does some fuzzing, finds new test cases, then uploads a new copy with more tests in it.

What I was thinking is some kind of persistent directory that's shared across jobs so that each CI fuzz task picks up the latest directory, does some fuzzing, finds new test cases, then uploads a new copy with more tests in it.

Makes sense. I pushed eea2e4b to handle this using Github's cache action (https://github.com/actions/cache?tab=readme-ov-file).

TheBlueMatt · 2025-10-14T13:10:16Z

contrib/generate_fuzz_coverage.sh

+                # Copy corpus files into the test_cases directory
+                find "$target_dir" -maxdepth 2 -type f \
+                \( -path "$target_dir/CORPUS/*" -o -path "$target_dir/INPUT/*" -o -path "$target_dir/NEW/*" -o -path "$target_dir/input/*" \) \
+                -print0 | xargs -0 -I{} cp -n {} "test_cases/$dest/" 2>/dev/null || true


Suggested change

-print0 | xargs -0 -I{} cp -n {} "test_cases/$dest/" 2>/dev/null || true

-print0 | xargs -0 -I{} cp -n {} "test_cases/$dest/"

Done. Thank you.

TheBlueMatt · 2025-10-14T13:10:33Z

contrib/generate_fuzz_coverage.sh

+        done
+        # Check if any files were actually imported
+        if [ -n "$(find test_cases -type f -print -quit 2>/dev/null)" ]; then
+            imported=1


Not sure its worth the extra effort just to print differently.

ldk-reviews-bot · 2025-10-14T13:10:39Z

👋 The first review has been submitted!

Do you think this PR is ready for a second reviewer? If so, click here to assign a second reviewer.

Anyitechs · 2025-10-14T17:28:59Z

Thank you for the review.

I've addressed all feedbacks and pushed a fixup here 1e4a7c5

TheBlueMatt

Responded at #4153 (comment)

Anyitechs · 2025-10-27T16:41:12Z

I rebased on main and that pulled in a dependency update to proptest 1.9.0 which has broken the 1.75.0 MSRV check. This seems unrelated to my changes, but CI is failing because of that.

EDIT: This seems to be blocking the build (and fuzzing as well).

We have some complexity in `ci-fuzz.sh` to limit each fuzzer to a rough runtime, but `honggfuzz` has a `--run-time` argument that we can simply use instead, which we do here.

This now slows us down as we run our fuzz job on a machine with more than one or two cores.

TheBlueMatt · 2025-10-28T01:29:40Z

Yea, sorry, CI is kinda a mess for three reasons all at once. Can you rebase on #4179? That should get at least the fuzz job running again, even if not others.

honggfuzz builds appear to fail on 1.75 - 1.79, so we just pick an MSRV of the first toolchain that actually works. Sadly, github for some reason drops a trailing in the env entry, so we copy the rust version a few places.

Because each CI job runs on a fresh runner and can't share data between jobs. We rely on Github Actions upload-artifact and download-artifact to share the CI generated fuzz corpus, then replay them in the `contrib/generate_fuzz_coverage.sh` script to generate the coverage report.

Anyitechs · 2025-10-28T09:57:16Z

Can you rebase on #4179? That should get at least the fuzz job running again, even if not others.

Yes, done! Thank you.

tnull · 2025-10-28T10:46:24Z

FWIW, remaining CI failures should be resolved shortly by #4180

TheBlueMatt · 2025-10-28T12:43:08Z

.github/workflows/build.yml

+        uses: actions/cache@v4
+        with:
+          path: fuzz/hfuzz_workspace
+          key: fuzz-corpus-${{ github.ref }}-${{ github.sha }}


Isn't this going to be per-pr? We don't want it to be per-pr we want it to be global.

Isn't this going to be per-pr? We don't want it to be per-pr we want it to be global.

Addressed this by adding a two-step logic to the workflow: a read-only per-pr step that seeds the fuzzer for a more effective run on PRs and a main branch step that does same but also writes to the global cache.

ldk-reviews-bot requested a review from tankyleo October 10, 2025 12:49

Anyitechs marked this pull request as draft October 10, 2025 18:42

Anyitechs force-pushed the upload-fuzz-coverage branch from dc493c2 to fdf6799 Compare October 13, 2025 00:25

Anyitechs marked this pull request as ready for review October 13, 2025 01:52

tankyleo requested review from TheBlueMatt and removed request for tankyleo October 13, 2025 22:16

TheBlueMatt reviewed Oct 14, 2025

View reviewed changes

Anyitechs requested a review from TheBlueMatt October 23, 2025 12:07

TheBlueMatt reviewed Oct 24, 2025

View reviewed changes

Anyitechs force-pushed the upload-fuzz-coverage branch 2 times, most recently from 19c1495 to eea2e4b Compare October 27, 2025 16:17

Anyitechs requested a review from TheBlueMatt October 27, 2025 16:42

TheBlueMatt added 2 commits October 28, 2025 00:47

Use honggfuzz's --run-time arg to limit per-fuzz runtime in CI

7730f3b

We have some complexity in `ci-fuzz.sh` to limit each fuzzer to a rough runtime, but `honggfuzz` has a `--run-time` argument that we can simply use instead, which we do here.

Disable codegen-units = 1 in CI fuzz job

89671a5

This now slows us down as we run our fuzz job on a machine with more than one or two cores.

TheBlueMatt and others added 3 commits October 28, 2025 02:01

Bump fuzz MSRV to 1.80

1ba5def

honggfuzz builds appear to fail on 1.75 - 1.79, so we just pick an MSRV of the first toolchain that actually works. Sadly, github for some reason drops a trailing in the env entry, so we copy the rust version a few places.

f copy to canonical name, drop guard and always replay via tests

cc1355f

Anyitechs force-pushed the upload-fuzz-coverage branch from eea2e4b to de9e1fd Compare October 28, 2025 08:55

TheBlueMatt reviewed Oct 28, 2025

View reviewed changes

Anyitechs force-pushed the upload-fuzz-coverage branch from de9e1fd to fcba095 Compare October 28, 2025 16:46

persist fuzz corpus between CI runs

6cd3f8f

Anyitechs force-pushed the upload-fuzz-coverage branch from fcba095 to 6cd3f8f Compare October 28, 2025 17:10

Anyitechs requested a review from TheBlueMatt October 28, 2025 19:24

	-print0 \| xargs -0 -I{} cp -n {} "test_cases/$dest/" 2>/dev/null \|\| true
	-print0 \| xargs -0 -I{} cp -n {} "test_cases/$dest/"

Uh oh!

Upload CI generated fuzz corpus coverage to codecov #4153

Are you sure you want to change the base?

Upload CI generated fuzz corpus coverage to codecov #4153

Conversation

Anyitechs commented Oct 10, 2025

Uh oh!

ldk-reviews-bot commented Oct 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Oct 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ldk-reviews-bot commented Oct 14, 2025

Uh oh!

Anyitechs commented Oct 14, 2025

Uh oh!

TheBlueMatt left a comment

Choose a reason for hiding this comment

Uh oh!

Anyitechs commented Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TheBlueMatt commented Oct 28, 2025

Uh oh!

Anyitechs commented Oct 28, 2025

Uh oh!

tnull commented Oct 28, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ldk-reviews-bot commented Oct 10, 2025 •

edited

Loading

codecov bot commented Oct 10, 2025 •

edited

Loading

Anyitechs commented Oct 27, 2025 •

edited

Loading