Fix coverage ratchet baseline freeze in generate-coverage - #353
Conversation
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 24 minutes Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (5)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![codescene-access[bot]](https://avatars.githubusercontent.com/in/3755875?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
The generate-coverage action's ratchet restored the baseline with a run-id-suffixed primary cache key (ratchet-baseline-<os>-<run_id>) and a prefix restore-key, but saved it with a constant, run-id-less key (ratchet-baseline-<os>) guarded by cache-hit != 'true'. GitHub Actions cache entries are immutable, so the constant key could only be written once; every later run reported a cache hit on the constant primary key and skipped the save, freezing the baseline at whatever the first post-eviction run measured. The baseline only ever changed when the 7-day cache eviction forced a fresh save at a random value. Consequently the ratchet could not advance when coverage improved, and for repositories with any coverage nondeterminism it false-tripped "Coverage decreased" on pull requests. For example, chutoro measured 85.20% on a pull request against a stale, frozen 85.23% baseline. Align the save key with the restore step's run-id-suffixed primary key (ratchet-baseline-<os>-<run_id>) and drop the cache-hit guard, so every main run persists a fresh baseline that later runs recover via the ratchet-baseline-<os>- restore-key prefix (newest matching entry wins). Add a provisional symmetric +/-1 percentage-point dead-band to the ratchet comparison in ratchet_coverage.py. Coverage within one absolute percentage point of the baseline is treated as noise: the run passes and the baseline is held. A drop of more than one point below the baseline fails; a rise of more than one point above the baseline advances the baseline. Holding the baseline within the band stops a nondeterministic low run false-tripping the gate and a lucky-high run inflating the baseline so the next normal run fails. The tolerance is a single named constant (RATCHET_TOLERANCE_PP = 1.0). The action's inputs and contract are unchanged. Add tests covering the cache-key contract (the save key varies per run, matches the restore primary key, and is not gated on cache-hit) and the dead-band comparison (within-band dip and rise pass and hold the baseline, a drop beyond the band fails, a rise beyond the band advances the baseline, and exact equality holds).
305dfaf to
cd4ec5d
Compare
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
The lint-test job's coverage ratchet false-tripped on a 0.07pp dip below a stale cached baseline (86.39% vs 86.46%), and the failure surfaced as a confusing empty-artefact-name upload error that masked it. Bump the `generate-coverage` and `upload-codescene-coverage` pins from 927edd45 to 1c1a46f0, which brings the ±1pp ratchet tolerance dead-band (leynos/shared-actions#353) that absorbs measurement jitter and the archive-masking fix (leynos/shared-actions#380) that surfaces a genuine ratchet failure instead of an empty-name error. The `setup-rust` pin is left unchanged.
…het fix (#147) * Bump whitaker-installer to 0.2.6 installer 0.2.5 provisions cargo-dylint 4.1.0, whose dylint driver cannot build on the suite's new nightly-2026-05-28 pin; 0.2.6 provisions cargo-dylint 6.0.1. * Bump whitaker-installer to 0.2.7 0.2.7 fixes the dylint-link install verification defect (leynos/whitaker#299) that made the lint gate fall back to an impossible source build. * Bump generate-coverage pin to clear the ratchet dead-band The lint-test job's coverage ratchet false-tripped on a 0.07pp dip below a stale cached baseline (86.39% vs 86.46%), and the failure surfaced as a confusing empty-artefact-name upload error that masked it. Bump the `generate-coverage` and `upload-codescene-coverage` pins from 927edd45 to 1c1a46f0, which brings the ±1pp ratchet tolerance dead-band (leynos/shared-actions#353) that absorbs measurement jitter and the archive-masking fix (leynos/shared-actions#380) that surfaces a genuine ratchet failure instead of an empty-name error. The `setup-rust` pin is left unchanged. --------- Co-authored-by: leynos <leynos@rohga>
1 participant
Summary
Two coherent coverage-ratchet corrections to the
generate-coverageaction.1. Baseline-freeze cache-key fix
The ratchet restored the baseline with a run-id-suffixed primary cache key
(
ratchet-baseline-<os>-<run_id>) plus a prefix restore-key(
ratchet-baseline-<os>-), but saved it with a constant, run-id-less key(
ratchet-baseline-<os>) guarded bycache-hit != 'true'.GitHub Actions cache entries are immutable, so the constant save key could only
ever be written once. After the first save, every later run reported
Cache hit occurred on the primary key ratchet-baseline-<os>, not saving cacheand the baseline froze at whatever the first post-eviction run measured — it
only ever changed when the 7-day cache eviction forced a fresh save at a random
value.
Estate impact: because the baseline could not advance when coverage
improved, and drifted stale, the ratchet false-tripped "Coverage decreased" on
pull requests. This is the root cause of chutoro's ratchet false-trip: a PR
measuring 85.20% against a stale, frozen 85.23% baseline.
The fix aligns the save key with the restore step's run-id-suffixed primary key
and removes the
cache-hitguard, so every main run persists a fresh baselinethat later runs recover via the restore-key prefix (newest matching entry
wins).
2. Symmetric ±1pp tolerance dead-band
ratchet_coverage.pynow compares coverage within a provisional symmetric ±1percentage-point dead-band (single named constant
RATCHET_TOLERANCE_PP = 1.0):current < baseline - 1.0→ fail ("Coverage decreased").current > baseline + 1.0→ pass and advance the baseline tocurrent.Treating ±1pp as noise stops a nondeterministic low run false-tripping the gate
and, crucially, stops a lucky-high run inflating the baseline so the next normal
run fails. Both values are rounded to 2dp before comparison.
No inputs or user-facing contract change.
Review walkthrough
action.yml— the "Save baselines" step now keys onratchet-baseline-${{ runner.os }}-${{ github.run_id }}(matching the restoreprimary key) and its
ifdrops thecache-hit != 'true'guard, keeping onlysuccess() && inputs.with-ratchet == 'true'. A comment explains why. Therestore step is unchanged.
scripts/ratchet_coverage.py— addsRATCHET_TOLERANCE_PPand thethree-way dead-band comparison; the baseline is written only on a genuine
improvement beyond the band.
tests/test_ratchet_baseline.py— new regression tests. Three assert theaction.ymlcache-key contract (save key varies per run, equals the restoreprimary key, not gated on
cache-hit) and fail against the pre-fixaction.yml. The remainder cover the dead-band: within-band dip and rise passand hold the baseline (incl. the 85.23→85.20 chutoro figures and both band
edges), a drop beyond the band fails and preserves the baseline, a rise beyond
the band advances it, exact equality holds, and a missing baseline is treated
as zero.
CHANGELOG.md/README.md— document both changes and the dead-bandsemantics.
Validation
uv run pytest .github/actions/generate-coverage/tests/test_ratchet_baseline.py— 12 passed.action.yml.uv run pytest --co -q— full suite collects with no errors.ruff check/ruff format --check— clean.ty checkonratchet_coverage.py— clean.action-validator .github/actions/generate-coverage/action.yml— exit 0.markdownlint-cli2andtypos(en-GB-oxendict) on the docs — clean.Downstream consumers must bump their
generate-coveragepin to a shared-actionsSHA that includes this fix to pick up the advancing baseline and the dead-band.