Skip to content

chore: reduce test.yml flake — pin mise tools, skip cargo-cooldown on CI#498

Merged
avihut merged 2 commits into
masterfrom
daft-484/chore/reduce-test-flake
May 15, 2026
Merged

chore: reduce test.yml flake — pin mise tools, skip cargo-cooldown on CI#498
avihut merged 2 commits into
masterfrom
daft-484/chore/reduce-test-flake

Conversation

@avihut
Copy link
Copy Markdown
Owner

@avihut avihut commented May 15, 2026

Summary

Two changes from #484 that reduce test.yml's flake surface. The third
ticket-suggested change (explicit actions/cache@v4 step) is intentionally
deferred — jdx/mise-action@v4 already caches ~/.local/share/mise keyed
on mise.toml + mise.lock, verified on run 25909874748
(cache restored from key: mise-v1-linux-x64-69f084f..., install completed
in ~3s). The cache amplifier diagnosis in the ticket body was wrong; ship
the two changes that do still help and gate the explicit cache step behind
post-merge observation (V4 below).

Commit 1 — chore(mise): pin tool versions to lockfile values

Replace latest with exact mise.lock values for lefthook, cocogitto,
bun, bat, and cargo:cargo-cooldown. Zero-behavior-change tightening
— each pin matches what these tools resolve to today. The win is removing
per-install version-resolution lookups against backend registries (aqua,
cargo); those round-trips happen even on cache HIT and are a flake surface
of their own.

neovim intentionally stays at "stable" — the
vfox:mise-plugins/vfox-neovim backend does not record concrete versions
in mise.lock, so there's nothing valid to substitute. Inline comment
explains.

Weekly mise-tool-updates.yml runs mise upgrade (respects
minimum_release_age = "7d") and will keep these current, auto-opening
PRs that update mise.toml + mise.lock together.

Commit 2 — ci(test): skip cargo-cooldown install in lint job

Set MISE_DISABLE_TOOLS: cargo:cargo-cooldown at the lint job level in
test.yml. Removes a ~2 min from-source cargo build (~125 transitive
crates from crates.io) on every cold-cache run — the exact flake surface
the cocogitto-style CDN 5xx came through.

scripts/check-lockfile-age.sh already enforces the same supply-chain
cooldown policy at PR boundary on Cargo.lock and bun.lock. The
cargo-cooldown tool wraps cargo build|check|test|run|update, which CI
doesn't exercise interactively, so the CI install covers nothing
scripts/check-lockfile-age.sh doesn't. Local devs still get the wrap —
mise.toml keeps the pin.

Job-level scope (not step-level) is deliberate: subsequent mise run clippy and mise run test:unit steps need the same disable to stay
consistent.

Verification

  • V1 — baseline (pre-merge): 5 most recent successful lint jobs
    inspected for Restoring mise cache outcome. Run 25909874748
    confirmed cache HIT.
  • V2 — pin lands cleanly: mise run clippy, mise run test:unit,
    mise run fmt:check all pass locally. CI on this PR will confirm
    Install mise log no longer shows latest for the 5 pinned tools.
  • V3 — cargo:cargo-cooldown skip works on CI: this PR's lint job
    log should not list cargo:cargo-cooldown in the mise install
    tool-list. (Backend-prefix matching is required — bare names silently
    no-op.)
  • V4 — full caches hit (post-merge): watch the next 5 PRs that don't
    touch mise.toml/mise.lock. Pass if ≥4/5 show cache restored from key: and Install mise finishes in <5s. If <4/5, open a follow-up to
    add the explicit actions/cache@v4 step.
  • V5 — weekly mise-tool-updates.yml: next daily mise upgrade run
    should auto-PR pinned-version bumps (not latest reverts).

Test plan

  • CI lint job on this PR passes
  • CI lint job's Install mise log shows lefthook 2.1.6, cocogitto 7.0.0, bun 1.3.13, bat 0.26.1, cargo:cargo-cooldown 0.3.0
    resolved (not latest)
  • CI lint job's Install mise log does not list
    cargo:cargo-cooldown as an install target
  • No regression in lint job wall-clock vs pre-merge baseline
  • Local: unset MISE_DISABLE_TOOLS; mise install --dry-run still
    lists cargo:cargo-cooldown (local dev unaffected)

Pitfalls noted

  • MISE_DISABLE_TOOLS value must include the cargo: backend prefix.
    Bare cargo-cooldown silently no-ops; V3 will catch.
  • First lint run post-merge incurs one cold cache miss because
    mise.toml content changes the file-hash key segment.
  • Scope is narrow to test.yml. bench.yml and mise-tool-updates.yml
    also use mise-action but were left alone to match ticket scope —
    separate ticket if their flake matters.

Fixes #484

🤖 Generated with Claude Code

avihut and others added 2 commits May 15, 2026 12:38
Replace `latest` with exact versions for lefthook, cocogitto, bun, bat,
and cargo:cargo-cooldown — each pin matches the corresponding entry in
mise.lock today, so this is a zero-behavior-change tightening.

The win is removing per-install version-resolution lookups against
backend registries (aqua, cargo). Even when mise-action's cache HITs on
the installs directory, mise still has to resolve `latest` against the
registry to decide whether the cached version is current, which is a
network round-trip on every CI run and a flake surface in its own right.

`neovim` stays at `stable` — the vfox:mise-plugins/vfox-neovim backend
does not record concrete versions in mise.lock, so there's no value to
substitute. Documented inline.

The weekly mise-tool-updates.yml workflow runs `mise upgrade` which
respects minimum_release_age=7d and will keep these pins current,
opening an auto-PR with both mise.toml and mise.lock bumps.

Refs #484

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The lint job currently spends ~2 min on a cold-cache build of
cargo-cooldown (~125 transitive crates from crates.io). That cold path
is the actual flake surface — once installs/ is cached, the tool is
free, but the build runs on every cache invalidation and on workflow
matrix breadth changes.

The policy cargo-cooldown enforces locally — refusing to consume
package versions younger than the cooldown.toml window — is already
enforced at PR boundary by scripts/check-lockfile-age.sh, which the
dep-age-check job runs against Cargo.lock and bun.lock. cargo-cooldown
itself wraps `cargo build|check|test|run|update`, which CI doesn't
exercise interactively. The CI install therefore covers nothing that
the lockfile-age check doesn't already cover.

Setting MISE_DISABLE_TOOLS at the job level (not the step) is
deliberate: later steps run `mise run clippy` and `mise run test:unit`,
and we want the disable consistent across every mise invocation in the
job. Job-level scope keeps the contract uniform.

mise.toml continues to pin cargo:cargo-cooldown = "0.3.0" so local devs
still get the supply-chain wrap on `cargo build|check|test|run|update`.

Refs #484

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@avihut avihut added this to the Public Launch milestone May 15, 2026
@avihut avihut added chore Maintenance tasks ci CI/CD changes labels May 15, 2026
@avihut avihut self-assigned this May 15, 2026
@avihut
Copy link
Copy Markdown
Owner Author

avihut commented May 15, 2026

/claude review

@claude
Copy link
Copy Markdown

claude Bot commented May 15, 2026

Claude finished @avihut's task in 1m 27s —— View job


Review

  • Read changed files
  • Analyze changes against CLAUDE.md guidelines
  • Provide review feedback

Both changes are correct and well-reasoned. No blocking issues. A few observations below.

mise.toml — version pins

All 5 pinned versions verified against mise.lock:

Tool Pinned Lock
lefthook 2.1.6 2.1.6
cocogitto 7.0.0 7.0.0
bun 1.3.13 1.3.13
bat 0.26.1 0.26.1
cargo:cargo-cooldown 0.3.0 0.3.0

neovim correctly stays at "stable"mise.lock confirms the vfox backend records no concrete version or checksum for it (just a URL pointing to the stable tag). The inline comment explains the limitation accurately.

Minor: rust = "1.93" in mise.toml while mise.lock resolves it to "1.93.1". Pre-existing inconsistency, not introduced here, but a candidate for a follow-up pin (or leave it — patch pinning Rust is rarely meaningful).

test.ymlMISE_DISABLE_TOOLS on lint job

The implementation is correct:

  • cargo: backend prefix is required — bare cargo-cooldown silently no-ops in mise, as the comment notes. This is the one failure mode worth watching (V3 in the test plan).
  • Job-level scope is right — both mise run clippy (line 160) and mise run test:unit (line 163) invoke mise, so step-level on one would be inconsistent.
  • Other jobs unaffected correctlymsrv-check, windows-check, xtask-test, and build don't use jdx/mise-action at all; integration-tests downloads a pre-built artifact. No stray cargo-cooldown installs to worry about.

Minor nit: The test.yml comment is 9 lines. For a CI workflow this length is understandable (non-obvious env var, silent failure mode), but it's on the verbose side given the PR description captures the same rationale permanently.

Supply-chain / dep-age compliance

Pinning to existing mise.lock values satisfies minimum_release_age = "7d" automatically — no new version resolution required, no allowlist entries needed.

Scope confirmation

bench.yml and mise-tool-updates.yml intentionally excluded — consistent with the stated scope. scripts/check-lockfile-age.sh covers the policy at PR boundary regardless.

Verdict: LGTM. The two changes are independent, narrowly scoped, and eliminate real flake surfaces without behavioral regression. V3 (CI log should not list cargo:cargo-cooldown as an install target) is the key thing to confirm on this PR's run.

@avihut avihut merged commit 0b045a0 into master May 15, 2026
12 checks passed
@avihut avihut deleted the daft-484/chore/reduce-test-flake branch May 15, 2026 09:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

chore Maintenance tasks ci CI/CD changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: reduce test.yml flake surface (mise install cache, pin versions, cargo-cooldown)

1 participant