Skip to content

test(enrichment): add flaky-test retry to the review-enrichment suite - #2436

Merged
JSONbored merged 2 commits into
JSONbored:mainfrom
RenzoMXD:fix/rees-flaky-test-retry-2425
Jul 2, 2026
Merged

test(enrichment): add flaky-test retry to the review-enrichment suite#2436
JSONbored merged 2 commits into
JSONbored:mainfrom
RenzoMXD:fix/rees-flaky-test-retry-2425

Conversation

@RenzoMXD

@RenzoMXD RenzoMXD commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Summary

Closes #2425.

What changed:

  • Added a shell-level retry to the REES test script (reruns the full suite once on failure), matching the main vitest suite's one-retry-before-fail policy. Node's built-in test runner doesn't support --test-retries=N\ (the issue's assumption was incorrect for Node 22/24), so the retry is implemented via
    ode --test ... || node --test ...\ in the npm script.
  • Fixed the root cause of the observed flake in \�uildBrief stays fail-open and captures a degraded analyzer: the test used \�udget.timeoutMs=200, but the balanced profile's
    esponseReserveMs\ floors at 150ms, leaving only ~50ms of execution headroom. Under CI load, the pre-flight budget gate in
    unAnalyzer\ (\�rief.ts:248-262) short-circuits to \capped\ before the analyzer runs at all, so the expected \degraded\ status is never reached. Raising the budget to 2000ms gives ~1600ms of headroom, deterministically reaching the analyzer throw path.

Rationale: A single transient flake must not red a required check and one-shot-close a contributor PR. The main vitest suite already has
etry: 1\ for this; the REES suite (Node's built-in test runner) had no equivalent.

Files changed:


  • eview-enrichment/package.json\ — added shell-level retry to the \ est\ script

  • eview-enrichment/test/sentry-degradation.test.ts\ — raised \�udget.timeoutMs\ from 200 to 2000 and updated the \capturedTimeoutMs\ assertion bound

Validation:


  • pm --prefix review-enrichment test\ — 358/360 pass (2 sentry-upload tests fail locally due to missing sentry-cli binary; pass in CI)
  • \git diff --check\ clean

Linked issue: Closes #2425

Node's built-in test runner (used by review-enrichment) has no --test-retries
flag (unlike vitest's retry: 1 in the main suite), so add a shell-level retry
that reruns the full suite once on failure — matching the main suite's
one-retry-before-fail policy.

Root cause of the observed flake in 'buildBrief stays fail-open and captures
a degraded analyzer': the test used budget.timeoutMs=200, but the balanced
profile's responseReserveMs floors at 150ms, leaving only ~50ms of execution
headroom. Under CI load the pre-flight budget gate in runAnalyzer
(brief.ts:248-262) short-circuits to 'capped' before the analyzer runs at
all, so the expected 'degraded' status is never reached. Raising the budget
to 2000ms gives ~1600ms of headroom, deterministically reaching the analyzer
throw path.

Closes JSONbored#2425
@RenzoMXD
RenzoMXD requested a review from JSONbored as a code owner July 1, 2026 23:29
@dosubot dosubot Bot added the size:XS label Jul 1, 2026
@loopover-orb

loopover-orb Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Warning

🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨

⏸️ Gittensory review result - manual review recommended

Review updated: 2026-07-02 04:31:32 UTC

2 files · 1 AI reviewer · no blockers · readiness 55/100 · CI green · clean

⏸️ Suggested Action - Manual Review

Review summary
The change cleanly separates the Node test invocation into `test:node` and retries that portion once, which addresses the flaky REES test runner path without retrying build or validation steps. The Sentry degradation test now uses a large enough timeout budget to exercise the analyzer failure path instead of the pre-flight budget cap, and the updated assertion remains tied to the configured budget. No visible diff issue breaks the package scripts or the adjusted test.

Nits — 4 non-blocking
  • nit: `review-enrichment/package.json:17` retries only `test:node`, while the PR description says the full suite is rerun once; either tighten the description or make the script retry the complete `test` command body.
  • In `review-enrichment/package.json:17`, consider naming the retry wrapper explicitly, e.g. `test:node:retry`, so future edits do not accidentally assume `test:node` itself contains retry behavior.
  • In `review-enrichment/test/sentry-degradation.test.ts:219`, a short comment explaining that `2000` avoids the pre-flight budget cap would preserve the root-cause context from the PR description.
  • Readiness score is below the configured threshold — Use the readiness panel as advisory maintainer context; the score does not block this PR.
Signal Result Evidence
Code review ✅ No blockers 1 reviewer
Linked issue ✅ Linked #2425
Related work ⚠️ 3 scoped overlaps Top overlaps are listed below; lower-confidence bulk is hidden.
Change scope ❌ 8/20 High review scope from cached public metadata (size label size:XS; 1 linked issue).
Validation posture ❌ 5/25 Preflight is holding this PR: the review lane is unavailable, so it is not ready for automated review.
Contributor workload ✅ 10/10 Author activity: 69 registered-repo PR(s), 32 merged, 7 issue(s).
Contributor context ✅ Confirmed Gittensor contributor RenzoMXD; Gittensor profile; 69 PR(s), 7 issue(s).
Gate result ✅ Passing No configured blocker found.
Review context
Contributor next steps
  • Review top overlaps.
  • Add a concise scope and risk note.
  • Await review-lane availability.
  • Triage stale or unlinked PRs.
  • Refresh registry data or choose a registered active repo.
  • Check active issues and PRs before submitting.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

  • Re-run Gittensory review

@loopover-orb loopover-orb Bot added gittensor gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. labels Jul 1, 2026
@dosubot dosubot Bot added the lgtm label Jul 2, 2026
@JSONbored
JSONbored merged commit 30215fe into JSONbored:main Jul 2, 2026
7 checks passed
@github-project-automation github-project-automation Bot moved this from Todo to Done in gittensory - v1 roadmap Jul 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. gittensor:priority Maintainer-selected Gittensor priority — scores a 1.5x multiplier.

Projects

No open projects
Status: Done

Development

Successfully merging this pull request may close these issues.

test(enrichment): add flaky-test retry to the review-enrichment suite

2 participants