ci(qa): cross-run history accumulation + must-pass gate; development on push (JG-18) - #70
Merged
Merged
Conversation
…on push (JG-18) The harder half of JG-18: per-run `bun:sqlite` results shipped in #52, but nothing accumulated them or gated on them. This adds both, and promotes `development` into the push matrix now that JG-31 (#68) is CHR-passed. - `qa.yaml` push/schedule active set is now `[stable, long-term, development]`. Best-effort (non-released) legs are `continue-on-error`, so a beta btest/EC-SRP5 flake never reds main; the must-pass authority is a new `accumulate-and-gate` job, not the individual legs. - New `accumulate-and-gate` job (`needs: chr-matrix`, `if: always()`): downloads every per-leg result, appends each run to a durable append-log on the `qa-history` orphan branch (per-run artifacts have finite retention; the channel→version drift is what long history captures), and fails only on a released-channel regression. History is written before the gate verdict so the `if: always()` commit persists even a failing run. - Policy lives once in `scripts/qa-results-db.ts` (`MUST_PASS_CHANNELS`, `channelPolicy`, `evaluateMustPassGate`) + JSONL round-trip helpers, mirrored by the matrix `continue-on-error`. Cross-run accumulator is the thin `scripts/qa-history.ts`. Both unit-tested (gate verdicts, JSONL skip/rebuild, end-to-end accumulation across runs). - Tiers instruction + CHANGELOG updated to match. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughAdds a durable QA run history system: ChangesQA Cross-Run History and Must-Pass Gate
Sequence Diagram(s)sequenceDiagram
participant Push as Push / Schedule event
participant Matrix as chr-test (stable, long-term, development)
participant Artifacts as GitHub Artifacts Store
participant Gate as accumulate-and-gate job
participant Script as scripts/qa-history.ts
participant Branch as qa-history branch
Push->>Matrix: trigger CHR jobs
Matrix->>Artifacts: upload qa-results-{channel} (sqlite)
Artifacts->>Gate: download all qa-results-*
Gate->>Branch: fetch or create orphan qa-history worktree
Gate->>Script: run --legs --history qa-runs.jsonl --summary
Script->>Script: collectLegRuns → evaluateMustPassGate
Script->>Branch: append runs to qa-runs.jsonl
Script-->>Gate: exit 0 (pass) or 1 (stable/long-term regressed)
Gate->>Branch: commit qa-runs.jsonl (always, rebase-retry push)
Gate-->>Push: fail main only on released-channel regression
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
Adds cross-run QA history accumulation and a must-pass gate to the qa.yaml workflow, persisting CHR results across runs and gating merges only on released-channel regressions (stable/long-term) while keeping development best-effort.
Changes:
- Introduces a durable JSONL-backed accumulation flow (
qa-historybranch) and a must-pass gate evaluated per run. - Adds policy + JSONL round-trip utilities to the QA results DB module, with new unit coverage.
- Updates QA workflow + contributor docs/changelog to reflect the new active set and gating behavior.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
scripts/qa-results-db.ts |
Adds must-pass policy evaluation + JSONL history serialization/parsing helpers. |
scripts/qa-history.ts |
New accumulator/gate orchestrator that appends per-leg DB results to history and writes a summary. |
.github/workflows/qa.yaml |
Runs development on push/schedule; adds accumulate-and-gate job and best-effort leg behavior. |
test/unit/qa-results-db.test.ts |
Adds unit tests for channel policy, gate evaluation, and JSONL round-trip/rebuild. |
test/unit/qa-history.test.ts |
Adds unit tests for end-to-end accumulation + gating behavior across runs. |
CHANGELOG.md |
Documents QA accumulation + must-pass gate and development joining push matrix. |
GLOSSARY.txt |
Adds CI vocabulary entry for worktree used by the new QA job. |
.github/instructions/ci-test-tiers-and-release-versioning.instructions.md |
Updates CI tier documentation to match new QA workflow behavior. |
Comment on lines
+25
to
+36
| import { Database } from "bun:sqlite"; | ||
| import { Glob } from "bun"; | ||
| import { | ||
| allRuns, | ||
| channelPolicy, | ||
| channelStatuses, | ||
| evaluateMustPassGate, | ||
| parseHistoryJsonl, | ||
| type QaRunRow, | ||
| rebuildHistoryDb, | ||
| serializeRun, | ||
| } from "./qa-results-db.ts"; |
Comment on lines
+130
to
+151
| // Append this run to the accumulating history (written before the verdict so | ||
| // an `if: always()` commit persists even a failing run). | ||
| const existing = (await Bun.file(historyPath).exists()) | ||
| ? await Bun.file(historyPath).text() | ||
| : ""; | ||
| const appended = currentRuns.map(serializeRun).join("\n"); | ||
| const prefix = | ||
| existing && !existing.endsWith("\n") ? `${existing}\n` : existing; | ||
| const next = appended ? `${prefix}${appended}\n` : prefix; | ||
| await Bun.write(historyPath, next); | ||
|
|
||
| const historyDb = rebuildHistoryDb(parseHistoryJsonl(next)); | ||
| const summary = gateSummaryMarkdown( | ||
| currentRuns, | ||
| historyDb, | ||
| gate.ok, | ||
| gate.failures, | ||
| ); | ||
| const summaryPath = flag(args, "--summary"); | ||
| if (summaryPath) { | ||
| await Bun.write(summaryPath, summary); | ||
| } |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/qa.yaml:
- Around line 203-213: Replace the mutable action version tags with immutable
commit SHAs in the accumulate-and-gate job. Change actions/checkout@v7 to a
pinned commit SHA reference and change actions/download-artifact@v7 to a pinned
commit SHA reference, following the same pattern as the oven-sh/setup-bun action
which is already correctly pinned with a commit SHA. This reduces supply-chain
exposure by ensuring these actions cannot be unexpectedly modified.
In `@scripts/qa-history.ts`:
- Around line 44-50: The Database handle opened with `new Database(path, {
readonly: true })` is not guaranteed to be closed if `allRuns(db)` throws an
error. To fix this, add a `finally` block after the `catch` block to ensure
`db.close()` is always executed regardless of whether an error occurs during the
try block. This prevents file handle leaks when scanning many artifacts.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: f6f08f79-edd3-4d71-9470-73ba5654b6fb
📒 Files selected for processing (8)
.github/instructions/ci-test-tiers-and-release-versioning.instructions.md.github/workflows/qa.yamlCHANGELOG.mdGLOSSARY.txtscripts/qa-history.tsscripts/qa-results-db.tstest/unit/qa-history.test.tstest/unit/qa-results-db.test.ts
Comment on lines
+203
to
+213
| uses: actions/checkout@v7 | ||
| with: | ||
| fetch-depth: 0 | ||
| - name: Setup Bun 1.3.13 | ||
| uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # setup-bun action v2.2.0 | ||
| with: | ||
| bun-version: 1.3.13 | ||
| - name: Install dependencies | ||
| run: bun install --frozen-lockfile | ||
| - name: Download per-leg QA results | ||
| uses: actions/download-artifact@v7 |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Verify all unpinned workflow action references (tag-based uses).
rg -nP '^\s*uses:\s*[^@\s]+@v[0-9]+(\.[0-9]+){0,2}\s*$' .github/workflowsRepository: tikoci/centrs
Length of output: 1797
🏁 Script executed:
sed -n '190,215p' .github/workflows/qa.yamlRepository: tikoci/centrs
Length of output: 1010
Pin action references to immutable SHAs in the accumulate-and-gate job.
actions/checkout@v7 and actions/download-artifact@v7 are mutable tags. In this merge-gating pipeline, these should be commit-SHA pinned to reduce supply-chain exposure.
🧰 Tools
🪛 zizmor (1.25.2)
[error] 203-203: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
[error] 213-213: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/qa.yaml around lines 203 - 213, Replace the mutable action
version tags with immutable commit SHAs in the accumulate-and-gate job. Change
actions/checkout@v7 to a pinned commit SHA reference and change
actions/download-artifact@v7 to a pinned commit SHA reference, following the
same pattern as the oven-sh/setup-bun action which is already correctly pinned
with a commit SHA. This reduces supply-chain exposure by ensuring these actions
cannot be unexpectedly modified.
Source: Linters/SAST tools
Comment on lines
+44
to
+50
| try { | ||
| const db = new Database(path, { readonly: true }); | ||
| runs.push(...allRuns(db)); | ||
| db.close(); | ||
| } catch (error) { | ||
| console.error(`::warning title=QA history::skipped ${path}: ${error}`); | ||
| } |
There was a problem hiding this comment.
Ensure DB handles are closed on read failures.
If allRuns(db) throws, the current code skips db.close(), which can leak file handles while scanning many artifacts.
Suggested fix
for await (const rel of glob.scan({ cwd: legsDir, onlyFiles: true })) {
const path = `${legsDir}/${rel}`;
+ let db: Database | undefined;
try {
- const db = new Database(path, { readonly: true });
+ db = new Database(path, { readonly: true });
runs.push(...allRuns(db));
- db.close();
} catch (error) {
console.error(`::warning title=QA history::skipped ${path}: ${error}`);
+ } finally {
+ db?.close();
}
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| try { | |
| const db = new Database(path, { readonly: true }); | |
| runs.push(...allRuns(db)); | |
| db.close(); | |
| } catch (error) { | |
| console.error(`::warning title=QA history::skipped ${path}: ${error}`); | |
| } | |
| let db: Database | undefined; | |
| try { | |
| db = new Database(path, { readonly: true }); | |
| runs.push(...allRuns(db)); | |
| } catch (error) { | |
| console.error(`::warning title=QA history::skipped ${path}: ${error}`); | |
| } finally { | |
| db?.close(); | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/qa-history.ts` around lines 44 - 50, The Database handle opened with
`new Database(path, { readonly: true })` is not guaranteed to be closed if
`allRuns(db)` throws an error. To fix this, add a `finally` block after the
`catch` block to ensure `db.close()` is always executed regardless of whether an
error occurs during the try block. This prevents file handle leaks when scanning
many artifacts.
CodeRabbit/Copilot: if allRuns(db) throws while scanning a per-leg artifact, db.close() was skipped, leaking the handle. Move the close into a finally and stringify the error explicitly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
Author
Review dispositionsFixed —
Resolved on merit (declined, with reason):
|
…ot recency)
The file header still described the old intended policy ("current long-term and
newer must pass") — which is recency-based and contradicts what this file now
implements: the gate is maturity-based (released channels gate; pre-release is
best-effort regardless of how new its version is). Recency matters only for
which pre-release channels are worth sampling. Also note testing/development are
not monotonically ordered.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
mobileskyfi
added a commit
that referenced
this pull request
Jun 22, 2026
…arts (#77) The release-tier job calls qa.yaml as a reusable workflow. Since #70, qa.yaml's accumulate-and-gate job requests contents:write (it commits results to the qa-history branch). A called workflow can't be granted more permission than the calling job, and release.yaml's workflow default is contents:read, so the v0.1.0 tag run failed at startup ("workflow file issue"). Grant the calling release-tier job contents:write; the publish job keeps contents:read + id-token:write. First surfaced now because the 2026-06-19 release dry-run predated the qa.yaml accumulate job. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This was referenced Jun 23, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Track B of the June-Gloom closeout: the harder half of JG-18. Per-run
bun:sqliteresults shipped in #52, but nothing accumulated them across runs orgated on them. This adds both, and promotes
developmentinto the push matrixnow that JG-31 (#68) is CHR-passed.
What changed
Policy (one source of truth, unit-tested) —
scripts/qa-results-db.tsMUST_PASS_CHANNELS = [stable, long-term],channelPolicy,evaluateMustPassGate.Released channels are must-pass;
development/testingare best-effort.serializeRun,parseHistoryJsonl,rebuildHistoryDb) forthe durable append-log, plus
allRuns.Accumulator —
scripts/qa-history.ts(new, thin orchestrator)qa-results.sqlite, gates over this run(stale history never reds a fresh clean run), appends to the history JSONL,
writes a job-summary table, exits non-zero only on a released-channel fail.
Workflow —
.github/workflows/qa.yaml[stable, long-term, development].continue-on-error(mirrorschannelPolicy), so a betabtest/EC-SRP5 flake never reds main — the new
accumulate-and-gatejob is thesole merge-reddening authority.
accumulate-and-gate(needs: chr-matrix,if: always()): downloads per-legartifacts → worktree on the
qa-historyorphan branch (created on firstrun) → accumulate + gate → commit history (
if: always()+ main-only, with arebase-retry push). History is written before the gate verdict so a failing
run still lands in history.
Docs — tiers instruction
qa.yamlbullet + CHANGELOG Unreleased.Decisions (locked with maintainer)
qa-historyorphan branch (durable; artifact retention isfinite and channel→version drift is the point) — vs. ephemeral artifact round-trip.
development= run on push, best-effort (must-pass = stable + long-term) —honors the JG-31 promotion while keeping the "never red a merge on a beta flake"
guarantee.
Verification
bun run lint,bun run lint:ci,bun run test(735 pass / 0 fail),bun run build— all green.across runs (
test/unit/qa-results-db.test.ts,test/unit/qa-history.test.ts).qa.yamlCHR dispatch on this branch to exercise theaccumulate-and-gatewiring end-to-end on real CHR (orphan-branch creation +gate; the history push only runs post-merge on main). Will attach the run.
🤖 Generated with Claude Code
Summary by CodeRabbit