Skip to content

mirror release harness checks on push CI#342

Merged
ndycode merged 3 commits intomainfrom
audit/pr6-ci-harness-realism
Apr 1, 2026
Merged

mirror release harness checks on push CI#342
ndycode merged 3 commits intomainfrom
audit/pr6-ci-harness-realism

Conversation

@ndycode
Copy link
Copy Markdown
Owner

@ndycode ndycode commented Apr 1, 2026

Summary

  • make push CI enforce the same script, pack, and vendor checks that PR CI already treats as release blockers

What Changed

  • added a dedicated release-harness job to ci.yml for typecheck:scripts, pack:check, and vendor:verify
  • added test/ci-workflows.test.ts so future workflow edits have to keep those commands in both ci.yml and pr-ci.yml

Validation

  • npm run lint
  • npm run typecheck
  • npm test
  • npm test -- test/documentation.test.ts
  • npm run build
  • Focused tests: npx vitest run test/ci-workflows.test.ts

Docs and Governance Checklist

  • README updated (if user-visible behavior changed)
  • docs/getting-started.md updated (if onboarding flow changed)
  • docs/features.md updated (if capability surface changed)
  • relevant docs/reference/* pages updated (if commands/settings/paths changed)
  • docs/upgrade.md updated (if migration behavior changed)
  • SECURITY.md and CONTRIBUTING.md reviewed for alignment

Risk and Rollback

  • Risk level: Low
  • Rollback plan: Revert 5c9efed

Additional Notes

  • Keeping the push-only release harness in its own job avoids duplicating these checks across the Node version matrix.

note: greptile review for oc-chatgpt-multi-auth. cite files like lib/foo.ts:123. confirm regression tests + windows concurrency/token redaction coverage.

Greptile Summary

this pr mirrors the three release-blocker checks (typecheck:scripts, pack:check, vendor:verify) from pr-ci.yml into push CI by adding a dedicated release-harness job and a scripts-windows job to ci.yml, plus a test file that enforces parity between the two workflow files going forward.

  • ci.yml gains permissions: contents: read, a concurrency: cancel-in-progress: true block (scoped to ci-${{ github.ref }} so rapid pushes to main cancel stale runs), release-harness (ubuntu, node 20), and scripts-windows (windows-latest, node 20) — all mirroring existing pr-ci.yml jobs
  • test/ci-workflows.test.ts adds three vitest cases: command parity, concurrency-block parity, and windows-job parity; uses a local extractJobBlock helper to slice job boundaries by 2-space-indented yaml keys
  • the extractJobBlock regex [a-z0-9][a-z0-9-]* only matches lowercase job names — adding A-Z or the i flag prevents silent over-extraction if a camelCase job is ever added
  • the first parity test uses whole-file toContain rather than scoped job-block assertions, so a command string surviving in a disabled or conditional job would still satisfy the test
  • windows filesystem token safety: readFileSync here is read-only and path-joined via node:path, so no EBUSY/EPERM exposure
  • npm test was left unchecked in the pr checklist; the focused npx vitest run test/ci-workflows.test.ts run covers the new file but the full suite hasn't been confirmed green on this branch

Confidence Score: 5/5

safe to merge — changes are additive CI jobs and a test file with no production code impact

all findings are P2: a lowercase-only regex in a test helper and a whole-file containment assertion that could be scoped more tightly. neither affects current correctness — all three commands are present in the correct jobs and the tests will pass as-is.

test/ci-workflows.test.ts — the extractJobBlock regex and containment-scope issue are worth tightening before this test file accrues further parity checks

Important Files Changed

Filename Overview
.github/workflows/ci.yml adds top-level permissions: contents: read and concurrency: cancel-in-progress: true, plus release-harness and scripts-windows jobs that mirror the three release-blocker checks already in pr-ci.yml; no logic issues found
test/ci-workflows.test.ts new parity test file; extractJobBlock regex only handles lowercase job names (P2), and the first test uses whole-file toContain rather than scoped job-block assertions, making it easier to accidentally satisfy with orphaned command strings

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    push["push to main"] --> wf["ci.yml workflow"]
    wf --> test["test (Node 20.x / 22.x matrix)\ntypecheck + coverage + build"]
    wf --> lint["lint\nnpm run lint"]
    wf --> rh["release-harness (NEW)\ntypecheck:scripts\npack:check\nvendor:verify"]
    wf --> sw["scripts-windows (NEW)\ntypecheck:scripts\n(windows-latest)"]
    wf --> cc["codex-compat\nsmoke tests"]
    style rh fill:#d4edda,stroke:#28a745
    style sw fill:#d4edda,stroke:#28a745
Loading

Fix All in Codex

Prompt To Fix All With AI
This is a comment left during a code review.
Path: test/ci-workflows.test.ts
Line: 15-16

Comment:
**regex silently drops uppercase job names**

the pattern `[a-z0-9][a-z0-9-]*` only matches lowercase-hyphen job names. if anyone adds a camelCase or PascalCase job (e.g. from a reusable-workflow call), `extractJobBlock` will fall through to `workflow.length` instead of finding the real boundary — potentially asserting against a block that spans multiple jobs and producing a false-passing test.

```suggestion
	const nextMatch = workflow
		.slice(start + 1)
		.match(/\n  [a-zA-Z0-9][a-zA-Z0-9-]*:\n/);
```

How can I resolve this? If you propose a fix, please make it concise.

---

This is a comment left during a code review.
Path: test/ci-workflows.test.ts
Line: 23-36

Comment:
**whole-file containment check can pass on orphaned strings**

`expect(ci).toContain(command)` is satisfied even if the command appears in a disabled job, a commented-out block, or a different workflow trigger branch. if one of these three commands migrates out of `release-harness` into a `release-only` gate with a different trigger, the test stays green. scoping the assertion to the `release-harness` job block (using the `extractJobBlock` helper already written below) would make this significantly harder to accidentally break.

```suggestion
	it("keeps push CI aligned with the PR release harness checks", () => {
		const ci = readWorkflow("ci.yml");
		const prCi = readWorkflow("pr-ci.yml");
		const ciReleaseJob = extractJobBlock(ci, "release-harness");
		const prValidateJob = extractJobBlock(prCi, "validate");
		const requiredCommands = [
			"npm run typecheck:scripts",
			"npm run pack:check",
			"npm run vendor:verify",
		];

		for (const command of requiredCommands) {
			expect(prValidateJob).toContain(command);
			expect(ciReleaseJob).toContain(command);
		}
	});
```

How can I resolve this? If you propose a fix, please make it concise.

Reviews (3): Last reviewed commit: "ci: tighten push workflow parity" | Re-trigger Greptile

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented Apr 1, 2026

📝 Walkthrough

Severity: Low-risk infrastructure change. This PR mirrors three release-blocker checks (typecheck:scripts, pack:check, vendor:verify) from PR workflows onto push CI in .github/workflows/ci.yml and adds a Windows-specific typecheck job for cross-platform validation. The changes strengthen push-time safety gates with low regression risk, as they are purely CI workflow additions without modifications to application code or public APIs.

Test Coverage: A new Vitest test suite (test/ci-workflows.test.ts) is introduced to enforce command parity between ci.yml and pr-ci.yml. However, the automated review notes that the Windows job coverage test uses independent substring matching and does not guarantee co-location of the runs-on: windows-latest and npm run typecheck:scripts step within the same job block—potentially allowing a false pass if either element drifts elsewhere in the workflow.

Workflow Changes:

  • New release-harness job (Ubuntu, Node 20): Runs the three release-blocker checks on every push to enforce parity with PR workflows.
  • New scripts-windows job (Windows, Node 20): Ensures typecheck:scripts validation occurs on Windows platforms, addressing cross-platform correctness.
  • Workflow-level configuration: Adds permissions: contents: read for least-privilege access.

Noted Parity Gap: The PR acknowledges that ci.yml still lacks the concurrency: cancel-in-progress: true configuration present in pr-ci.yml, flagged as a remaining improvement opportunity.

Validation Status: Lint, typecheck, and build checks passed; focused unit test execution confirmed the workflow-parity tests run successfully. Full test suite validation remains incomplete per PR checklist.

Walkthrough

workflow configuration adds permissions, concurrency controls, and two new ci jobs (release-harness and scripts-windows). new test file enforces parity between ci.yml and pr-ci.yml workflows, verifying shared commands and platform-specific coverage.

Changes

Cohort / File(s) Summary
Workflow Configuration
.github/workflows/ci.yml
Added permissions: contents: read and concurrency with cancel-in-progress: true. Introduced release-harness job (ubuntu, node 20) running typecheck, pack, and vendor checks. Added scripts-windows job (windows, node 20) for platform-specific script typechecking.
Workflow Parity Testing
test/ci-workflows.test.ts
New test suite validating that ci.yml and pr-ci.yml both include required npm commands (typecheck:scripts, pack:check, vendor:verify) and concurrency settings. Verifies scripts-windows job configuration across both workflows.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes


notes:

  • flag: test/ci-workflows.test.ts:1-60 reads and parses yaml manually. consider using a proper yaml parser to avoid brittle line-based extraction logic if job ordering changes.

  • flag: windows edge case — scripts-windows job only runs typecheck:scripts, while release-harness runs three commands. need to verify this intentional split and whether windows should also run pack/vendor checks.

  • flag: concurrency risk — concurrency: ci-${{ github.ref }} will cancel runs for the same branch. document expected behavior if simultaneous workflows hit the workflow gate (e.g., stale artifact state).

  • missing regression test: no assertions validating that existing test, lint, codex-compat jobs remain unchanged. consider snapshot testing the entire workflow structure to catch unintended job removals.

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Title check ✅ Passed title follows conventional commits format (type: summary), is 40 chars, and accurately describes the main change of mirroring release checks to push CI.
Description check ✅ Passed description covers summary, what changed, validation status, docs checklist, risk/rollback, and notes; mostly complete despite unchecked test items and a noted concurrency parity gap.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch audit/pr6-ci-harness-realism
✨ Simplify code
  • Create PR with simplified code
  • Commit simplified code in branch audit/pr6-ci-harness-realism

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@chatgpt-codex-connector
Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@test/ci-workflows.test.ts`:
- Around line 39-47: The test currently uses substring checks on strings ci and
prCi (from readWorkflow) which can falsely pass; update the test "keeps push CI
using the same stale-run cancellation as PR CI" to parse the workflow YAML into
an object (e.g., via readWorkflow returning parsed YAML or by passing the string
into a YAML parser) and assert the top-level concurrency block exists and
contains cancel-in-progress: true for both prCi and ci (check prCi.concurrency
and ci.concurrency and their cancel-in-progress property) instead of using
expect(...).toContain on raw strings so the assertion is scoped to the
workflow-level concurrency block.
- Around line 24-37: The test currently only checks that each command in
requiredCommands appears somewhere in ci and prCi strings; instead parse the
workflow YAML (using the existing readWorkflow result stored in ci and prCi) and
assert that there exists a single job whose steps (or run commands) contain all
three requiredCommands together. Locate and update the test block (the it("keeps
push CI aligned with the PR release harness checks", ...) that uses variables
ci, prCi and requiredCommands) to load the YAML for each workflow, iterate jobs,
and assert at least one job per workflow has all requiredCommands present in its
step run texts.
- Around line 7-21: The extractJobBlock function's boundary regex only matches
lowercase-hyphen IDs and Unix newlines, causing failures for valid job ids
(e.g., foo_bar) and CRLF files; update the nextMatch regex in extractJobBlock to
accept CRLF or LF and a broader job-id charset (e.g., /\r?\n 
[A-Za-z0-9_-]+:\r?\n/) so it detects lines like "  JOB_ID:" across platforms and
with underscores, and keep using the existing start/nextMatch math to compute
end and return the slice.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 67ecee1c-de05-4dbf-905b-0f67587ab142

📥 Commits

Reviewing files that changed from the base of the PR and between b9c9273 and 23e5431.

📒 Files selected for processing (2)
  • .github/workflows/ci.yml
  • test/ci-workflows.test.ts
📜 Review details
🧰 Additional context used
📓 Path-based instructions (1)
test/**

⚙️ CodeRabbit configuration file

tests must stay deterministic and use vitest. demand regression cases that reproduce concurrency bugs, token refresh races, and windows filesystem behavior. reject changes that mock real secrets or skip assertions.

Files:

  • test/ci-workflows.test.ts
🔇 Additional comments (4)
.github/workflows/ci.yml (3)

7-13: good hardening at workflow scope.

top-level permissions: contents: read and stale-run cancellation are correctly set for push ci. citation: lib/.github/workflows/ci.yml:7-13.


78-103: release harness job is correctly wired to release blockers.

this job runs the exact three commands needed for push-side parity and uses pinned node 20.x setup consistently. citation: lib/.github/workflows/ci.yml:78-103.


104-123: windows coverage for script typecheck is correctly added.

the dedicated windows job is scoped and minimal, and it executes npm run typecheck:scripts as intended. citation: lib/.github/workflows/ci.yml:104-123.

test/ci-workflows.test.ts (1)

49-59: windows parity intent is solid here.

good use of per-job extraction before asserting runs-on and typecheck:scripts; this avoids the prior cross-job false positive class. citation: test/ci-workflows.test.ts:49-59.

Comment thread test/ci-workflows.test.ts
Comment on lines +7 to +21
function readWorkflow(name: string): string {
return readFileSync(join(projectRoot, ".github", "workflows", name), "utf-8");
}

function extractJobBlock(workflow: string, jobName: string): string {
const start = workflow.indexOf(` ${jobName}:`);
if (start === -1) {
throw new Error(`Missing workflow job: ${jobName}`);
}
const nextMatch = workflow
.slice(start + 1)
.match(/\n [a-z0-9][a-z0-9-]*:\n/);
const end = nextMatch ? start + 1 + nextMatch.index + 1 : workflow.length;
return workflow.slice(start, end);
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

make job-block extraction deterministic across windows and valid job ids.

line [18] only matches lowercase-hyphen ids and \n, so a valid job id like foo_bar or crlf input can break block boundaries and create false positives. citation: test/ci-workflows.test.ts:11-20.

proposed fix
 function readWorkflow(name: string): string {
-	return readFileSync(join(projectRoot, ".github", "workflows", name), "utf-8");
+	return readFileSync(join(projectRoot, ".github", "workflows", name), "utf-8").replace(/\r\n/g, "\n");
 }

 function extractJobBlock(workflow: string, jobName: string): string {
 	const start = workflow.indexOf(`  ${jobName}:`);
 	if (start === -1) {
 		throw new Error(`Missing workflow job: ${jobName}`);
 	}
 	const nextMatch = workflow
 		.slice(start + 1)
-		.match(/\n  [a-z0-9][a-z0-9-]*:\n/);
+		.match(/\n  [A-Za-z0-9_][A-Za-z0-9_-]*:\n/);
 	const end = nextMatch ? start + 1 + nextMatch.index + 1 : workflow.length;
 	return workflow.slice(start, end);
 }

as per coding guidelines, test/**: tests must stay deterministic and use vitest. demand regression cases that reproduce concurrency bugs, token refresh races, and windows filesystem behavior. reject changes that mock real secrets or skip assertions.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@test/ci-workflows.test.ts` around lines 7 - 21, The extractJobBlock
function's boundary regex only matches lowercase-hyphen IDs and Unix newlines,
causing failures for valid job ids (e.g., foo_bar) and CRLF files; update the
nextMatch regex in extractJobBlock to accept CRLF or LF and a broader job-id
charset (e.g., /\r?\n  [A-Za-z0-9_-]+:\r?\n/) so it detects lines like " 
JOB_ID:" across platforms and with underscores, and keep using the existing
start/nextMatch math to compute end and return the slice.

Comment thread test/ci-workflows.test.ts
Comment on lines +24 to +37
it("keeps push CI aligned with the PR release harness checks", () => {
const ci = readWorkflow("ci.yml");
const prCi = readWorkflow("pr-ci.yml");
const requiredCommands = [
"npm run typecheck:scripts",
"npm run pack:check",
"npm run vendor:verify",
];

for (const command of requiredCommands) {
expect(prCi).toContain(command);
expect(ci).toContain(command);
}
});
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

assert release-blocker commands are co-located in the intended job block.

current checks only prove each command exists somewhere in each workflow, not that one release-harness-style job contains all three. this can miss parity regressions. citation: test/ci-workflows.test.ts:24-37.

proposed fix
 	it("keeps push CI aligned with the PR release harness checks", () => {
 		const ci = readWorkflow("ci.yml");
 		const prCi = readWorkflow("pr-ci.yml");
+		const ciReleaseHarness = extractJobBlock(ci, "release-harness");
+		const prValidate = extractJobBlock(prCi, "validate");
 		const requiredCommands = [
 			"npm run typecheck:scripts",
 			"npm run pack:check",
 			"npm run vendor:verify",
 		];

 		for (const command of requiredCommands) {
-			expect(prCi).toContain(command);
-			expect(ci).toContain(command);
+			expect(ciReleaseHarness).toContain(command);
+			expect(prValidate).toContain(command);
 		}
 	});

as per coding guidelines, test/**: tests must stay deterministic and use vitest. demand regression cases that reproduce concurrency bugs, token refresh races, and windows filesystem behavior. reject changes that mock real secrets or skip assertions.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@test/ci-workflows.test.ts` around lines 24 - 37, The test currently only
checks that each command in requiredCommands appears somewhere in ci and prCi
strings; instead parse the workflow YAML (using the existing readWorkflow result
stored in ci and prCi) and assert that there exists a single job whose steps (or
run commands) contain all three requiredCommands together. Locate and update the
test block (the it("keeps push CI aligned with the PR release harness checks",
...) that uses variables ci, prCi and requiredCommands) to load the YAML for
each workflow, iterate jobs, and assert at least one job per workflow has all
requiredCommands present in its step run texts.

Comment thread test/ci-workflows.test.ts
Comment on lines +39 to +47
it("keeps push CI using the same stale-run cancellation as PR CI", () => {
const ci = readWorkflow("ci.yml");
const prCi = readWorkflow("pr-ci.yml");

expect(prCi).toContain("concurrency:");
expect(prCi).toContain("cancel-in-progress: true");
expect(ci).toContain("concurrency:");
expect(ci).toContain("cancel-in-progress: true");
});
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

scope the concurrency assertion to the top-level block.

line [43]-line [46] use global substring checks, so the test can pass even if cancel-in-progress: true is moved out of the workflow-level concurrency section. citation: test/ci-workflows.test.ts:39-47.

as per coding guidelines, test/**: tests must stay deterministic and use vitest. demand regression cases that reproduce concurrency bugs, token refresh races, and windows filesystem behavior. reject changes that mock real secrets or skip assertions.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@test/ci-workflows.test.ts` around lines 39 - 47, The test currently uses
substring checks on strings ci and prCi (from readWorkflow) which can falsely
pass; update the test "keeps push CI using the same stale-run cancellation as PR
CI" to parse the workflow YAML into an object (e.g., via readWorkflow returning
parsed YAML or by passing the string into a YAML parser) and assert the
top-level concurrency block exists and contains cancel-in-progress: true for
both prCi and ci (check prCi.concurrency and ci.concurrency and their
cancel-in-progress property) instead of using expect(...).toContain on raw
strings so the assertion is scoped to the workflow-level concurrency block.

@ndycode ndycode merged commit 95a153e into main Apr 1, 2026
2 checks passed
@ndycode ndycode deleted the audit/pr6-ci-harness-realism branch April 1, 2026 18:17
ndycode added a commit that referenced this pull request Apr 6, 2026
mirror release harness checks on push CI
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant