feat(action): refactor github action input surface around args; harden env passthrough

[bai-uipath]

Upstream half of giving UiPath/skills' run-coder-eval dispatch a real evalboard link while moving it onto the published composite action. Along the way the action's input surface got rewritten rather than extended. The skills-side PR consumes it and is blocked on this one being merged and released, since uses: UiPath/coder_eval@v0 reads its inputs from the action at that tag.

Why the input surface changed instead of growing

The action had no real consumers, only this repo's own action-dogfood job and the synthetic e2e in verify-published-action.yml, so a bad release surfaced in a self-test rather than in someone's workflow. Making run-coder-eval a real dependent meant adding inputs for a suite in a subdirectory, an agent extra, and a plugin inside the environment the CLI runs from. Five new inputs took the count to fifteen, and at that point the shape of the thing was the problem, not the gaps.

coder-eval run has 21 flags. The action promoted five of them to named inputs with no principle behind the choice: --tags got one, its sibling --exclude-tags did not, and extra-args's own description admitted it covered "--tags exclusions". A forwarding input buys nothing and costs a lot, because GitHub silently ignores an input the referenced tag does not define. One that is mistyped, or newer than the consumer's pin, yields a run that measured something else and still exits 0. A wrong CLI flag is a hard error instead.

The surface: 15 inputs to 8, none of them a CLI flag

An input now exists only where the action does something with the value besides pass it along.

Kept	Because
version, extras, extra-packages, install-flags	compose the install spec, which the action owns
working-directory	applied to the action's own steps; illegal on a uses: step, and job-level defaults.run does not reach into a composite
env	exported into the action's own shell, so a forwarded secret never touches $GITHUB_ENV
args	the single passthrough
run-dir	read back for the outputs

Removed	Where it went
tasks, tags, model, extra-args	all fold into args
prerelease	generalises to install-flags, which also covers --extra-index-url for a private index
junit-path	derived as <run-dir>/junit.xml, which is where both existing consumers already put it
step-summary	replaced by a run-md-path output. A consumer that has to redact the report first cannot undo a write that already happened, so the write is the consumer's call. cat "$RUN_MD" >> "$GITHUB_STEP_SUMMARY" is the whole of the default this replaces
minimum-task-score	deleted, not relocated — see below

Four different encodings (whitespace-split, line-delimited, comma-separated, NAME=VALUE) collapse to one clean_lines parser, copied into both step scripts because they are separate bash processes, with a test asserting the copies stay byte-identical.

Folding task globs into args removes a hazard

They now reach the CLI unexpanded and expand_task_files handles them. Verified directly:

recursive glob : 48 files      # **/*.yaml — no globstar needed
flat glob      : 21 files
non-matching   : exits 1       # "No task files found!"

So three documented "sharp edges" are deleted rather than reworded: ** silently degrading with globstar off (which was silently dropping every top-level task), a non-matching glob reaching the CLI as a literal path, and the need for an explicit per-depth ladder.

The other correctness win is the one that motivated args in the first place, reproduced with one touch:

$ touch 'sandbox.docker.env_passthrough_extra=A'
# whitespace-split (the old extra-args):
  [-D] [sandbox.docker.env_passthrough_extra=A]
# args:
  [-D] [sandbox.docker.env_passthrough_extra=[AUTH_TOKEN,BASE_URL]]

[...] is a bash character class. Silent, and it changes what the run measures. That path no longer exists.

The one capability this removes

minimum-task-score and its ~50 lines of embedded Python are gone with no replacement. Both call sites set it to "0.0", a no-op, so it had no real user. It is also in the wrong place: a score floor is policy over run.json, so it belongs in the CLI as --min-task-score, where it is unit-testable and reachable from the ADO pipelines and the nightly VM, which cannot use a GitHub action at all and reimplement their own gating today. Small follow-up, deliberately not in this PR.

Evalboard side

A third source, gha -> container runs-gha, registered and deliberately unlisted: no tab, no listing page, no aggregate view. Registration and surfacing turn out to be independent (NAV is a hardcoded array that does not iterate SOURCES), and app/runs/[id] reads by id on demand, so a direct link resolves with no listing in existence. That is what lets this skip the one piece of evalboard work with a documented footgun.

Its own container rather than a prefix inside runs, because getAdhocRunListing loads per-run metadata for every non-date-shaped id before truncating to the display limit, and because the storage account now carries a 14-day expiry rule scoped to runs-gha/ that must never be able to reach nightly history.

Three stale surfaces found while trimming the comments

Each one described an input that no longer exists, and each was invisible to the pinning lint:

• The ci skill still told the agent to pass an experiment via extra-args:, so an emitted workflow would have had its experiment silently ignored — exactly the forwarding-input failure this PR is about, aimed at the workflows the plugin writes. Now -e and the path as two args lines.
• docs/CI_GATE.md carried "Extras and plugins" twice; the second copy predated the rewrite and still named prerelease. Its frontmatter and intro also still promised a per-task score floor and a job-summary write.
• test_ci_skill_does_not_recommend_a_recursive_task_glob enforced the opposite of the new behaviour: it banned ** and required the skill to explain globstar. It was passing only because the rewritten skill contains the phrase "no globstar caveat". Removed — CE026's unknown-input check already covers a stale tasks: in a snippet.

After merging

Two dispatches, in this order, before the skills PR can do anything:

1. Release (release.yml), which bumps the version, re-pins action.yml, and moves v0. Until v0 moves, a consumer passing these inputs gets them silently ignored. Note the reverse window too: between this merge and that release, verify-published-action.yml on main passes the new with: block to the old @v0, so its nightly runs the same single task on the default model instead of haiku. Cheap, not broken, but a reason to release promptly.
2. deploy-evalboard.yml in coder_eval_uipath (manual dispatch, no post-deploy health check). The gha source only exists on the deployed site once that runs, so sequence it before the first upload or the emitted link 404s on arrival.

Verification

• tests/test_action_inputs.py (37 cases) executes both step scripts pulled out of action.yml with uv/coder-eval stubbed to record argv, so it asserts the text that ships rather than a copy. Mutation-checked: flattening the junit derivation fails 5 of them.
• The stub is written in bash, not Python, because Git Bash rewrites absolute-POSIX-looking arguments on the way to a native binary, so a python-shebang stub mangled /action-checkout on the Windows runner.
• CE026, the repo's own lint rule for doc snippets passing inputs action.yml does not declare, caught a surface I had missed (plugins/coder-eval/skills/ci/SKILL.md). Worth noting the rule earned its keep.
• action-dogfood covers what a unit test cannot: working-directory on a composite step, a plugin installed through extra-packages being discovered at runtime, and now the run-md-path summary-append recipe the docs tell consumers to use. The discovery probe is a coder-eval plan on a task naming the BYOA fixture's agent kind, which exits 1 with No agent registered for type 'byoa-demo' if the entry point was not found.
• Full local sweep: 426 lint/action tests, 608 evalboard vitest cases, tsc --noEmit, next build, ruff and pyright clean. actionlint findings unchanged against baseline on every workflow touched.

🤖 Generated with Claude Code

[github-actions]

Claude finished @bai-uipath's task in 1m 32s —— View job

---

Code Review

Todo List

• Read code review guidelines (.github/code_review.md)
• Read project conventions (CLAUDE.md)
• Analyze the full diff
• Review each changed file with full context
• Perform cross-file consistency checks
• Complete "what's missing" analysis
• Provide comprehensive review feedback

Starting review...

[uipreliga]

Review: coder_eval — pr:147 (13 files) axis:1,2,3,4,5,6,7,8

Scope: pr:147 (13 files) axis:1,2,3,4,5,6,7,8 · branch bai/action-inputs-and-gha-source · a747ce3 · 2026-09-01T22:00Z · workflow variant

Change class: complex — a breaking redesign of the published GitHub Action's input surface (eight inputs, no CLI-flag passthrough) plus new argv-construction semantics in action.yml and a new unlisted evalboard source, so correctness needs reasoning about shell word-splitting, input plumbing, and consumer-visible contracts

The Python evaluation core remains excellent — clean typing, strong architectural lint enforcement, and a harness axis with zero findings — but this change concentrates its risk in the shipped CI surface, where an unguarded env passthrough can rewrite the action's own argv and PATH, a stale glob can turn a green gate into an unmeasured subset, and seven removed inputs (including the score floor) reach @v0 consumers with no alias, no migration note, and no version signal; the bottom line is that the code is healthy while the published contract is not, so fix the gate-correctness and input-migration items before the next release moves the major tag.

Summary

Axis	Score	🔴	🟠	🟡	🔵	Top Issue
1. Code Quality & Style	8.8 / 10	0	0	2	2	Caller-supplied --run-dir/--junit-xml in args last-win over the action's injected copies while run-dir/junit-path/run-md-path outputs still advertise the input-derived paths
2. Type Safety	9.9 / 10	0	0	0	1	_step_script declares -> str but returns an unvalidated Any out of yaml.safe_load, and indexes a shape it never checks
3. Test Health	8.9 / 10	0	0	2	1	BASH = shutil.which("bash") in the new test module is resolved naively — `str
4. Security	9.2 / 10	0	0	1	3	env passthrough exports into the step's own shell before argv is built, so an entry named PATH / CE_ARGS / CE_RUN_DIR / GITHUB_OUTPUT hijacks the step (action.yml:216)
5. Architecture & Design	9.4 / 10	0	0	1	1	The deleted minimum-task-score capability is still advertised in six surfaces, including the published plugin skill's activation description and a tutorial cross-reference to a now-deleted section
6. Error Handling & Resilience	8.4 / 10	0	1	1	1	Docs and the ci skill promise "a glob matching nothing fails loudly / exits 1", but with the multi-line args shape the same section recommends, a non-matching pattern silently vanishes and the gate goes green on a subset
7. API Surface & Maintainability	7.4 / 10	0	2	1	1	Seven documented action inputs removed with no alias and no consumer-facing migration note — existing @v0 consumers break with an error naming neither the removed input nor the migration
8. Evaluation Harness Quality	10 / 10	0	0	0	0	—

Overall Score: 9 / 10 · Weakest Axis: API Surface & Maintainability at 7.4 / 10
Totals: 🔴 0 · 🟠 3 · 🟡 8 · 🔵 10 across 8 axes.

Blockers

1. [Axis 6] Docs and the ci skill promise "a glob matching nothing fails loudly / exits 1", but with the multi-line args shape the same section recommends, a non-matching pattern silently vanishes and the gate goes green on a subset (docs/CI_GATE.md:90) — docs/CI_GATE.md:90-91 states: - **A glob matching nothing exits 1** with \No task files found!`, rather than / reaching the CLI as a literal path or vanishing. and plugins/coder-eval/skills/ci/SKILL.md:104-107 repeats it (- A glob matching nothing fails loudly with `No task files found!` and exit 1…- One path per line. Two globs are two lines). That guarantee only holds when NO pattern matches anything. expand_task_files` (src/coder_eval/cli/run_helpers.py:81-94) accumulates matches across all patterns and raises only when the union is empty:

    if not all_task_files:
        console.print("[red]No task files found![/red]")
        raise typer.Exit(1)

Verified directly: expand_task_files([Path('a/*.yaml'), Path('b/*.yaml')]) with only a/ populated returns [PosixPath('a/t1.yaml')] and exits 0. So a two-line args: block where one glob is stale (renamed/moved suite) runs the surviving subset, the CLI exits 0, and the CI gate is green having never measured the dropped tasks — the exact silent-subset failure the old globstar paragraph existed to prevent. Either fix expand_task_files to fail on any pattern that matched nothing (and keep the doc claim), or correct both surfaces to say that a non-matching entry is silently dropped whenever another entry matches, and tell integrators to assert the task count.
2. [Axis 7] Seven documented action inputs removed with no alias and no consumer-facing migration note — existing @v0 consumers break with an error naming neither the removed input nor the migration (action.yml:24) — git diff of the inputs: block shows seven removals — tasks, tags, model, extra-args, junit-path, step-summary, minimum-task-score — and five additions. Nothing detects a consumer still passing an old name: action.yml's own header states the hazard (line 18-19: "GitHub silently / # IGNORES an input the referenced tag does not define") but the action never acts on it, and there is no INPUT_* introspection, no alias, and no deprecation shim in the run step (action.yml:169-245). I grepped the whole PR HEAD tree for the removed names: the only hits are CHANGELOG.md history, two synthetic test fixtures, and .claude/harness-candidates.md — no migration table, no upgrade section, nothing in README.md or docs/CI_GATE.md tells an existing user what to do. Most legacy workflows do fail loudly (with tasks: ignored the action runs coder-eval run with no paths, which hits discover_default_tasks and exits 1), which is why this is High rather than Critical. The sharp residual is minimum-task-score: a user who follows the new docs and converts tasks:/model: into args: but leaves minimum-task-score: "0.8" in place gets a GREEN gate with the score floor silently gone — the deleted docs/CI_GATE.md section called it "An additional gate on top of coder-eval's own exit code", and src/coder_eval/cli/run_command.py has no CLI equivalent (no --min-score-style option among its 21 flags), so this is a removed capability with no replacement. Add (a) a ## Migrating from the pre-1.0 inputs table to README.md and docs/CI_GATE.md mapping each removed input to its args line or to "removed, gate on run.json in a following step" with a worked snippet, and (b) an explicit BREAKING CHANGE: footer naming all seven so the generated CHANGELOG carries them.
3. [Axis 7] verify-published-action.yml passes HEAD's new input contract (args:, dropped junit-path:) to the published @v0, which still declares the old schema — the nightly/e2e gate goes red with a misleading PyPI-pin diagnosis until a release moves the major tag (.github/workflows/verify-published-action.yml:376) — The e2e smoke step is uses: UiPath/coder_eval@v0 # major asserted by the preflight job above (line 370) — the PUBLISHED major tag, deliberately, since line 372-373 says "version: intentionally omitted -- the whole point is to exercise the default pin baked into action.yml at the v0 tag." The PR rewrites its with: block to args: | / tasks/published_smoke.yaml (lines 376-377), but v0 still resolves to the OLD action.yml, which declares tasks/model/junit-path and not args. Releases are manual (.github/workflows/release.yml: on: workflow_dispatch only — "Merges to main do NOT auto-release"), so the v0 tag does not move at merge. Until someone dispatches a release, every 06:17 nightly (schedule: cron: "17 6 * * *") runs the old action with CE_TASKS="" and CE_JUNIT=coder-eval-junit.xml, i.e. coder-eval run --run-dir runs/verify-published --junit-xml coder-eval-junit.xml with no task paths → exit 1, no report at runs/verify-published/junit.xml → the gate at line 407 fires ::error::no JUnit report at $JUNIT. If the action failed during install, the pinned version is probably not installable — a misleading diagnosis for an input-schema skew. The preflight job reads action.yml at the major tag (lines 89-98) but only for the # <-- kept in sync version pin, never for input parity, and tests/test_verify_published_workflow.py asserts nothing about with: keys. Extend preflight to parse git show "$MAJOR:action.yml" for inputs: keys and assert every with: key of the smoke step is present, failing with an explicit "the v0 tag predates this workflow's inputs — cut a release" message; and sequence the merge so a release is dispatched immediately, or gate the smoke step on that parity check.

Non-blocking, but please consider before merge

1. [Axis 1] Caller-supplied --run-dir/--junit-xml in args last-win over the action's injected copies while run-dir/junit-path/run-md-path outputs still advertise the input-derived paths (action.yml:225) — action.yml:225 injects the flag unconditionally, then appends the caller's args after it:

args=(run --run-dir "$CE_RUN_DIR" --junit-xml "$junit")
while IFS= read -r arg; do
  args+=("$arg")
done < <(clean_lines <<< "$CE_ARGS")

But the args input is documented at action.yml:57 as "The only channel for the CLI's flags" — and --run-dir is one of the CLI's flags, so the contract invites the collision. Reproduced by executing the shipped run script with CE_ARGS="--run-dir\nruns/other\ntasks/a.yaml":

argv    = ['run','--run-dir','runs/ci','--junit-xml','runs/ci/junit.xml','--run-dir','runs/other','tasks/a.yaml']
outputs = {'run-dir':'runs/ci','junit-path':'runs/ci/junit.xml','run-md-path':'runs/ci/run.md'}

Click keeps the last value for a non-multiple option, so the run and run.md land in runs/other while all three outputs advertise runs/ci — and the recipe the docs hand every consumer (cat "${{ steps.eval.outputs.run-md-path }}" >> "$GITHUB_STEP_SUMMARY", README.md:163) then fails on a missing file with no hint why. Nothing warns. Either reject a --run-dir/--junit-xml entry in args with an explicit ::error:: naming the run-dir input, or drop the run-dir input and derive the report paths by reading the run directory back out of coder-eval's output — one source, not two.
2. [Axis 1] The copied job-summary recipe (if: always() + bare cat run.md) drops both guards the removed action code carried (file-existence check and 1 MiB cap) across three doc/skill surfaces (README.md:163) — The deleted action code guarded the append twice:

if [ "$CE_SUMMARY" = "true" ] && [ -f "$CE_RUN_DIR/run.md" ]; then
  head -c 1000000 "$CE_RUN_DIR/run.md" >> "$GITHUB_STEP_SUMMARY"
fi

The replacement recipe keeps neither, and is copied verbatim to four places: README.md:163, docs/CI_GATE.md:158, plugins/coder-eval/skills/ci/SKILL.md:86 (all as - if: always() steps), and .github/workflows/pr-checks.yml:1054. Two consequences. (a) head -c 1000000 was there because GitHub aborts a step-summary upload above 1 MiB; a large suite's run.md now reddens the job on a size limit the action used to absorb. (b) The -f check is gone while if: always() was added, so the step fires on the failure path — and action.yml:236-238 states in the same PR that "Output propagation from a failed composite step is not a documented guarantee, so a consumer chasing a failed run should key off the paths it passed in rather than these". steps.eval.outputs.run-md-path can therefore be empty exactly when if: always() runs it, giving cat: '': No such file or directory — or, at pr-checks.yml:1053, the hard-coded test -f "$RUN_MD" || { echo "run-md-path output does not exist: $RUN_MD"; exit 1; }, a second red step whose message reads like a broken action contract and masks the real failure. Note verify-published-action.yml:386-397 gets this right for the same reason ("Asserted against the LITERAL paths passed in the with: block above, not against steps.run.outputs.*"), so the two in-repo consumers now disagree. Make the documented recipe run-md="<run-dir>/run.md"; [ -f "$run_md" ] && head -c 1000000 "$run_md" >> "$GITHUB_STEP_SUMMARY" keyed off the literal run-dir the caller passed, and update all four sites together.
3. [Axis 3] BASH = shutil.which("bash") in the new test module is resolved naively — str | None flows unnarrowed into subprocess.run, with no None skip and no System32/WSL guard the existing bash-using modules have (tests/test_action_inputs.py:41) — Line 41 is BASH = shutil.which("bash"), and _run() (line 125) passes it straight into subprocess.run([BASH, "-c", script], ...) with no guard. The repo already solved this in the parallel module tests/test_verify_published_workflow.py, which lifts workflow bash the same way: its _resolve_bash() rejects a hit containing system32 ("C:\\Windows\\System32\\bash.exe is the WSL launcher stub … prints … in UTF-16 … and exits 1, which makes every lifted snippet fail for a reason that has nothing to do with the snippet") and falls back to Git Bash, and _bash_result skips when it is None (pytest.skip("no POSIX bash on this host; the lifted workflow snippets need one")). windows-smoke runs pytest tests/ -v -m "not live and not lint", so this module executes on the Windows pool where the PATH order decides which of the two shells is found; on a host with no POSIX bash the module raises TypeError from subprocess.run([None, ...]) on all 21 tests instead of skipping. Import _resolve_bash (or lift it into a shared helper) and add the BASH is None skip inside _run.
4. [Axis 3] New action surface is unproven by tests: the ** glob/exit-1 contract, working-directory wiring, and extras/install-flags all lack any real assertion (only a stubbed uv or the paid dogfood job) (tests/test_action_inputs.py:300) — test_task_globs_are_ordinary_args stops exactly where the risk starts: it stubs coder-eval, so it proves only that skills/**/*.yaml reaches argv unmangled. Its own comment makes the claim it does not test — "The CLI expands globs itself (expand_task_files), so passing them unexpanded is not a loss — and it exits 1 when nothing matches" — and this PR deletes the guard that previously made the defect class unreachable (test_ci_skill_does_not_recommend_a_recursive_task_glob, which banned ** in every documented tasks: value) while adding the opposite promise to the docs (docs/CI_GATE.md: "** works. tests/tasks/**/*.yaml is recursive, no globstar needed", plus args: tests/tasks/**/*.yaml snippets in README.md, docs/tutorials/02-ci-pipeline.md and the ci skill). src/coder_eval/cli/run_helpers.py::expand_task_files has zero direct tests — the only three references (test_run_command_junit.py, test_cli_telemetry.py) patch(...) it out — and no CI invocation uses ** (all use shell-expanded tasks/*.yaml). I verified the behavior is correct today (Path().glob('tasks/**/*.yaml') → ['tasks/sub/deep.yaml', 'tasks/top.yaml']), so this is a coverage gap, not a live bug: a switch to glob.glob(pattern) (non-recursive by default) would silently drop every top-level task in every consumer's copied snippet with nothing failing. Add a unit test over expand_task_files asserting a ** pattern returns both depths and that an unmatched pattern exits 1.
5. [Axis 4] env passthrough exports into the step's own shell before argv is built, so an entry named PATH / CE_ARGS / CE_RUN_DIR / GITHUB_OUTPUT hijacks the step (action.yml:216) (action.yml:216) — action.yml:216 is export "$name=${line#*=}", inside the loop at lines 200-217 that runs BEFORE run_dir="${CE_RUN_DIR%/}" (221), clean_lines <<< "$CE_ARGS" (228) and coder-eval "${args[@]}" (231). export mutates the CURRENT shell, not just the child, and the name filter ^[A-Za-z_][A-Za-z0-9_]*$ (213) admits PATH, CE_ARGS, CE_RUN_DIR, LD_PRELOAD and GITHUB_OUTPUT. I reproduced both halves against the extracted step script: with CE_ENV="CE_ARGS=--evil-flag\nCE_RUN_DIR=/tmp/hijacked" the argv became run --run-dir /tmp/hijacked --junit-xml /tmp/hijacked/junit.xml --evil-flag and the args: input tasks/real.yaml vanished; with an env line PATH=<attacker-dir>:... a planted coder-eval shim in that dir executed instead of the installed one (PWNED-shim-executed). The reachable path is not a hostile workflow author but a VALUE that carries a newline: the loop is line-based, so one entry such as API_BASE=${{ github.event.inputs.base }} (or a multi-line secret) whose value contains \nPATH=... becomes a second, honoured entry. The comment at 196-199 ("Only NAME is validated; VALUE is opaque data and is never eval'd") is true about eval but overstates the guarantee. Fix: (a) snapshot the control variables before the loop (args_raw=$CE_ARGS; run_dir_in=$CE_RUN_DIR; out=$GITHUB_OUTPUT) and read only the snapshots afterwards, or build the argv before parsing env; (b) reject reserved names (PATH, LD_*, BASH_ENV, IFS, GITHUB_*, CE_*) with a named error, mirroring the existing ::error::env entry #$n has an invalid name branch; (c) invoke the CLI by an absolute path resolved before the loop rather than via PATH. CVSS:3.1/AV:N/AC:H/PR:H/UI:N/S:U/C:H/I:H/A:H
6. [Axis 5] The deleted minimum-task-score capability is still advertised in six surfaces, including the published plugin skill's activation description and a tutorial cross-reference to a now-deleted section (plugins/coder-eval/skills/ci/SKILL.md:2) — The PR removes the input and the skill's whole "Step 6 — Choose the floor", but line 2 is unchanged: description: Generate a GitHub Actions workflow ... with the agent runtime, credentials, JUnit output and a score floor wired correctly. The frontmatter description is the skill's activation text (and is budget-capped per CLAUDE.md's SKILL_LISTING_BUDGET_CHARS), so /coder-eval:ci still advertises a capability its own body now denies at line 210: There is no score floor input; a suite that needs one gates on run.json in a following step. The same stale phrase survives in the second in-scope surface, README.md:224 — | [CI Gate & GitHub Action](docs/CI_GATE.md) | Run Coder Eval as a CI gate — the Marketplace Action, JUnit output, score floor | — which is GENERATED from mkdocs.yml:92 (CI_GATE.md: "Run Coder Eval as a CI gate — the Marketplace Action, JUnit output, score floor"), so the same string is also live at docs/index.md:90 and docs/llms.txt:38. CE028 only checks generated-vs-SSOT parity, and the SSOT itself is stale, so it stays green. Fix: drop "and a score floor wired correctly" from SKILL.md:2, edit the blurb in mkdocs.yml:92 and re-run make docs-indexes (regenerating README.md:224, docs/index.md:90, docs/llms.txt:38). While there, give the skill's prose escape hatch a concrete snippet — the deleted embedded gate handled bool/None/NaN rows fail-closed, and every consumer told to "gate on run.json in a following step" will now re-derive that by hand until the deferred --min-task-score CLI flag exists.
7. [Axis 6] An empty run-dir is not validated and now derives filesystem-root report paths (/junit.xml) — a regression from the removed junit-path input — failing (or silently writing to /) only after the paid run is spent (action.yml:221) — action.yml:221-225 derives the report paths by string concatenation with no emptiness check:

        run_dir="${CE_RUN_DIR%/}"
        junit="$run_dir/junit.xml"
        run_md="$run_dir/run.md"

        args=(run --run-dir "$CE_RUN_DIR" --junit-xml "$junit")

GitHub applies an input's default: only when the key is absent, not when it expands to the empty string, so a consumer wiring run-dir: ${{ inputs.run_dir }} from an unset caller input gets CE_RUN_DIR="" → junit=/junit.xml, run_md=/run.md, and --run-dir "" (which Path("") resolves to the cwd). The agent run then executes in full — real API spend — and only dies at the very end when the JUnit writer hits /junit.xml (src/coder_eval/cli/run_command.py:531-535, whose comment notes "A write error propagates"), while the step still advertises junit-path=/junit.xml / run-md-path=/run.md as outputs. The action already fails fast on a malformed extras (line 146-148) and a malformed env name (line 213-215); give run-dir the same treatment: [ -n "$CE_RUN_DIR" ] || { echo "::error::run-dir must not be empty"; exit 1; } before line 221, and add a CE_RUN_DIR="" case to tests/test_action_inputs.py::TestOutputs (which currently covers only the trailing-slash case).
8. [Axis 7] docs/CI_GATE.md:92-93 wrongly states zero-argument task discovery resolves against tasks/ relative to the working directory (it resolves against the installed package) (docs/CI_GATE.md:92) — The new bullet reads: "- Omitting args entirely does not run your suite. Zero-argument discovery / resolves against tasks/ relative to the working directory. Pass your paths." The second sentence is false and contradicts the first — a reader whose suite IS at tasks/ relative to working-directory will conclude that omitting args runs it. Actual behavior: src/coder_eval/cli/run_command.py:391 calls discover_default_tasks(), and src/coder_eval/cli/run_helpers.py:16-17 resolves _PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent.parent / DEFAULT_TASKS_DIR = _PROJECT_ROOT / "tasks" — the installed package's location, which under the action's uv tool install is <tool-venv>/lib/python3.13/tasks, never the working directory; it then exits 1 with "Default tasks directory not found". The version on origin/main said this correctly ("zero-argument discovery resolves against the installed package's location, not your checkout. It finds nothing and exits 1."), so this is a factual regression in the CE026-gated reference page integrators copy from. Restore the accurate wording, and while there, state in action.yml's args description (line 54-65) that args is effectively required despite required: false / default: "".

Nits

1. [Axis 1] Dead _stub() call, unused fixture parameter, and a third hand-rolled copy of _run() in the new test module (tests/test_action_inputs.py:373) — In test_outputs_are_written_before_a_failing_exit (lines 370-392): line 373 _stub(bindir, "coder-eval") writes an executable that lines 374-375 immediately overwrite and re-chmod, and its returned record path is discarded — it is pure dead code. The test also declares run_script as a parameter (line 370) but never uses it, re-reading the same script at line 378 with _step_script("Run coder-eval"). And lines 377-390 hand-roll the subprocess.run(...) block that _run already implements at lines 125-137, duplicating the PATH/HOME/GITHUB_OUTPUT/GITHUB_STEP_SUMMARY env assembly a third time. Give _stub an exit_code: int = 0 parameter and route this case through _coder_eval(run_script, tmp_path, ...) like every other test in the module; the whole body collapses to two asserts. (ruff does not catch any of these — ARG/unused-argument rules are not in the repo's select list.)
2. [Axis 1] The client-safety regression-guard comment is now orphaned from the describe block it explains (evalboard/lib/__tests__/sources.test.ts:117) — Lines 117-121 are the explanatory comment for describe("client-safety") ("lib/sources.ts is imported by CLIENT components ... so a Node builtin import here fails next build with UnhandledSchemeError ... Catch it here so the failure surfaces in a fast test"). The PR inserted a second, unrelated comment (122-128) and the whole describe("unlisted sources", ...) block (129-144) between it and describe("client-safety") at line 146. A reader now sees the webpack/UnhandledSchemeError rationale sitting directly above a test about the gha nav tab, which it does not explain. Move the new unlisted sources describe and its comment above line 117, or below line 156, so each comment stays adjacent to the block it documents.
3. [Axis 2] _step_script declares -> str but returns an unvalidated Any out of yaml.safe_load, and indexes a shape it never checks (tests/test_action_inputs.py:66) — Line 66 is data = yaml.safe_load(ACTION_YML.read_text(encoding="utf-8")) (typed Any), line 67 for step in data["runs"]["steps"]: and line 69 return step["run"]. Neither the mapping shape nor the returned value is checked, so the -> str annotation is asserted rather than proven: a restructure of action.yml (or a uses:-only step matching the looked-up name) surfaces as a bare KeyError: 'runs' / KeyError: 'run' at import-adjacent fixture time instead of the clear failure the function already knows how to produce — it raises AssertionError(f"action.yml has no step named {step_name!r}") for the one miss it does handle. The sibling module covering the same YAML does verify: tests/test_verify_published_workflow.py:78-80 is def _load(path: Path) -> dict[str, Any]: / data = yaml.safe_load(...) / assert isinstance(data, dict), f"{path} did not parse as a mapping". Mirror that: assert isinstance(data, dict), assert steps is a list, and assert isinstance(run, str) before returning it.
4. [Axis 3] Retargeted ci skill assertion keys on a bare -e token — passes on any unrelated occurrence and fails on the equivalent --experiment (tests/test_custom_lint.py:1760) — Line 1760 is assert "-e" in raw.split() and "experiment" in text, (. raw.split() is a whitespace split of the whole SKILL.md, so any standalone -e anywhere in the file (a test -e in an unrelated shell snippet, a bullet dash) satisfies the first conjunct even if the experiment guidance were deleted, while the equally correct long form --experiment fails it despite the skill being right — the assertion pins the flag spelling rather than the behavior it claims to guard ("the ci skill does not say how to pass an experiment through to the run"). Scope it to the emitted snippet instead: parse the skill's args: block and assert one of its lines is -e or --experiment followed by an experiment path — same as the CE026 clauses already parse snippet YAML rather than grepping prose.
5. [Axis 4] Copy-verbatim doc snippets interpolate ${{ }} directly into a run: body, the pattern the repo's own dogfood step deliberately avoids (README.md:163) — README.md:163 is run: cat "${{ steps.eval.outputs.run-md-path }}" >> "$GITHUB_STEP_SUMMARY", repeated verbatim at docs/CI_GATE.md:158 and — worse, because the ci skill WRITES this workflow into a user's repository — at plugins/coder-eval/skills/ci/SKILL.md:86. GitHub's hardening guidance is to never expand an expression inside a run: body; the runner textually substitutes the value before bash parses it, so the double quotes do not contain $(...) or a backtick. The value is not a fixed literal: run-md-path is "$run_dir/run.md" built from CE_RUN_DIR (action.yml:221,223,242), which is the caller's run-dir input, and finding #3 shows that output can be rewritten outright. The same repo already does it correctly for the same value — .github/workflows/pr-checks.yml:1050-1054 passes RUN_MD: ${{ steps.dogfood.outputs.run-md-path }} under env: and then runs cat "$RUN_MD" >> "$GITHUB_STEP_SUMMARY". Make all three doc snippets match that form. While editing, note the base action capped this write (head -c 1000000, origin/main:action.yml:160) and the replacement snippet does not, so an oversized run.md now blows GitHub's 1 MiB job-summary limit. CVSS:3.1/AV:N/AC:H/PR:H/UI:R/S:U/C:L/I:L/A:L
6. [Axis 4] $GITHUB_OUTPUT is written with the plain key=value form, so a newline in run-dir injects arbitrary output keys (action.yml:240) — action.yml:239-243 is { echo "run-dir=$CE_RUN_DIR"; echo "junit-path=$junit"; echo "run-md-path=$run_md"; } >> "$GITHUB_OUTPUT". $CE_RUN_DIR is un-normalised caller input and is never checked for newlines, which is exactly the case the heredoc delimiter form of $GITHUB_OUTPUT exists to prevent. Reproduced against the extracted step script with CE_RUN_DIR=$'runs/ci\nrun-md-path=/etc/passwd': the file received run-dir=runs/ci, run-md-path=/etc/passwd, junit-path=runs/ci, run-md-path=/etc/passwd/junit.xml, … — i.e. attacker-chosen values for outputs the action declares, which then feed the documented cat "${{ steps.eval.outputs.run-md-path }}" snippet (finding #2) and any report_paths: consumer. Reject a run-dir containing a newline with a ::error:: before the run starts (alongside the extras validation at 146-148), and/or emit the outputs with the delimiter form: delim="ghadelim_$RANDOM"; printf '%s<<%s\n%s\n%s\n' run-dir "$delim" "$CE_RUN_DIR" "$delim" >> "$GITHUB_OUTPUT". CVSS:3.1/AV:N/AC:H/PR:H/UI:R/S:U/C:L/I:L/A:N
7. [Axis 4] The gha source's stated containment ("nothing enumerates it") is not enforced — /runs/latest?src=gha lists the container (evalboard/lib/sources.ts:50) — evalboard/lib/sources.ts:49-51 claims "DELIBERATELY UNLISTED: registered here but absent from NAV in app/layout.tsx, so there is no tab and nothing enumerates it — a run is reachable only by the direct link from the GitHub run that produced it", and evalboard/README.md repeats it ("reachable only by the direct link in the GitHub run summary"). Absence from NAV is not the only enumeration path: app/runs/latest/page.tsx:16-18 does const source = sourceById(scalarParam((await searchParams).src)); const id = await latestRunId(source); and lib/runs.ts:668-685 routes that to listRunIdsRemote(source.container), which at lib/blob.ts:114-134 returns every top-level prefix in the container with no date-shape filter and no run.json check (unlike the local branch at blob.ts:140-156). So /runs/latest?src=gha discovers and redirects to the newest ad-hoc run in runs-gha, and there is no auth layer in evalboard/app or evalboard/lib to fall back on. Either drop the containment sentence from the comment and the README (keeping only the real rationale — separate container for the 14-day expiry rule), or make it true by having /runs/latest refuse a source that is not in NAV. The new sources.test.ts "gha is registered but has no nav tab" test pins the NAV half only and would not catch this. CVSS:3.1/AV:N/AC:H/PR:N/UI:N/S:U/C:L/I:N/A:N
8. [Axis 5] Three near-identical hand-rolled line parsers (clean_lines() and friends) duplicated across action.yml steps; the byte-identity test guards only two (action.yml:200) — clean_lines() is defined twice (line 129 in Install coder-eval, line 184 in Run coder-eval) — unavoidable, since the steps are separate bash processes, and tests/test_action_inputs.py::TestSharedParser::test_clean_lines_is_byte_identical_in_both_steps pins the copies. But the env loop at lines 200-217 is a THIRD copy of the same preamble, inside the SAME process as the copy at line 184: line="${line%$'\r'}" (203), line="${line#\"${line%%[![:space:]]*}\"}" (204), [ -z "$line" ] && continue (205), case "$line" in '#'*) continue ;; esac (206) — identical to lines 187, 188, 190, 191, differing only by the deliberate omission of right-trim (documented at 197-199). Nothing guards that third copy, so a future edit to clean_lines (a new comment marker, a BOM strip) silently diverges the env path while the byte-identity test still passes. Fix within the run step: factor the shared preamble into one helper the env loop reuses (e.g. clean_lines taking a keep_trailing flag, or a _normalize_line primitive both call), so the divergence is one explicit argument rather than a copy.
9. [Axis 6] Dogfood probe writes a scratch task YAML into the repo's tasks/ tree and only removes it on the success path (.github/workflows/pr-checks.yml:1079) — .github/workflows/pr-checks.yml:1067-1079 runs under set -euo pipefail with working-directory: tasks:

          cat > byoa-probe.yaml <<'YAML'
          …
          coder-eval plan byoa-probe.yaml
          rm -f byoa-probe.yaml

If coder-eval plan exits non-zero — the exact case this probe exists to detect — set -e aborts before the rm, leaving byoa-probe.yaml inside the repository's canonical task tree for the rest of the job. Harmless on an ephemeral runner today, but it is a cleanup that lives only on the happy path and it plants a synthetic task where a later tasks/** glob would pick it up. Write the probe to "$RUNNER_TEMP" (or a mktemp -d) and pass the absolute path, or move the rm -f into a trap 'rm -f byoa-probe.yaml' EXIT.
10. [Axis 7] working-directory is documented as applying to "every step" but the first step cannot take it (action.yml:78) — The description reads "Directory every step of this action runs in, and what run-dir, the task / paths in args and relative extra-packages entries resolve against" (lines 78-79), and docs/CI_GATE.md:134 repeats "It applies to every step / the action runs". Only two of the three steps carry it: - name: Install coder-eval (line 115) and - name: Run coder-eval (line 172). The first, - name: Install uv / uses: astral-sh/setup-uv@38f3f104... (lines 111-112), is a uses: step and cannot take working-directory: — the very limitation the input's own description cites two lines later. Harmless today (setup-uv's cwd-sensitive cache lookup is unused here), but the wording will mislead the next person who adds a cwd-sensitive uses: step. Say "every run: step of this action" in both places.

What's Missing

Parallel paths:

• 🟡 The ci skill — the one surface that WRITES a workflow into a consumer's repo — was updated for args/reports only and never mentions three of the five new inputs: extras, extra-packages, install-flags, working-directory (grep of plugins/coder-eval/skills/ci/SKILL.md returns zero hits for all four). Its Step 1 discovery already locates the task tree and reads the repo's agent: config, which is exactly where those decisions belong: a suite under tests/ needs working-directory, an out-of-tree plugin needs extra-packages, and a repo whose tasks use agent.type: codex now gets an emitted workflow that installs coder-eval without extras: codex and dies at agent dispatch. The skill's Step 5/6 renumber shows it was edited in this PR, so the omission is a miss, not deferral. (trigger: plugins/coder-eval/skills/ci/SKILL.md)
• 🟠 Two in-repo consumers of the action were updated inconsistently: .github/workflows/pr-checks.yml (uses: ./, HEAD action) got the new args:/working-directory: schema and works, while .github/workflows/verify-published-action.yml got the same new schema but still resolves uses: UiPath/coder_eval@v0, which declares no args input. Only one of the two paths can be right until the major tag moves. (trigger: .github/workflows/verify-published-action.yml) (restates: Axis 7: verify-published-action.yml passes HEAD's new input contract to the published @v0)
• 🟡 The job-summary recipe forked: pr-checks.yml:1049-1054 keeps set -euo pipefail plus a test -f "$RUN_MD" guard, while the three copy-verbatim surfaces (README.md:163, docs/CI_GATE.md:158, plugins/coder-eval/skills/ci/SKILL.md:86) ship a bare cat under if: always(). The repo's own consumer and the recipe it hands integrators no longer match, and neither retains the deleted head -c 1000000 cap. (trigger: README.md) (restates: Axis 1: The copied job-summary recipe drops both guards the removed action code carried)
• 🔵 .claude/harness-candidates.md (lines 301 and 436) still documents minimum-task-score semantics and "consumer-simulating jobs pass minimum-task-score: \"0.0\"" — the last live description of the deleted input outside CHANGELOG history, and a backlog surface a future contributor will implement against. (trigger: action.yml) (restates: Axis 5: The deleted minimum-task-score capability is still advertised in six surfaces)

Tests:

• 🟡 The new 446-line test module asserts only what is INSIDE the two run: bodies — _step_script (tests/test_action_inputs.py:66-70) returns step["run"] and discards the rest of the step mapping. Nothing asserts the inputs.* -> CE_* wiring (CE_ARGS: ${{ inputs.args }}, CE_RUN_DIR, CE_ENV, CE_EXTRAS, …) or that each run: step carries working-directory: ${{ inputs.working-directory }}: rename an input, or typo ${{ inputs.arg }}, and GitHub expands it to the empty string while all 37 tests still pass — only the paid action-dogfood job can catch it. Add a structural check over action.yml (every CE_* env value references a declared input; every run: step carries the working-directory expression), which also mechanically settles the "every step" wording gap on the working-directory description. (trigger: tests/test_action_inputs.py)
• 🟡 CE026's find_unknown_action_inputs scans only default_doc_paths (README.md + docs/**/*.md + plugins/**/*.md, tests/lint/action_docs.py:252) — .github/workflows/** is excluded, so neither in-repo with: block is checked against action.yml's inputs. That is precisely the defect class this PR shipped: an unknown key in the dogfood job would be silently ignored and the gate would stay green. Extending the existing helper to the workflow files (resolving the tag for uses: UiPath/coder_eval@vN via git show <tag>:action.yml) is a small delta over code the PR already touches. (trigger: tests/test_custom_lint.py) (restates: Axis 7: verify-published-action.yml passes HEAD's new input contract to the published @v0)
• 🟡 TestOutputs exercises only run-dir: /tmp/runs and the trailing-slash case; the two run-dir values that actually misbehave have no case at all — the empty string (yields --run-dir "" plus /junit.xml and /run.md) and a value containing a newline (injects attacker-chosen $GITHUB_OUTPUT keys). Both were reproduced against the shipped step script during review, so the tests are cheap to add alongside the guards. (trigger: tests/test_action_inputs.py) (restates: Axis 6: An empty run-dir is not validated and now derives filesystem-root report paths)
• 🟡 This PR deletes test_ci_skill_does_not_recommend_a_recursive_task_glob (the rule that banned ** on every CE026 doc surface) and replaces it with the opposite promise on four surfaces, but adds no test for the behavior it now relies on: src/coder_eval/cli/run_helpers.py::expand_task_files still has zero direct tests (all four references patch it out), and no CI invocation passes a ** pattern. A unit test asserting a ** pattern returns both a top-level and a nested task, and that an unmatched pattern raises typer.Exit(1), is the regression guard the deleted lint rule used to provide. (trigger: tests/test_custom_lint.py) (restates: Axis 3: New action surface is unproven by tests (the ** glob/exit-1 contract))
• 🔵 TestSharedParser pins the two clean_lines copies byte-for-byte, but nothing pins the THIRD copy of the same normalization preamble — the CRLF strip / left-trim / blank / # handling open-coded in the env loop (action.yml:200-206) inside the same step as the copy at line 184. A future edit to clean_lines diverges the env path with the byte-identity test still green. (trigger: action.yml) (restates: Axis 5: Three near-identical hand-rolled line parsers duplicated across action.yml steps)
• 🔵 TestEnvPassthrough covers only shape errors (NOEQUALS, leading digit, space, dash). It has no case for a reserved name the regex admits (PATH, CE_ARGS, CE_RUN_DIR, GITHUB_OUTPUT) and none for a value containing an embedded newline — the two inputs that let an env: entry rewrite the step's own argv or shadow the CLI on PATH. (trigger: tests/test_action_inputs.py) (restates: Axis 4: env passthrough exports into the step's own shell before argv is built)

Downstream consumers:

• 🟡 The new gha source is a cross-repo contract that this repo neither creates nor verifies: the runs-gha container, the storage RBAC the deployment's identity needs to read it, the expire-runs-gha-14d lifecycle rule, and the ?src=gha link emitted by UiPath/skills' run-coder-eval dispatch all live outside this tree (no infra file in the repo references any container — aria-runs has no definition here either). The tests pin only the in-repo half (id -> container), so a mismatch on the producer side, or a container that does not exist yet, surfaces as a broken link with no failing check; the PR does not state the ordering (create container + grant read + add lifecycle rule, then merge). (trigger: evalboard/lib/sources.ts)
• 🟡 Registering the source makes it live on every ?src=-aware consumer at once — /runs/latest?src=gha (which enumerates the container via listRunIdsRemote), /api/download?src=gha, /api/file?src=gha, /api/refresh?src=gha — while the PR's containment claim and its new tests cover only the NAV array. Either the routes get the same treatment (refuse a source absent from NAV) or the "nothing enumerates it" sentence comes out of sources.ts:50 and evalboard/README.md. (trigger: evalboard/lib/sources.ts) (restates: Axis 4: The gha source's stated containment ("nothing enumerates it") is not enforced)
• 🟠 Seven documented inputs are removed with no alias, no deprecation shim, no migration table in README.md / docs/CI_GATE.md, and no BREAKING CHANGE: footer — while pyproject.toml's major_on_zero = false means the change ships as a MINOR and release.yml's promote job moves the same v0 tag every existing consumer pins. The delivery path is therefore "the break arrives with no version signal", and the minimum-task-score residual is the sharp one: a converted workflow that leaves that key in place gets a green gate with the floor silently gone and no CLI replacement. (trigger: action.yml) (restates: Axis 7: Seven documented action inputs removed with no alias and no consumer-facing migration note)

Display & mapping dicts:

• 🔵 GHA_SOURCE.label ("Ad-hoc (GH)") is dead data: Source.label is documented in sources.ts as "used for the nav tab and page headings", but the only renderer is NAV in app/layout.tsx (which excludes gha) and no run page renders a source badge. A run opened from a GitHub link is therefore visually identical to a nightly run under the same id, and the header's bare href="/" plus every NAV tab drop ?src=gha on the first click. Either render the source label on run pages for any non-default source, or drop the field for unlisted sources so it is not a promise nothing keeps. (trigger: evalboard/lib/sources.ts)

Daily/nightly:

• 🟠 The PR does not state its effect on the 06:17 nightly verify-published-action run, which is the repo's only scheduled paid job in scope: from merge until someone dispatches a release, the smoke step hands args:/run-dir: to the OLD v0 action, --model claude-haiku-4-5-20251001 is dropped as an unknown input (so the unattended run bills the default model), and the JUnit gate fails with an install-oriented diagnostic that misdescribes an input-schema skew. Either sequence a release immediately after merge, gate the smoke step on a preflight input-parity check, or say explicitly that the nightly is expected red until the tag moves. (trigger: .github/workflows/verify-published-action.yml) (restates: Axis 7: verify-published-action.yml passes HEAD's new input contract to the published @v0)
• 🔵 The per-PR action-dogfood job's blast radius is unstated: it now runs from working-directory: tasks, installs a local plugin through --with, and writes a synthetic byoa-probe.yaml into the repo's canonical task tree with cleanup only on the happy path — so a failing coder-eval plan leaves a task file inside tasks/ that any later tasks/** glob in the same job would pick up. (trigger: .github/workflows/pr-checks.yml) (restates: Axis 6: Dogfood probe writes a scratch task YAML into the repo's tasks/ tree and only removes it on the success path)

Harness & Lint Improvements

Condensed to fit GitHub's 65 536-character comment limit — full rationale for each item is in 00-summary.md.

Static checks (proposed lint rules):

• CE047 — composite-action shell hygiene
• CE048 — with: key parity resolved at the ref the step actually uses.
• CE049 — removed-input vocabulary must not survive in prose or in the docs-index SSOT.
• CE050 — one bash resolver in the test suite.
• CE051 — narrow yaml.safe_load before subscripting or returning it.
• Put tests/ under pyright.
• Enable ARG (flake8-unused-arguments)
• CE052 — copy-verbatim workflow snippets must be hardened and guarded.
• Wire GitHub-Actions static analysis into make verify / CI: actionlint + shellcheck + zizmor.

Harness improvements:

• Make the task-path expansion contract executable, and decide it.
• Add a removed-input tripwire inside the action, proven by the dogfood job.
• Turn action-dogfood into a real output-contract e2e, plus a bash matrix over the lifted step script.
• Ship the job-summary append as a tested artifact instead of an untested inline recipe.
• Assert a clean working tree at the end of jobs that write scratch files.
• Make the evalboard gha containment claim executable, or delete it.
• Sequence the published-action gate against the release, and make its diagnostic honest.

Top 5 Priority Actions

1. Make task-path resolution fail closed: expand_task_files (src/coder_eval/cli/run_helpers.py:81-94) raises only when the union of all patterns is empty, so one stale glob among several silently drops a suite and the gate reports green over unmeasured tasks — fail on any pattern that matched nothing, or correct the contrary promise now published at docs/CI_GATE.md:90 and plugins/coder-eval/skills/ci/SKILL.md:104.
2. Stop the env passthrough from mutating the action's own shell (action.yml:216): export runs before argv is built, so an entry named CE_ARGS, CE_RUN_DIR or PATH — reachable from any value containing a newline — rewrites which tasks run, where results land, or which coder-eval binary executes, contradicting the documented "for the coder-eval process only" contract at action.yml:68-70; snapshot the control variables before the loop and reject reserved names.
3. Ship a migration path for the seven removed action inputs (action.yml:24): with major_on_zero = false and a moving v0 tag, every existing consumer receives the break unannounced, and a workflow that keeps minimum-task-score: "0.8" gets a green gate with the score floor silently gone — add a migration table to README.md and docs/CI_GATE.md, a real BREAKING CHANGE: footer, and a worked run.json floor snippet.
4. Fix the published-action gate at .github/workflows/verify-published-action.yml:376, which hands HEAD's new args:/dropped junit-path: schema to the still-old @v0 tag: the unattended nightly silently drops the haiku pin (paid run on the default model) and then reports a misleading "pinned version is probably not installable" error — add an input-parity check to the preflight job against git show "$MAJOR:action.yml", or sequence a release with the merge.
5. Harden and re-align the report-path surface in action.yml: validate a non-empty run-dir before line 221 (an empty input yields --run-dir "" plus /junit.xml, failing only after the spend, or writing to / in a root container), reject or honor a caller-supplied --run-dir/--junit-xml in args at line 225 instead of emitting outputs that no longer match argv, and restore the accurate zero-argument-discovery wording at docs/CI_GATE.md:92 with a guarded head -c 1000000 job-summary recipe across README.md:163, docs/CI_GATE.md:158 and plugins/coder-eval/skills/ci/SKILL.md:86.

---

Stats: 0 🔴 · 3 🟠 · 8 🟡 · 10 🔵 across 8 axes reviewed.

[uipreliga]

Fix what you agree with and 🚢