Skip to content

fix(ci): the icarus gate was blind to every test failure (missing pipefail) - #374

Merged
gHashTag merged 1 commit into
mainfrom
fix/ci-pipefail-blind-gates
Aug 9, 2026
Merged

fix(ci): the icarus gate was blind to every test failure (missing pipefail)#374
gHashTag merged 1 commit into
mainfrom
fix/ci-pipefail-blind-gates

Conversation

@gHashTag

@gHashTag gHashTag commented Aug 9, 2026

Copy link
Copy Markdown
Owner

The required icarus check could not fail on a failing spec

Its loop is:

if ! $T27C icarus-simulate specs/${spec}.t27 | tee /tmp/${spec}.simlog; then

A bash pipeline's exit status is the last command's — tee, which always succeeds. No shell: or defaults: is declared, so GitHub's default bash -e {0} applies without pipefail. t27c correctly exits 1 and prints Error: Icarus simulation reported test/bench failures, but the pipe swallowed it and the gate reported green.

Demonstrated: injecting a failing assertion into crc16 and running the exact CI construct yields STATUS=0 — a false green. With set -o pipefail the same input yields STATUS=1.

So the gate effectively only ran the bench budgets and the wave-55 coverage guard. Every "N specs pass under icarus" claim rested on local verification, not on this check. This is the broken-ruler failure mode: the instrument meant to measure correctness could not report incorrectness.

Fixes both pipe-swallowing sites

  • gft-verilog-kat.yml — the icarus simulate loop (the critical one)
  • spec-drift-guard.yml — the typecheck | grep -q loop, same class

The typecheck change was verified safe first: across all 107 specs the t27c exit code and the grep verdict never diverge, so pipefail cannot introduce a false positive there.

Verified both directions

  • Negative: an injected failing assertion is now detected (STATUS=1, previously 0)
  • Positive: the healthy 107-spec corpus still simulates green under pipefail (STATUS=0) — no false positives

🤖 Generated with Claude Code

…efail)

The required 'GF-T spec KATs under Icarus Verilog' check could not fail
on a failing spec. Its loop is

  if ! $T27C icarus-simulate specs/${spec}.t27 | tee /tmp/${spec}.simlog; then

and a bash pipeline's exit status is the LAST command's -- tee, which
always succeeds. No 'shell:' or 'defaults:' is declared, so GitHub's
default 'bash -e {0}' applies WITHOUT pipefail. t27c correctly exits 1
and prints 'Error: Icarus simulation reported test/bench failures', but
the pipe swallowed it and the gate reported green.

Demonstrated: injecting a failing assertion into crc16 and running the
exact CI construct yields STATUS=0 -- a false green. With
'set -o pipefail' the same input yields STATUS=1.

So the gate effectively only ran the bench budgets and the wave-55
coverage guard. Every 'N specs pass under icarus' claim rested on local
verification, not on this check. This is the broken-ruler failure mode:
the instrument that was supposed to measure correctness could not
report incorrectness.

Fixes both pipe-swallowing sites:
  - gft-verilog-kat.yml: the icarus simulate loop (the critical one)
  - spec-drift-guard.yml: the typecheck '| grep -q' loop, same class

The typecheck change was verified safe first: across all 107 specs the
t27c exit code and the grep verdict never diverge, so pipefail cannot
introduce a false positive there.

Verified both directions: an injected failing assertion is now DETECTED
(STATUS=1, previously 0), and the healthy 107-spec corpus still
simulates green under pipefail (STATUS=0) -- no false positives.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
@gHashTag
gHashTag merged commit f60ca06 into main Aug 9, 2026
4 checks passed
@gHashTag
gHashTag deleted the fix/ci-pipefail-blind-gates branch August 9, 2026 04:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant