ci: gate fixture runner on analyzer binary presence#2
Conversation
Two fixtures were failing CI on master: - rust_analyzer/ — rust-analyzer is not installed in CI (the workflow only installs pyright + semgrep). The runner had no notion of optional analyzer binaries, so the fixture failed with 'expected hallucinated-import diagnostic'. - semgrep/ — the equivalent cargo integration test (semgrep_fixture_flags_multiple_rules_when_installed) passes, so semgrep itself works. The shell runner was redirecting all daemon stderr to /dev/null, which made the failure undebuggable. Fix: - Map each fixture name to its required external binary; skip when missing. rust-analyzer / semgrep / pyright / tsc are now treated as optional capabilities the way daemon/tests/fixtures.rs already does. - Capture daemon stderr to a temp file and dump it on failure so the next regression doesn't require source-code spelunking. - Update the docstring + the misleading 'expected hallucinated-import' message — the script checks a generic non-green invariant; detailed per-analyzer expectations live in daemon/tests/fixtures.rs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CI on the previous commit revealed two refinements needed: 1. `command -v rust-analyzer` returns true on a stock rustup install because rustup ships a shim at ~/.cargo/bin/rust-analyzer that fails unless `rustup component add rust-analyzer` has been run. The daemon's own `binary_present()` check (`<bin> --version`) catches this; the shell runner now mirrors that. 2. The semgrep fixture is producing 0 findings via the daemon binary even though semgrep itself is installed and the equivalent cargo integration test passes. Captured stderr just shows `semgrep pass complete n=0` with no clue why. Add diagnostic logging in semgrep.rs that, on a 0-result run, surfaces the parsed `errors` field plus a tail of semgrep's stderr — that should pinpoint the misconfig in the next CI run. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Root cause for the semgrep fixture failing in CI but passing in cargo
tests:
When the target directory lives inside a git repo, semgrep auto-applies
a "tracked by git only" filter. From semgrep's own vantage point in CI
(target = /repo/test/fixtures/ai-slop/semgrep, daemon CWD = /repo) the
filter sees 0 files and the scan returns 0 findings, even though the
files are tracked at the repo root. Captured semgrep stderr from the
prior debugging commit:
Ran 15 rules on 0 files: 0 findings.
Scan was limited to files tracked by git
The cargo integration test passes because `isolate()` copies the
fixture into a tempdir outside any git repo, so the filter never
engages.
`--no-git-ignore` disables both the .gitignore respect AND the
git-tracked-only filter — IVE owns workspace traversal end-to-end, so
we never want semgrep second-guessing which files to scan.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Second piece of the semgrep CI fix. With --no-git-ignore alone, semgrep
still consulted its built-in default .semgrepignore, whose patterns
include test/ — and since semgrep walks up from the fixture target to
the IVE git root, both app.py and requirements.txt got filtered as
'test/fixtures/ai-slop/semgrep/...'. Captured stderr from the previous
CI run:
Ran 15 rules on 0 files: 0 findings.
Files matching .semgrepignore patterns: 2
The only override mechanism semgrep CLI accepts is a .semgrepignore
file at the project (git) root, which it then uses *instead of* the
defaults. Empty contents = scan everything semgrep would otherwise
ignore.
This is the right behaviour for IVE specifically: scanner::walk_workspace
already filters node_modules / target / .git / .ive, and our
test/fixtures/ai-slop/ tree exists precisely to be scanned by the
semgrep analyzer in fixture tests.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
StatusThe original CI failure on
Two follow-up items surfaced1.
|
1 participant
Summary
CI on
masteris red on thefixture integrationstep at commit 2a31911. Two fixtures fail with "expected hallucinated-import diagnostic":rust_analyzer/—rust-analyzerbinary is not installed in CI (the workflow only installspyrightandsemgrep). The runner had no notion of optional analyzer binaries, so the fixture failed unconditionally.semgrep/— the equivalent cargo test (semgrep_fixture_flags_multiple_rules_when_installed) passes in the same CI run, so semgrep itself works. The shell runner was suppressing daemon stderr (2>/dev/null), which made this undebuggable.The deeper issue:
test/run_fixtures.shchecked an overly broad "≥1 diagnostic per fixture" invariant. Detailed per-analyzer assertions already live indaemon/tests/fixtures.rs; this script is just the end-to-end smoke check.Changes
daemon/tests/fixtures.rsalready uses.Test plan