Skip to content

feat: --fail-on, --quiet, exit code 2, rules and explain (#25) - #76

Merged
hermanngeorge15 merged 1 commit into
mainfrom
feat/cli-surface-25
Aug 25, 2026
Merged

feat: --fail-on, --quiet, exit code 2, rules and explain (#25)#76
hermanngeorge15 merged 1 commit into
mainfrom
feat/cli-surface-25

Conversation

@hermanngeorge15

Copy link
Copy Markdown
Contributor

Covers CLI-06 and CLI-07 from #25. Depended on #21 — with everything graded CRITICAL or HIGH there was nothing for --fail-on to threshold on, which is why that issue lists this as blocked.

Exit codes are the real interface to CI

0  no findings
1  findings at or above --fail-on
2  findings, all below --fail-on

The third one is the point. Without it, --fail-on critical would exit 0 on a file full of HIGH findings, and any pipeline checking for zero would call it clean — the flag would quietly become a way to hide things rather than a way to prioritise them.

Verified as a matrix:

document --fail-on critical high medium low
CRITICAL payload 1 1 1 1
MEDIUM payload 2 2 1 1
clean 0 0 0 0

--fail-on raises the bar for failing, not for reporting. Findings below it are still printed, because a user who can't see them can't judge whether the bar is set right. Pinned by its own test.

rules and explain

The question a blocked commit creates:

$ injection-scanner rules
ID       SEVERITY  CATEGORY               NAME
PI001    CRITICAL  role_override          ignore-previous-instructions
PI003    MEDIUM    role_override          you-are-now
PI041    LOW       encoding               zero-width-chars

$ injection-scanner explain pi035
PI035  jailbreak-prompt  [LOW]
Category:    jailbreak
Detects:     Explicit jailbreak prompt reference
Remediation: Remove jailbreak prompt.
Pattern:     (?i)\bjailbreak\s+prompt\b

Suppress one occurrence with:
  <!-- injection-scanner:ignore PI035 -->

Both show the effective severity, resolved against the category default — a listing with a blank for every pattern that inherits would be worse than none. explain is case-insensitive, ends with the exact suppression directive, and on an unknown id suggests nearby ones instead of just failing. rules --format json is machine-readable.

Two bugs worth naming

{:<9} silently did nothing to the severity column. A width specifier only pads a custom Display impl if that impl routes through Formatter::pad — this one doesn't, so every column ran together. Rendered to a String first.

The new items landed between #[derive(Subcommand)] and enum Commands, quietly reattaching the derive to the wrong type. Every #[arg] in the file then failed to resolve, with an error pointing at lines I hadn't touched.

Tests

7 new, including the full exit-code matrix and the below-threshold-still-reported guarantee. Full suite 23 binaries green, clippy clean.

Not in this PR: --baseline (CLI-08), --no-color, --stdin-name. --baseline is the larger adoption feature and deserves its own change.

The CLI was one `check` subcommand with a binary exit code, so a team could not
adopt the scanner incrementally or tune what blocks their build.

Exit codes are this tool's real interface to CI, and there are now three:

  0  no findings
  1  findings at or above --fail-on
  2  findings, all below --fail-on

The third is the one that matters. Without it `--fail-on critical` would exit 0
on a file full of HIGH findings, and any pipeline checking for zero would call
it clean — the flag would quietly become a way to hide things rather than a way
to prioritise them. Matches the convention spec-linter already uses here.

`--fail-on` raises the bar for FAILING, not for REPORTING. Findings below it are
still printed, because a user who cannot see them cannot judge whether the bar
is set right. Pinned by a test.

`rules` and `explain <PI0XX>` answer the question a blocked commit creates.
Both report the EFFECTIVE severity — resolved against the category default —
since a listing showing a blank for every pattern that inherits would be worse
than no listing. `explain` is case-insensitive, ends with the exact suppression
directive for that pattern, and on an unknown id suggests nearby ones rather
than just failing.

Two bugs worth naming:

  - `{:<9}` silently did nothing to the severity column. A width specifier only
    pads a custom Display impl if that impl routes through `Formatter::pad`, and
    this one does not, so every column ran together. Rendered to a String first.

  - The new items landed between `#[derive(Subcommand)]` and `enum Commands`,
    which quietly reattached the derive to the wrong type. Every `#[arg]` in the
    file then failed to resolve.

Depended on #21: with everything graded CRITICAL or HIGH there was nothing for
`--fail-on` to threshold on, which is why that issue lists this as blocked.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@hermanngeorge15
hermanngeorge15 merged commit c9f1404 into main Aug 25, 2026
1 check passed
@hermanngeorge15
hermanngeorge15 deleted the feat/cli-surface-25 branch August 25, 2026 13:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant