rivet validate: actionable errors are drowned by high-volume low-signal warnings — add severity filtering / noise controls

[avrabe]

Context

Found while dogfooding (triaging agent reports + marking artifacts implemented on rivet's own repo, 0.14.0-track). Same friction class an agent independently reported in #350 (37+ INFO: field 'crate' is not defined drowning the actionable errors).

The friction

rivet validate on rivet's own repo currently prints ~158 warnings for 0 errors. The vast majority are low-signal:

• WARN: [X] Every requirement should be satisfied by at least one design decision or feature — lifecycle aspiration, fires on nearly every draft/roadmap REQ.
• WARN: [X] prose mentions 'REQ-124' but no typed link to it — fires whenever a REQ description references another REQ id narratively (e.g. "addressed by REQ-124/REQ-125"), which is legitimate prose, not a missing dependency.
• per-rule lifecycle-coverage gaps.

When there IS an actionable error (a broken link, a real cardinality violation), it scrolls past in the wall of warnings. There's no easy way to say "show me only what I must act on."

Why it matters

For an AI agent or a human doing mark-implemented / bring-to-green work, the cost is real: you can't tell at a glance whether validate is clean-modulo-aspirations or genuinely broken. It also trains people to ignore validate output — the opposite of what a gate should do.

Suggestions (smallest first)

1. rivet validate --severity error (and --min-severity warn) — filter output to a floor. Quickest win.
2. A summary line that separates signal from aspiration: e.g. 0 errors · 12 actionable warnings · 146 advisory (lifecycle/prose), with advisory collapsed by default.
3. Make prose-mention-without-typed-link smarter or opt-in: a REQ description that mentions another REQ id narratively shouldn't warn unless the mention looks like an intended dependency. At minimum let it be disabled per-rule.
4. Relatedly: the unknown-field noise from Marking a sw-req 'implemented' + verified requires the full ASPICE design chain — no direct test→sw-req link, and the completeness error doesn't guide you there #350 (REQ-134) and these advisory warnings share a root cause — validate has no notion of "advisory vs actionable" tiers below error/warning/info.

What I used / what would have helped

I read the full rivet validate output and visually filtered for ERROR: — a --severity error flag would have replaced that. Counting warnings by rule (rivet validate | grep WARN | sed ... | sort | uniq -c) revealed that 2 rules produce the bulk of the noise; a built-in validate --by-rule summary would surface that directly.

Environment: rivet 0.14.0-track (089ec6f), rivet's own repo (dev schema).

[avrabe]

Triage (issue-triage agent run 2026-05-31): the body lays out the friction cleanly and the "Suggestions (smallest first)" list is well-shaped, but it's framed as alternatives to consider — not a bulleted, testable set of done-statements. For autonomous pickup under the oracle-gated workflow this needs an explicit ## Acceptance Criteria block. The suggestions are tractable as separate, small PRs — please commit to which slice the agent should take.

Sketch of the shape expected (matching #237 / #249 / #248 and the format the in-flight PR #361 uses for cluster work):

## Acceptance Criteria — slice 1 (severity floor)

- [ ] `rivet validate --severity {error,warn,info}` filters diagnostics to ≥ the given floor; default unchanged (`info`)
- [ ] `--format json` honours the same filter (diagnostics array contains only entries at or above the floor)
- [ ] Exit code unchanged: only `error`-level diagnostics fail the run, regardless of `--severity`
- [ ] Summary footer prints filtered vs total: `Result: PASS (0 errors, 12 actionable warnings shown, 146 advisory suppressed)`
- [ ] Regression test asserts the filter on a fixture with mixed severities
- [ ] `rivet validate` on the rivet repo still PASS with no flag
- [ ] Commit trailer: `Implements: REQ-XXX, Refs: #357`

Optional follow-up slices that can be split into their own issues once slice 1 lands:

• slice 2 — by-rule summary (--by-rule group-and-count)
• slice 3 — per-rule disable (prose-mention-without-typed-link opt-in)
• slice 4 — advisory tier below warn (overlaps with the unknown-field noise from Marking a sw-req 'implemented' + verified requires the full ASPICE design chain — no direct test→sw-req link, and the completeness error doesn't guide you there #350 / REQ-134; might be better tracked there)

Slice 1 is the smallest single-PR target and matches the "smallest first" framing in the body. Once the AC block is on the body the agent will pick it up next pass.

Not implemented this run.

---

Generated by Claude Code

[avrabe]

Implemented the highest-value suggestion (the --severity filter) in #366.

rivet validate --min-severity <error|warning|info> now displays only diagnostics at or above the floor. On rivet's own repo:

$ rivet validate --min-severity error
... Result: PASS (163 warnings)
(showing 0 of 356 diagnostics at or above 'error'; counts above are the full set)

So an agent doing mark-implemented work can run --min-severity error and immediately see there's nothing to act on, instead of scrolling 160 advisory warnings. Crucially it's display-only: the summary counts and exit code are computed from the full set and unchanged, and a one-line note reports how many were suppressed (no silent dropping — loud-fail register).

Reused the existing --fail-on severity parser, so the two flags compose: --fail-on warning --min-severity warning both gates and shows warnings. Regression test added.

The other two suggestions remain open as follow-ups:

• Advisory/actionable summary tier ("0 errors · 12 actionable · 146 advisory") — needs a notion of diagnostic tiers below error/warning/info; bigger change, tracked.
• Smarter prose-mention-without-typed-link (don't warn on narrative REQ cross-references) — separate, also tracked.

Filed as REQ-137 (implemented).

[avrabe]

The headline ask (a severity filter) shipped in #366 (merged): rivet validate --min-severity error displays only diagnostics at/above the floor, with a note of how many were suppressed — counts/exit unchanged. Closing. The two follow-ups remain tracked: an advisory/actionable summary tier, and smarter prose-mention-without-typed-link.