Skip to content

fix(cli): add --gate and --require-metric flags to report subcommand#160

Merged
decko merged 2 commits into
mainfrom
soda/139
Apr 23, 2026
Merged

fix(cli): add --gate and --require-metric flags to report subcommand#160
decko merged 2 commits into
mainfrom
soda/139

Conversation

@decko
Copy link
Copy Markdown
Owner

@decko decko commented Apr 23, 2026

Summary

  • Adds --gate and --require-metric flags to the report subcommand, achieving flag parity with the run subcommand
  • Implements correct exit codes: 0 (all gates pass), 1 (gate violation), 2 (invalid syntax/threshold parse error)
  • N/A metrics are skipped (not treated as failures) unless the metric is listed in --require-metric, in which case they are counted as failures

Acceptance Criteria

  • --gate option added to report subcommand (multiple=True, same semantics as run)
  • --require-metric option added to report subcommand for fail-on-N/A semantics
  • Exit code 0 when all gates pass
  • Exit code 1 when any gate is violated
  • Exit code 2 on invalid threshold syntax
  • N/A metrics are skipped unless marked required via --require-metric
  • -q/--quiet suppresses output while still applying gates and returning correct exit codes
  • Flag naming and help text consistent with run subcommand

Review Results

Python Perspective

  • MINOR: HTML confirmation message (cli.py:823) printed in quiet mode — inconsistent with run subcommand which guards it with if not quiet:
  • MINOR: --gate/--require-metric silently ignored when combined with --diff (follows pre-existing pattern but could confuse users)

Security Perspective

  • MINOR: Gate parse error at cli.py:840 not wrapped in redact_sensitive() — inconsistent with other error paths (pre-existing in run subcommand too)

Documentation Perspective

  • MINOR: ci-integration.md doesn't document the new raki report --gate workflow for post-hoc gating of saved reports

All findings are minor and do not block approval. Core functionality is correct and well-tested (10 new TestReportGates tests: 822 passed, 1 skipped, 0 failures).

Refs #139


Assisted-by: Claude Opus 4.6 (1M context) noreply@anthropic.com
Assigned-by: decko

decko and others added 2 commits April 23, 2026 07:30
…139)

The report subcommand was missing --gate and --require-metric flags that
exist on the run subcommand, making it impossible to apply quality gates
against a previously-saved JSON report without re-running the evaluation.

Adds --gate, --require-metric, and -q/--quiet options to `raki report`
so that CI pipelines can gate on an existing report file:

    raki report report.json --gate "faithfulness>0.80" --require-metric faithfulness

Also adds -q/--quiet to suppress the summary and gate output while
still applying the gate and returning the correct exit code.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Documents the addition of --gate, --require-metric, and -q flags
to the report subcommand.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@decko decko added the ai-assisted Implemented with AI assistance label Apr 23, 2026
@decko decko merged commit 7d98bb6 into main Apr 23, 2026
4 checks passed
@decko decko deleted the soda/139 branch April 23, 2026 15:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-assisted Implemented with AI assistance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant