Conversation
#184) Add `raki gate-check <report.json>` command that evaluates a saved RAKI report against SODA-tuned quality thresholds. Default gates (SODA_DEFAULT_GATES): first_pass_success_rate>=0.6 and rework_cycles<=0.5, derived from v0.8.0 baseline data. These defaults apply when no --gate flags or manifest thresholds are provided. Priority chain: CLI --gate > manifest thresholds > SODA defaults. Supports: - --baseline for regression detection across milestones - --gate for custom threshold override - --manifest (-m) to read thresholds: block from a manifest file - --require-metric to fail on N/A metrics - --json for structured CI output (passed, gates, regressions) - --quiet to suppress non-essential output Exit codes match the existing gate/regression scheme: 0=pass, 1=threshold violation, 3=regression, 4=both. Assisted-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Assigned-by: decko
…#184) Test coverage for the new gate-check command: - Default SODA gates pass/fail with appropriate exit codes - Custom --gate overrides defaults entirely - Manifest thresholds: block overrides defaults when --gate not given - CLI --gate wins over manifest thresholds (priority chain) - --baseline regression detection (exit 3) and clean run (exit 0) - --json outputs structured JSON with passed/gates/regressions keys - --quiet suppresses Quality Gates output - Exit 2 for missing report file - --require-metric fails on N/A metric Also fixes gate-check --json mode: suppress human-readable output when --json is active so stdout contains only the JSON object. Assisted-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Assigned-by: decko
Ready-to-use RAKI manifest for evaluating SODA pipeline sessions. Points at .soda/ session directory with SODA-tuned thresholds based on v0.8.0 cycle baseline data: - first_pass_success_rate>=0.6 (baseline: 0.44, target: 0.60) - rework_cycles<=0.5 (baseline: 0.72, target: 0.50) - review_severity_distribution>=0.5 (baseline: 0.63) - cost_efficiency<=10.0 (baseline: $7.28 avg) Use with: raki gate-check <report.json> -m examples/soda-gate.yaml Assisted-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Assigned-by: decko
…#184) New standalone doc covering the full RAKI-as-gate workflow for SODA pipeline evaluation: 1. Overview — motivation and metrics (first_pass_success_rate, rework_cycles, review_severity_distribution, cost_efficiency) 2. Prerequisites — dependencies and required prior raki run 3. Workflow — 5-step process from evaluation to trend visualization 4. Baseline thresholds — v0.8.0 table with targets and gate expressions 5. Customizing thresholds — --gate flag, manifest thresholds:, priority chain 6. CI integration — GitHub Actions workflow snippet for milestone gate checks 7. Interpreting results — diagnosis guide for each failing gate, diff and trends commands for follow-up investigation Assisted-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Assigned-by: decko
Add '## SODA pipeline quality gate' section before the Tips section. Covers: - The gate-check command and its SODA-specific defaults - Quick-start 4-step workflow (run → gate-check → baseline compare → manifest) - GitHub Actions workflow snippet for milestone gate checks - Link to docs/soda-pipeline-gate.md for the full workflow Assisted-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Assigned-by: decko
Assisted-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Assigned-by: decko
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
raki gate-checkcommand for periodic SODA pipeline quality evaluationexamples/soda-gate.yamlfor SODA session evaluationdocs/soda-pipeline-gate.md(workflow guide) +docs/ci-integration.md(CI section)Test plan
uv run pytest tests/ -v -m "not slow"— 1326 passedgate-checkcommand accepts manifest, thresholds, baseline reportRefs #184
🤖 Generated with SODA + Claude Code