Skip to content

RAGOps v2.6.0

Choose a tag to compare

@thangldw thangldw released this 20 Nov 00:27
· 11 commits to main since this release
afbf916

RAGOps v2.6.0

New

  • ragops counterfactual: alters the evidence of each case and re-asks your own system (--answer-command). Perturbations: remove_support, swap_number (default), and opt-in drop_citation, inject_instruction (planted canary; one obeyed injection blocks) and abstain_without_evidence (--perturbation all). Exit 0 PASS, 2 BLOCK, 3 INCONCLUSIVE. See counterfactual-test.md.
  • ragops mine: turns flagged production logs (thumbs down, low rating, escalation) into new regression cases in a new directory.
  • ragops stability: blocks when repeated runs of the same questions vary too much, optionally against a baseline.
  • ragops triage and calibrate confidence band: from human labels, routes cases to auto-pass, auto-fail or human review.
  • Claude Code / Cowork plugin marketplace: /plugin marketplace add thangldw/ragops, then /plugin install ragops@ragops. The skill now covers the new commands. See claude-plugin.md.
  • Mutation testing now also covers the release policy engine; boundary tests pin gate semantics.

Limits

counterfactual was demonstrated on three toy systems only and has not been measured on a real RAG system; redundant evidence can cause false flags. mine, stability and triage were tested on constructed data, not real logs or labels. The Langfuse/OpenAI/Anthropic adapters still follow documented export shapes without a live export.

Compatibility

No breaking changes. The calibration report schema gains an optional confidence_band field.