ContextTrace v0.7.0

ContextTrace v0.7.0

ContextTrace v0.7.0 adds local RAG regression suites: a workflow for turning saved RAG failures into repeatable endpoint tests and CI gates.

This release builds on v0.6.0 evidence QA. Instead of only verifying one trace, developers can now capture a failure, save it as a suite case, replay the same query against the current RAG endpoint, compare evidence quality against the baseline, and fail when the issue still reproduces or a good case regresses.

Highlights

• Regression suite creation with contexttrace suite create
• Endpoint replay with contexttrace suite run
• Suite reports with contexttrace suite report
• Suite case management with suite add, suite list, suite remove, and suite prune
• CI-friendly gates for failed cases, endpoint errors, unsupported claims, should-abstain failures, high/medium risk, and regressions
• Local HTML suite reports showing failed cases, resolved failures, risk levels, primary issues, and current evidence QA summaries
• GitHub Actions starter workflow at examples/contexttrace-suite-ci.yml
• Validation story for the saved-failure-to-CI loop

Core Workflow

contexttrace suite create examples/suite/refund_processing_failure.json \
  --name refund-suite \
  --out .contexttrace/suites/refund-suite.json

contexttrace suite add .contexttrace/suites/refund-suite.json traces/new_failure.json

contexttrace suite list .contexttrace/suites/refund-suite.json

contexttrace suite run .contexttrace/suites/refund-suite.json \
  --endpoint http://localhost:8000/query \
  --report \
  --fail-on failed_case \
  --fail-on error

contexttrace suite report .contexttrace/suites/refund-suite_results.json