ContextTrace v1.1.0

ContextTrace v1.1.0

ContextTrace 1.1.0 freezes the calibrated semantic verifier as
semantic_v1_calibrated and establishes the contracts needed for a genuinely
untouched evaluation track.

This release adds versioned public artifact schemas, strict privacy controls,
bounded and streaming-safe integrations, concurrent run isolation, explicit
verification limits, golden v1 trace compatibility coverage, and adversarial
release tests. It also raises the coverage gate to 80% and expands Python,
optional-integration, dependency-audit, and cross-platform wheel checks.

The exact release artifacts were first published and smoke-tested as
1.1.0rc1 on TestPyPI. Existing RAGTruth, Diag-150, and Naturalistic Eval v2
results remain calibration evidence and are not independent test results.

ContextTrace v1.0.0

{"value" => "# ContextTrace v1.0.0\n\nContextTrace 1.0 is the first stable release of the local-first evidence-chain\nforensics SDK and CLI for RAG and AI agents.\n\n## Highlights\n\n- Trace failures from retrieval and selected context through claims, citations,\n evidence spans, verdicts, root causes, repair guidance, and regression tests.\n- Diagnose RAG and agent traces locally with lexical, semantic, local-ML, NLI,\n or explicitly configured judge modes.\n- Capture existing endpoints, compare runs, audit retrieval, generate static\n HTML reports, and replay saved failures in CI suites.\n- Integrate with LangChain, LlamaIndex, FastAPI, LangGraph, and OpenTelemetry.\n- Keep SQLite state, traces, reports, caches, and judge data local by default.\n- Separate grounding, truth, source authority, freshness, and citation status\n so evidence support is not presented as independent correctness.\n\n## Benchmark Evidence\n\nThe versioned benchmark archive contains the checksummed RAGTruth, ARES, CRAG,\nand ContextTrace-Diag-150 evidence used for the current readiness report.\n\nThe broad SOTA gate remains fail-closed at 8/10: primary RAGTruth independent\nreview and Diag-150 independent sign-off are still open. This release therefore\nclaims a stable, benchmarked product, not broad state-of-the-art performance.\n\n## Install\n\nbash\npip install contexttrace==1.0.0\ncontexttrace init\ncontexttrace verify-demo unsupported_claim --report\n\n\n## Upgrade\n\nbash\npython -m pip install --upgrade contexttrace==1.0.0\ncontexttrace --version\ncontexttrace doctor\n\n\nThe stable release preserves the existing local-first CLI and SDK behavior from\nthe 0.9 series while formalizing the supported public surface.\n", "PSPath" => "C:\Users\mannv\OneDrive\Desktop\Projects\ContextTrace\release\v1.0.0.md", "PSParentPath" => "C:\Users\mannv\OneDrive\Desktop\Projects\ContextTrace\release", "PSChildName" => "v1.0.0.md", "PSDrive" => {"CurrentLocation" => "Users\mannv\OneDrive\Desktop\Projects\ContextTrace", "Name" => "C", "Provider" => "Microsoft.PowerShell.Core\FileSystem", "Root" => "C:\", "Description" => "", "MaximumSize" => nil, "Credential" => "System.Management.Automation.PSCredential", "DisplayRoot" => nil}, "PSProvider" => {"ImplementingType" => "Microsoft.PowerShell.Commands.FileSystemProvider", "HelpFile" => "System.Management.Automation.dll-Help.xml", "Name" => "FileSystem", "PSSnapIn" => "Microsoft.PowerShell.Core", "ModuleName" => "Microsoft.PowerShell.Core", "Module" => nil, "Description" => "", "Capabilities" => 52, "Home" => "C:\Users\mannv", "Drives" => "C"}, "ReadCount" => 1}

ContextTrace v0.9.0

ContextTrace v0.9.0

ContextTrace v0.9.0 adds local-first claim grounding upgrades for RAG and agent systems: span-scoped verification, optional local NLI, calibration tooling, and source trust/freshness scoring.

Highlights

• Separated grounding from truth with support_status, truth_status, and source_status.
• Span-scoped judge verification so local LLM judges see selected evidence spans with offsets, not broad retrieved chunks or fluent answer prose.
• Optional local NLI mode with --mode nli, contexttrace[nli], and contexttrace[nli-onnx].
• Local NLI calibration with contexttrace nli-calibrate for exact match, entailment precision, contradiction recall, dangerous false-green rate, citation match, and latency.
• Source trust and freshness scoring from local trace metadata: source_authority, source_timestamp, source_version, canonical, and canonical_source.
• Source-aware statuses including grounded_but_stale, grounded_but_conflicted, grounded_by_low_authority_source, and supported_by_canonical_source.
• Verification reports now include Source Trust & Freshness details.

Local NLI Calibration Note

The bundled calibration command was tested locally with MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli downloaded to a local model directory.

Observed calibration on the bundled all case set:

• exact match rate: 0.667
• entailment precision: 1.000
• entailment recall: 0.571
• contradiction precision: 0.909
• contradiction recall: 1.000
• dangerous miss rate: 0.000
• unsupported marked supported: 0
• p95 case latency on CPU: 1644.758 ms

Interpretation: this model was strict and avoided false green supported labels, but it did not pass ContextTrace's default exact-match gate. Treat --mode nli as an optional local verifier that should be calibrated against your data before CI gating.

Example

pip install "contexttrace[nli]"
set CONTEXTTRACE_NLI_MODEL_PATH=C:/models/contexttrace-nli

contexttrace verify trace.json --mode nli --report
contexttrace nli-calibrate --case-set all --report

Validation

Validated locally before release with:

pytest packages/contexttrace/tests -q
pytest apps/api/tests -q
python -m contexttrace.cli verify-benchmark --mode semantic --case-set all --json
python -m contexttrace.cli verify-benchmark --mode local_ml --case-set all --json
powershell -ExecutionPolicy Bypass -File scripts/release_check.ps1

ContextTrace remains a local diagnostic and regression tool. Grounded means supported by selected evidence; it does not mean independently true, current, or authoritative.

ContextTrace v0.8.0

ContextTrace v0.8.0

This release makes ContextTrace much stronger while keeping the core promise local-first: local traces, local reports, local judge cache, and no remote model calls unless explicitly configured.

Highlights

• Added first-class local judge support for Ollama, LM Studio, vLLM, and local OpenAI-compatible servers.
• Added --mode local_ml with offline hash-embedding similarity by default and optional local SentenceTransformers model support via contexttrace[local-ml].
• Added contexttrace judge-calibrate to score judge quality against bundled golden RAG failure cases.
• Added local judge caching at .contexttrace/judge_cache.json.
• Added local-only guardrails that block remote judge URLs while local_only: true is active.
• Improved semantic contradiction checks for location, attribution, causal direction, version, number, URL, and negation conflicts.
• Expanded adversarial benchmark coverage for privacy, endpoint, numeric, version, and relation mistakes.
• Improved OpenTelemetry spans for retrieval, LLM answers, verification, and agent/tool events.
• Shortened and clarified the README for GitHub and PyPI.

Install

pip install -U contexttrace==0.8.0

ContextTrace v0.7.0

ContextTrace v0.7.0

ContextTrace v0.7.0 adds local RAG regression suites: a workflow for turning saved RAG failures into repeatable endpoint tests and CI gates.

This release builds on v0.6.0 evidence QA. Instead of only verifying one trace, developers can now capture a failure, save it as a suite case, replay the same query against the current RAG endpoint, compare evidence quality against the baseline, and fail when the issue still reproduces or a good case regresses.

Highlights

• Regression suite creation with contexttrace suite create
• Endpoint replay with contexttrace suite run
• Suite reports with contexttrace suite report
• Suite case management with suite add, suite list, suite remove, and suite prune
• CI-friendly gates for failed cases, endpoint errors, unsupported claims, should-abstain failures, high/medium risk, and regressions
• Local HTML suite reports showing failed cases, resolved failures, risk levels, primary issues, and current evidence QA summaries
• GitHub Actions starter workflow at examples/contexttrace-suite-ci.yml
• Validation story for the saved-failure-to-CI loop

Core Workflow

contexttrace suite create examples/suite/refund_processing_failure.json \
  --name refund-suite \
  --out .contexttrace/suites/refund-suite.json

contexttrace suite add .contexttrace/suites/refund-suite.json traces/new_failure.json

contexttrace suite list .contexttrace/suites/refund-suite.json

contexttrace suite run .contexttrace/suites/refund-suite.json \
  --endpoint http://localhost:8000/query \
  --report \
  --fail-on failed_case \
  --fail-on error

contexttrace suite report .contexttrace/suites/refund-suite_results.json

ContextTrace v0.6.0

ContextTrace v0.6.0

ContextTrace v0.6.0 adds a complete local evidence QA workflow for RAG outputs.

This release moves ContextTrace beyond separate debugging commands. Developers can now capture a RAG response, inspect trace quality, verify claim-level evidence, audit retrieval and corpus failures, and produce one local QA report with risk level and recommended next actions.

Highlights

• Evidence QA workflow with contexttrace qa trace.json --corpus docs/ --report.
• Trace inspection with contexttrace inspect trace.json.
• Endpoint capture with contexttrace capture endpoint.
• Saved response capture with contexttrace capture response response.json.
• Risk scoring in QA output, including risk level, primary issue, risk signals, and CI gates.
• Stronger retrieval audit diagnostics with failure stages, evidence status, diagnostic signals, failure paths, developer summaries, and recommended actions.
• Local QA HTML report covering inspection warnings, claim verification, optional corpus audit, risk signals, and next actions.
• Validation docs and harnesses for public-source RAG app runs and saved-response debugging workflows.

Core Commands

contexttrace capture endpoint --endpoint http://localhost:8000/query --query "What is the refund policy?" --out traces/refund_trace.json --verify --report

contexttrace capture response response.json --query "What is the refund policy?" --out traces/refund_trace.json --verify --report

contexttrace inspect traces/refund_trace.json

contexttrace qa traces/refund_trace.json --corpus docs/ --report

ContextTrace v0.5.0

ContextTrace v0.5.0

ContextTrace v0.4.0

ContextTrace v0.4.0 - Regression Diffing for RAG Grounding

ContextTrace v0.4.0 adds local regression diffing for claim-level evidence verification.

You can now compare a baseline RAG trace against a current trace and detect whether a prompt, retriever, chunking, or model change made grounding worse.

What’s New

• Added contexttrace compare baseline.json current.json
• Supports raw portable trace JSON and saved contexttrace verify --json outputs
• Added local HTML regression reports
• Added CI-friendly --fail-on gates
• Detects:
  • new unsupported claims
  • support-rate drops
  • citation mismatch regressions
  • should-abstain flips
  • new root causes
  • added, removed, changed, and resolved claims

Example

contexttrace compare baseline.json current.json
contexttrace compare baseline.json current.json --json
contexttrace compare baseline.json current.json --report
contexttrace compare baseline.json current.json --fail-on new_failure

ContextTrace v0.3.0

ContextTrace v0.3.0

ContextTrace v0.3.0 adds local claim-level evidence verification with root-cause diagnosis for RAG and AI agent outputs.

Instead of only saying whether an answer passed or failed, ContextTrace now shows where the evidence chain broke: unsupported claim, wrong citation, answer overreach, conflicting context, retrieval miss, or should-have-abstained.

What’s New

• Claim-level evidence verification with contexttrace verify
• Local HTML verification reports
• Typed required, matched, and missing fact diagnostics
• Evidence spans with character offsets and stable span hashes
• Multi-span evidence support across retrieved contexts
• Citation mismatch detection:
  • citation_ok
  • cited_source_missing
  • cited_source_does_not_support_claim
  • claim_supported_by_different_source
  • claim_has_no_citation
• Claim-level root-cause diagnosis:
  • retrieval_miss
  • answer_overreach
  • partial_context_support
  • wrong_source_cited
  • missing_cited_source
  • conflicting_contexts
  • stale_context
  • insufficient_context
  • should_have_abstained
• Real verification benchmark:
  • 32 ContextTrace repo-doc cases
  • 13 external OSS docs/GitHub issue cases
  • 45 total real cases with --case-set all
• Local benchmark HTML reports with misses to inspect

Example Commands

contexttrace verify trace.json
contexttrace verify trace.json --json
contexttrace verify trace.json --report
contexttrace verify-demo citation_mismatch --report

ContextTrace v0.2.0

ContextTrace v0.2.0

ContextTrace v0.2.0 adds local claim-level evidence verification for RAG traces.

This release helps developers answer a concrete question: is each generated claim actually supported by retrieved evidence?

Highlights

• contexttrace verify trace.json for portable RAG trace verification
• contexttrace verify-demo bundled proof demos for PyPI users
• Local HTML reports with claim verdicts, citation status, highlighted evidence, and suggested fixes
• Atomic claim decomposition for simple compound claims
• Verdicts: supported, partially_supported, unsupported, contradicted, unverifiable
• Citation mismatch detection
• Should-abstain detection
• CI gates with --fail-on
• --mode semantic for local paraphrase-aware matching
• contexttrace verify-benchmark with exact-match rate and per-label precision/recall/F1

Quickstart

pip install contexttrace==0.2.0