Skip to content

context-report 0.1.0: an open, re-derivable report format for agent context artifacts

Latest
Latest

Choose a tag to compare

View all tags
@jothimani-rajendran jothimani-rajendran released this 06 Sep 23:51
· 88 commits to main since this release
v0.1.0
6f2134a
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

context-report 0.1.0 is the first public release: an open report format for whether an agent context artifact (a plugin, hook, skill, instruction file, MCP server or subagent) actually works, with a reference producer and verifier, a run manifest for paired-ablation measurement, and three measurements you can re-derive.

What it is

A report is an in-toto Statement. Each row says what was measured and on what basis: re-derivable rows recompute from the artifact and carry an inputHash, so a catalog can check them without trusting the author; claimed rows (efficacy) are labelled as the author's claim, bound to a model and a date. No row is a verdict; thresholds belong to the catalog that reads the report.

Commands

  • context-report produce: reachability of a plugin's hooks (discovered from the plugin itself), malformed-input behaviour, latency, context weight, conformance.
  • context-report verify: schema validity, and with --subject, that every re-derivable row recomputes.
  • context-report run run.json: every subject against every model over your tasks, written as claude plugin eval case directories. Every run is kept under out/runs/<id>/; out/SUMMARY.md lays runs side by side; --resume continues an interrupted run without re-spending recorded calls.
  • context-report compare --history --rules: per rule, adherence with and without it in every run, and what the newest run changed.
  • context-report judge: re-grade recorded transcripts with another judge model.
  • Two model backends, anthropic and claude-cli; no model is hardcoded, every model comes from the manifest.

Library

from context_report import validate, verify, produce_statement, load_manifest, run and friends; see docs/library.md for the catalog-side example. Typed, Python 3.10 to 3.13.

Measurements in the paper

  • chock's 88 plugin bundles across Claude Code, Codex CLI, GitHub Copilot and Cursor
  • 18 public Claude Code plugins: three ship a hook that never starts (a dispatch script without its execute bit); hooks that shell out to npx cost roughly 20 times a local script
  • seven third-party instruction files and skills, three of them ablated on opus, sonnet and fable with a fixed judge: at four observations per arm no row reaches PASSED, a prompt-injection rule moved nothing on any model, a naming rule was the one consistent positive

Every statement is committed under paper/measurements/ and re-derives from the recorded commits.

Known limits

Fault rows that need a live client (timeout, crash posture) are NotAvailable in v0.1. Efficacy is always a claimed row. The tokenizer is an approximation. Coding agents only for now.

Install

pip install context-report
pip install "context-report[efficacy]"   # adds the anthropic backend

Spec: https://open-coder-ai.github.io/context-report/ · Paper: paper/context-report.md · Licence: Apache-2.0

Assets 2
Loading