Skip to content

v0.2.0 — Whitepaper v0.3 + AgentMeasure Lab v0.4 (open experiment engine)

Choose a tag to compare

@github-actions github-actions released this 22 Aug 05:24
· 25 commits to main since this release

What's in this release

AgentMeasure Lab v0.4 — the open experiment engine for the agent channel. Zero dependencies (Python 3.9+ stdlib only), fully offline, no registration.

python3 lab/am lab selftest     # planted uplift recovered + honest null, in seconds
python3 lab/am lab init         # workspace + example experiment
python3 lab/am lab preregister am-lab/experiments/example-manifest.json
python3 lab/am lab run am-lab/experiments/example-manifest.prereg.json
  • Preregistered experiments (hypothesis / primary metric / guardrails / analysis plan hashed before the run; tampering refuses to run) with a scale / power / budget preview
  • Honest statistics: Wilson/Newcombe intervals, honest nulls with next-round sizing guidance, "undetermined" with required n — never an underpowered verdict
  • Fake-growth rejection at the decision exit: a selection uplift that loses consumption is verdicted unverified_growth — do not ship; margin is computed with the measured (lower) consumption
  • Guardrails & dominance: significant wins that breach guardrails are effective_not_qualified; candidates that make no more money at higher cost are called out as dominated
  • Bilingual decision-maker one-pager at the top of every report (conclusion / uplift / monthly margin / certainty / recommended action)
  • Calibration loop (am lab calibrate): offline vs production re-test, per-condition transfer effects with intervals — never a single global transfer coefficient; not_comparable gaps are named, never assumed
  • Connector data plane (am connector): three-tier per-class authorization, immediate revocation, HMAC-signed aggregate-only exports
  • Read-only MCP interface (am mcp serve) so agents and CI query the same evidence engineers see
  • Real harness adapters: claude-code (full; integration-tested against scripted transcripts — first live runs are validation runs) and codex (experimental), with candidate-set injection via a local MCP tool server
  • Open formats (FMT-001..004) and a Core 0.4.4 mapping doc; local experiment history (am lab history); 74 tests

Whitepaper v0.3 (EN + zh-CN, structurally in sync): experimentation & calibration as first-class measurement semantics; anti-fake-growth elevated to a core thesis (with the x402 wash-trading evidence); data-rights claim tiers in the observation-surfaces chapter; the value formula bridging measurement to margin; new evidence (Hasan et al. arXiv 2602.14878, BiasBusters ICLR 2026, Arcade ToolBench).

Also: baseline funnel in product/local-analytics.py (with explicit provider-side observability boundaries), README front-door for Lab (EN/zh).

Honest limits, stated up front: the shipped demo runs on a synthetic harness with planted ground truth (validates the engine, not real-agent claims); production ingestion requires a data-rights agreement (G0); adapter live-CLI validation is pending.