v0.2.2 — AgentMeasure

v0.2.2 — Conformance Hardening

Released automatically from docs/RELEASE-v0.2.2.md by
.github/workflows/release.yml (tag v0.2.2).
Both fixes in this release were found by the first external conformance
pass (Urusilla / @jaden3824, langfuse#16383)
and were committed "before the next tag" per
#8 and
#9. This release is
that commitment, delivered.

---

The two trust-debt fixes

#8 — Validator: root sibling constraints after oneOf

The bundled validator returned after a oneOf branch matched, so records
that matched a branch but were missing root-level required fields (siblings of
the root oneOf in FMT-002) validated successfully. Invalid records passed.
oneOf/anyOf are compositional keywords, not terminators: sibling keywords
are now always evaluated after a branch matches, and zero-match errors carry
the first branch error for diagnosis. The registry validator additionally
fails closed on unsupported composition keywords instead of silently ignoring
them. 6 regression tests added.

#9 — Aggregator: reconcile declared operation summaries against attempt rows

The lab aggregator trusted operation_result declarations (attempts,
outcome) without checking the underlying attempt rows — a declared
"4 attempts, success" was taken at face value even if the rows said otherwise.
Now every declaration is reconciled per operation:

• declared attempt count vs actual attempt rows
• declared outcome vs the rows' derived outcome (rule op-success-any/1,
  last-attempt outcome disclosed for diagnosis)

Mismatches surface as an explicit operation_reconciliation: failed block
with per-operation reasons — counts and outcomes use measured rows, never
declarations. The canonical collector gained the same discipline:
task_outcome payloads persist task_success/attempt_count (optional,
never trusted) and compute() emits operation_summary_reconciliation.
Gate scenarios core-6/core-7 cover consistent, conflicting, and absent
declarations.

First external conformance vector: urusilla-001

The conformance pass that found #8/#9 is now an upstream regression vector —
conformance/vectors/external/urusilla-001/:

• 8 project-authored synthetic events (FMT-002): one operation with a
  fail→fail→fail→success retry chain, fallback and cache roles preserved in a
  sidecar, unreduced token usage (25 units)
• expected.json pins every metric (reach 1, selected 1, operations 1,
  4 attempts/op, cost 25, success/consumption 1.0) and
  schema_valid_under_fmt_002: true
• the original mapping document ships alongside

conformance/runners/run_external_fixture.py guards four things:
schema validity, #8 (stripping each root-required field must be rejected),
exact metric reproduction, and #9 (tampered declarations must surface
reconciliation: failed, never silent trust).

Claim boundary (unchanged from the source project): this fixture is
synthetic, project-authored evidence. It is not an AgentMeasure endorsement,
a Langfuse adoption, an external reproduction, or a real provider-cost
observation.

Conformance bundle (reproduce the pass)

Everything needed to reproduce the external pass against the fixed code:

git checkout v0.2.2
python3 conformance/runners/run_external_fixture.py
# URUSILLA-001 CONFORMANCE PASS: schema + #8 + metrics + #9 all guarded

python3 -m unittest discover -s lab/tests          # 84 tests OK
python3 scripts/canonical_core_gate.py             # 7 scenarios PASS
python3 verify_vectors.py                          # ALL VECTORS PASS

Bundle contents (all in-repo, versioned with this tag):

Artifact	Path
Events (FMT-002)	conformance/vectors/external/urusilla-001/agentmeasure_urusilla_fixture_001.events.jsonl
Expected metrics	…/agentmeasure_urusilla_fixture_001.expected.json
Mapping / methodology	…/AGENTMEASURE_URUSILLA_MAPPING_2026-08-23.md
Guard runner	conformance/runners/run_external_fixture.py
Fixes	lab/agentmeasure_lab/schemas.py, lab/agentmeasure_lab/analysis.py, reference/collector/aggregator/aggregator.py, reference/collector/correlator/correlator.py

Also in this release

• CI runs the full lab test suite (lab/** previously did not trigger
  workflows) — pending the workflow-scope commit; see repo activity for
  landing status
• README (EN/中文) two-layer first fold: the measurement pain line first,
  the Capability Economy second
• Website hero: "Know what your agent usage metrics actually count."

Verification at tag time: 84 lab tests OK · core gate 7 scenarios PASS ·
all vectors PASS · registry + metric registry + canonical observation VALID ·
urusilla-001 guards PASS.

v0.2.1 — codex adapter live-validated

Live validation on a real harness 🎯

The codex adapter is now live-validated (codex-cli 0.149.0-alpha, gpt-5.6-sol): the first preregistered controlled A/B on a real agent.

What the live run produced (4 tasks × 2 variants, description-clarity A/B):

arm	subject selection	ops	success	consumption	real tokens/op	median steps
control (plain description)	0/4	4	100%	4/4	188,520	9.5
clear (rewritten description)	1/4	4	100%	4/4	215,021	22.5

Verdict: honest null (+25pp observed at n=4/arm, p=0.29) with next-round sizing guidance (≈31/arm) — exactly the discipline the engine exists to enforce. A weak default description losing to better-described competitors matches Hasan et al. (arXiv 2602.14878).

Fixes found by the live run (scripted tests could not catch these): mcp_tool_call item parsing; --approve-for-me for MCP calls in exec mode; stdin DEVNULL; toolserver contract fix (spec id→name, crashed tools/list) with a contract regression test; candidate-set steering in the episode prompt; token usage metered as cost units (1 unit = 1 token) so the budget breaker bounds real spend.

claude-code shares these fixes and remains scripted-transcript-tested pending its own live run. 78 tests.

v0.2.0 — Whitepaper v0.3 + AgentMeasure Lab v0.4 (open experiment engine)

What's in this release

AgentMeasure Lab v0.4 — the open experiment engine for the agent channel. Zero dependencies (Python 3.9+ stdlib only), fully offline, no registration.

python3 lab/am lab selftest     # planted uplift recovered + honest null, in seconds
python3 lab/am lab init         # workspace + example experiment
python3 lab/am lab preregister am-lab/experiments/example-manifest.json
python3 lab/am lab run am-lab/experiments/example-manifest.prereg.json

• Preregistered experiments (hypothesis / primary metric / guardrails / analysis plan hashed before the run; tampering refuses to run) with a scale / power / budget preview
• Honest statistics: Wilson/Newcombe intervals, honest nulls with next-round sizing guidance, "undetermined" with required n — never an underpowered verdict
• Fake-growth rejection at the decision exit: a selection uplift that loses consumption is verdicted unverified_growth — do not ship; margin is computed with the measured (lower) consumption
• Guardrails & dominance: significant wins that breach guardrails are effective_not_qualified; candidates that make no more money at higher cost are called out as dominated
• Bilingual decision-maker one-pager at the top of every report (conclusion / uplift / monthly margin / certainty / recommended action)
• Calibration loop (am lab calibrate): offline vs production re-test, per-condition transfer effects with intervals — never a single global transfer coefficient; not_comparable gaps are named, never assumed
• Connector data plane (am connector): three-tier per-class authorization, immediate revocation, HMAC-signed aggregate-only exports
• Read-only MCP interface (am mcp serve) so agents and CI query the same evidence engineers see
• Real harness adapters: claude-code (full; integration-tested against scripted transcripts — first live runs are validation runs) and codex (experimental), with candidate-set injection via a local MCP tool server
• Open formats (FMT-001..004) and a Core 0.4.4 mapping doc; local experiment history (am lab history); 74 tests

Whitepaper v0.3 (EN + zh-CN, structurally in sync): experimentation & calibration as first-class measurement semantics; anti-fake-growth elevated to a core thesis (with the x402 wash-trading evidence); data-rights claim tiers in the observation-surfaces chapter; the value formula bridging measurement to margin; new evidence (Hasan et al. arXiv 2602.14878, BiasBusters ICLR 2026, Arcade ToolBench).

Also: baseline funnel in product/local-analytics.py (with explicit provider-side observability boundaries), README front-door for Lab (EN/zh).

Honest limits, stated up front: the shipped demo runs on a synthetic harness with planted ground truth (validates the engine, not real-agent claims); production ingestion requires a data-rights agreement (G0); adapter live-CLI validation is pending.

v0.1.1 — AgentMeasure

v0.1.1 — External Ready

Released automatically from docs/RELEASE-v0.1.1.md by
.github/workflows/release.yml (tag v0.1.1).
Supersedes v0.1.0 — the original release contained several
measurement-model issues (reported 42/126 observations, an unverifiable
"no self-reported numbers" claim, and a non-reproducible demo); all are
documented and fixed in the changelog. History is kept, not deleted.

---

What's in this release

The measurement layer for the Agent Capability Economy — a review-driven
convergence that makes the SDK safe to hand to the first external provider.

Standard (Draft 0.4.3)

• Canonical Observation Envelope + 6 payload types; qualification resolution
  (Context × Validity); observe-first with unknown defaults
• Vocabulary single source of truth: registry/vocabularies.yaml drives the
  schema enums, TypeScript unions and Python tuples (CI-checked)
• Validity discipline: providers may only claim validity values they can
  know; normal is collector-derived; provider_configuration validity is
  never strong qualification

SDK @agentmeasure/mcp v0.1.1

• Non-blocking: emit() only enqueues; background flusher batches to rotating
  spool files (dir 0700 / files 0600); loss accounting in bufferHealth and in
  every persisted batch's collection_health
• Per-request caller resolution (v2 _meta.clientInfo / v1 _meta.sessionId
  echo); server-level claim is fixture fallback only
• MCP SDK v2 (@modelcontextprotocol/server) as the primary path, v1 compatible;
  the SDK itself is MCP-version-agnostic (no MCP dependency)
• Lineage snake_case (operation_id / task_id / retry_of)
• 21 tests (schema / fail-open / privacy / lineage / caller / concurrency /
  buffer / mcp-integration v1+v2); deterministic demo fixture — same fixture +
  same policy = bitwise-identical semantic output (42 calls → 84 observations,
  6 failures, fixed latency sequence)

Product

• Local analytics: --days, latency histogram (p50/p95), caller attribution,
  spool directory/glob ingestion
• Pipeline Validation #001 (deterministic baseline); Measurement Report #001
  is reserved for the first external provider

Benchmark (Draft 0.3)

• Multi-axis Evidence Profile per TRUST (Authentication strictly A0/A1/A2) +
  Source Attribution axis; no composite scores
• Claim Completeness (Qualification is a Core term); primary-source rule —
  demonstrated by correcting claim #003 to its primary source (Ahrefs
  server-log/web-analytics study of 137,210 domains, May 2026)
• Machine-readable benchmark/claims/*.json with excerpt+hash snapshots

Correction to v0.1.0 messaging

v0.1.0 said "No self-reported numbers. Measurement happens at the callee
boundary." That overstated the guarantee. The accurate statement:

Provider-observed rather than caller-self-reported; stronger claims still
require independent corroboration or attestation (TRUST §3).

---

Install

npm install @agentmeasure/mcp   # publish pending npm scope + token — see CHANGELOG
# or: git clone https://github.com/roy-tong/AgentMeasure && cd sdk && npm install
# or: download the attached agentmeasure-mcp-0.1.1.tgz and:
#     npm install ./agentmeasure-mcp-0.1.1.tgz

Demo

./examples/demo-e2e.sh   # isolated workspace, deterministic, 2 minutes

v0.1.0 — First public release: the measurement layer for the Agent Capability Economy

Superseded by v0.1.1. The original release contained several measurement-model issues (reported 42/126 observations, an unverifiable "no self-reported numbers" claim, and a non-reproducible demo); all are documented and fixed in the changelog. History is kept, not deleted.

The measurement layer for the Agent Capability Economy

AgentMeasure is an open standard + reference implementation for measuring how AI agents discover, choose, use, and derive value from software capabilities — Reach → Choice → Use → Utility → Value.

This is the first public release. It ships the pieces needed to start instrumenting real agent-facing software today, with strict honesty about what is and isn't proven yet.

What's in this release

Standard (Draft 0.4.3)

• Core specification: canonical observation schema, 6 payload types, validation, qualification resolution — standard/CORE.md
• Metrics registry: 14 metrics with a single source of truth and bidirectional consistency checks — registry/metrics.yaml
• Measurement integrity: observe-first policy, caller identity graded declared → correlated → attested (never guessed), validity defaults to unknown until evidence upgrades it

SDK

• @agentmeasure/mcp v0.1.0 — TypeScript SDK for MCP servers, built on the official MCP SDK. Fail-open, zero-content by default, missing-call accounting — sdk/

Product (reference implementation)

• Canonical end-to-end pipeline: adapters → canonical observations → collector → metrics — product/
• local-analytics.py: six classes of trustworthy metrics, verified end-to-end with 42 calls / 126 observations / 0 schema rejections (synthetic traffic, honestly labeled)
• Measurement Report #001 — first measurement, with every limitation stated

Docs

• Whitepaper: Measuring Software Used by AI Agents (EN) · 白皮书（中文）
• Roadmap · Governance · Contributing

Design principles

1. Observe first, qualify later. We default to unknown, not "this was production agent usage." Only evidence upgrades it.
2. No self-reported numbers. Measurement happens at the callee boundary, so callers can't inflate their own stats.
3. Not a payment rail, marketplace, or ranking. We standardize the economic facts those systems need — measurement before monetization.

Get involved

• Report where our definitions break: Issues
• Debate metric semantics: Discussions
• Build on the SDK and tell us what you measure

MIT licensed. Built in the open, in public.