bench: agent-effectiveness benchmark suite proving enrichment and the memory-insight-knowledge lifecycle (design)

[cjimti]

Purpose

The platform's core methodology — semantic-first data access, cross-enrichment, curated knowledge, and the memory-insight-knowledge lifecycle — is currently argued, not measured. We have micro-level numbers (raw_query speedups, coverage, mutation scores) but no evidence for the headline claim: an agent connected to this platform answers real data questions more correctly, more efficiently, and with less re-teaching than the same agent connected to bare data tools. This issue designs and delivers a benchmark suite that proves or disproves that claim with published numbers, and that then serves as the standing regression instrument for every future enhancement: any change to enrichment, search, knowledge, or memory must move these numbers or explain why it does not.

Why this is measurable at all: prior art

• BIRD (text-to-SQL) is the anchor precedent. It attaches hand-curated "external knowledge evidence" to each question and measures execution accuracy with and without it: GPT-4 scores ~54.9% WITH curated evidence and ~34.9% without — a ~20-point swing from knowledge alone (https://bird-bench.github.io/, analysis: https://datost.com/blog/text-to-sql-accuracy-benchmarks). BIRD supplies the evidence by hand, per question. This platform's entire premise is retrieving that evidence automatically (enrichment, search, knowledge pages, curated queries). Our benchmark measures exactly the gap BIRD proved exists, with retrieval replacing hand-curation. BIRD's Valid Efficiency Score is also precedent for scoring efficiency alongside correctness.
• MCP-family agent benchmarks establish harness mechanics: MCP-Atlas (1,000 human-verified tasks, LLM-judge scoring with retry handling and a 100-tool-call budget per task, https://labs.scale.com/leaderboard/mcp_atlas), MCP-Universe (execution-based evaluators, extensible framework, https://arxiv.org/pdf/2508.14704), MCP-AgentBench (outcome-oriented MCP-Eval scoring, https://arxiv.org/pdf/2509.09734), MCP-Bench (https://arxiv.org/pdf/2508.20453). All compare MODELS across fixed servers. None ablate the SERVER holding the model constant — that inversion is our design and, to our knowledge, novel.
• Memory benchmarks: LongMemEval scores five abilities — information extraction, multi-session reasoning, temporal reasoning, knowledge updates, abstention (https://arxiv.org/abs/2410.10813); LOCOMO scores multi-session recall with single-hop/multi-hop/temporal categories (https://snap-research.github.io/locomo/). Our memory subsystem already uses the LOCOMO dimension taxonomy (pkg/memory), so the lifecycle suite adopts these ability categories directly.
• tau-bench contributes pass^k: run each task k times and report the probability all k succeed, exposing reliability rather than best-of-k luck. Agent runs are stochastic; single-run numbers are not publishable.

Design

Principle 1: ablate the platform, not the model

Every run holds the model, prompt scaffold, seed data, and task set constant, and varies only the platform configuration. Four arms:

• A0 baseline: raw toolkit tools only (trino_, s3_), no search tool, no enrichment, gates off. Equivalent to wiring the standalone toolkit libraries.
• A1 enrichment: A0 + semantic cross-enrichment (DataHub context on describe/query results).
• A2 knowledge: A1 + the discovery layer (search-first gate, curated queries, knowledge pages, context documents, glossary).
• A3 lifecycle: A2 + memory/insight capture and apply_knowledge promotion, exercised by multi-episode protocols.

The platform's own machinery is the ablation mechanism: arms are personas plus config profiles (enrichment toggles, workflow.require_search, tool allowlists), so the benchmark needs no code forks — one deployment per arm profile, booted by the environment scripts. This is also a live demonstration that the config surface can express the ablation, which is itself a product property worth proving.

Principle 2: the audit log is the measurement instrument

Efficiency metrics are not instrumented in the harness; they are read back from the platform's own audit store via the admin API (/api/v1/admin/audit/events already records per-call tool name, parameters, duration, success/error, enrichment mode, and token counts, keyed by the dps_ session handle the harness controls). The harness tags each task run with its session handle and scores tool-call count, error count, wall-clock, and enrichment volume from audit rows. Dogfooding the audit pipeline makes the benchmark cheaper to build and doubles as a standing correctness check on audit itself: a task whose audit trail is incomplete fails the run.

Principle 3: seeded ground truth, deterministic graders first

The benchmark ships its own deterministic dataset generator (fixed RNG seed) extending the existing e2e seed: warehouse tables, denormalized analytics indices, DataHub metadata (descriptions, tags, glossary, curated queries, context documents), and knowledge-page fixtures. Because the generator knows the data, every analytical task's ground truth is computed by direct SQL at generation time and stored with the task. Grading is deterministic wherever possible:

• numeric answers: exact or tolerance match against stored truth
• discovery answers: entity/URN match
• SQL-producing tasks: execution-result comparison (BIRD-style execution accuracy)

An LLM judge is used ONLY for rubric items that are inherently judgment calls (below), with the judge model pinned, the rubric versioned in-repo, and a human-audited calibration subset (~30 items) whose judge agreement rate is published with the results. MCP-Atlas's evolution (judge upgrades changed scores) is the cautionary tale: version everything.

Task suites

S1 Discovery (~20 tasks). "Where would I find X / which dataset answers Y." Graded by entity match. Hypothesis: A2 >> A1 > A0 on accuracy and A2 << A0 on tool calls (search fans out once vs browse-walking the catalog).

S2 Analytical accuracy (~40 tasks). Numeric questions over the seeded data at BIRD-style difficulty tiers (single-table, join, temporal, top-N, cross-tab). Graded numerically. Hypothesis: enrichment narrows the schema-comprehension gap; knowledge (curated queries) narrows the query-formulation gap.

S3 Knowledge-dependent traps (~25 tasks). The distinctive suite. Each task is answerable plausibly-but-wrongly without the knowledge layer and correctly with it, because the disambiguating fact lives ONLY in knowledge pages/context docs/column descriptions, never in the schema. Seeded trap classes, each mirroring a real fixture the generator plants: gross-vs-net revenue semantics, a fiscal calendar offset from the calendar year, a data-freshness cutoff that makes the analytics index wrong for same-day questions, tier boundary definitions, a deprecated-table trap (the catalog marks a plausible table deprecated in favor of another), and a units trap (a column stored in cents with the fact recorded only in its description). Graded on the numeric answer plus a judged rubric item: did the answer carry the required caveat. This suite is the platform's BIRD-evidence experiment: A2 minus A0 here IS the methodology's value, in points.

S4 Efficiency (derived, no new tasks). Over S1-S3 from audit data: median/p90 tool calls, error-before-success count, wall-clock, tokens, and a composite efficiency-adjusted accuracy in the spirit of BIRD's VES. Also the raw_query-discovery rate: how often each arm finds the fast aggregation path vs scanning (discoverable only via knowledge in A2).

S5 Memory-insight-knowledge lifecycle (~15 protocols). Multi-episode scripts, each a sequence of fresh sessions:

1. Teach: episode 1 states a fact conversationally ("timestamps before March are UTC, after are America/Chicago"). Score: capture rate (insight recorded, correct sink-class, correct entity linkage) — read back via the admin insights API.
2. Personal recall: episode 2, same identity, fresh session, question requiring the fact. Score: recall accuracy and whether search surfaced the memory unprompted.
3. Promote: reviewer applies the insight via apply_knowledge (scripted, both sinks: entity description vs knowledge page).
4. Transfer: episode 3, DIFFERENT identity/persona, same question. Score: cross-user answer correctness — the teach-once-answer-forever claim.
5. Update/supersede (LongMemEval "knowledge updates"): a later episode corrects the fact; score that recall flips to the new value and the old insight is superseded, not duplicated (recall-first supersede is an existing product mechanism; this measures it).
6. Abstention (LongMemEval): questions about facts never taught; score that the agent says it does not know rather than fabricating — measuring whether the knowledge layer suppresses or amplifies hallucination.

Metrics: capture rate, personal recall@ep2, transfer rate@ep3, update correctness, duplicate rate, abstention rate. All lifecycle state is verified through the platform APIs, not inferred from transcripts.

Harness architecture

• bench/ directory: Go runner using the official go-sdk MCP client, with a provider-pluggable LLM adapter (model, temperature, max tool-call budget per task — adopt MCP-Atlas's budget approach, start at 30 — all pinned in the run manifest). Go-native rather than driving an external CLI so the loop is deterministic, versioned with the repo, and free of client-side behaviors we do not control.
• Task definitions as YAML in bench/tasks/: id, suite, prompt, arm-applicability, ground truth (value/URN/SQL), rubric refs, budgets. Ground truth generated, not hand-typed, wherever the generator can compute it.
• Environment: compose profile extending docker-compose.e2e.yml with DataHub (heavy is acceptable: this runs scheduled/manual, not per-commit) plus seed scripts; make bench-up, make bench-run ARM=a2 SUITE=s3 K=3, make bench-report, make bench-down. A --smoke profile (~10 tasks, 1 arm-pair, k=1) for quick sanity.
• Runs are k-repeated (default k=3); report pass^k, medians, and bootstrap CIs. Every run emits a manifest (git commit, platform version, model id, seed, judge version, task-set hash) and a results JSON; the report generator renders the arm-comparison tables.

Publishing and the regression instrument

• Results live in docs/reference/benchmarks.md in the tuning-and-scaling house style: manifest, arm-by-suite table, the S3 trap-class breakdown, lifecycle funnel, and honest caveats (model dependence, seed data scale, judge agreement rate). Raw results JSON committed under bench/results/<version>/ so history accumulates.
• Regression mode: bench-run --baseline <results.json> compares against the last published run and exits nonzero on regression beyond a per-suite threshold (accuracy points and efficiency deltas). A workflow_dispatch GitHub workflow runs the smoke profile (model API key via secret); full runs are release-time, like verify-release. This is the "prove future enhancements" contract: a PR claiming to improve search or enrichment cites its arm-delta.
• Publishable framing: the headline is always arm-vs-arm on a pinned model (the platform's effect), never model-vs-model. Model identity is disclosed but not the subject.

Explicitly out of scope

• Load/throughput (tracked in test(load): load-test harness with published results #921; that suite answers "how much", this one answers "how well").
• Model leaderboards.
• Live third-party data sources; the environment is fully seeded and airgapped except the LLM API.

Phasing

1. Design ratification + pilot: task schema, generator with 2 trap classes, 10 tasks across S1/S3, arms A0/A2, k=3, manual report. Exit: the pilot either shows a measurable A2-A0 gap or teaches us why not — both outcomes are the point.
2. Full S1-S4 with all four arms, deterministic graders, judge calibration subset, report generator.
3. S5 lifecycle protocols with API-verified state transitions.
4. Publishing + regression gate: docs page, results history, baseline-comparison mode, smoke workflow.

Acceptance criteria

• make bench-up && make bench-run && make bench-report produces the arm-comparison report from a clean checkout with only an LLM API key supplied; smoke profile completes in under 30 minutes.
• Reproducibility: two runs with identical manifest differ only within the reported CIs.
• Deterministic graders cover S1/S2 and the numeric half of S3 with zero LLM-judge involvement; judge rubric, version, and calibration agreement rate are committed alongside any judged scores.
• All efficiency metrics derive from the platform audit API, and a run fails loudly if audit rows are missing for its sessions.
• Published docs/reference/benchmarks.md with the first full-run numbers and bench/results/ history; docs/llms.txt pair updated.
• Regression mode demonstrably fails when replayed against a deliberately degraded config (e.g. enrichment disabled in an A2-labeled run).
• make verify green and unchanged in runtime (bench is not part of verify, same guard-comment convention as mutation testing).

Definition of done

Phase 1 lands before any further phase is scoped in detail; each phase branch-first, reviewed, make verify green before its PR.

[cjimti]

Phased into a ticket series per the definition of done (phase 1 lands before later phases are scoped in detail):

• bench(phase 1): pilot — A0 vs A2 on S1/S3 with deterministic seed generator and audit-derived metrics #942 phase 1: pilot — A0 vs A2 on S1/S3, deterministic seed generator, audit-derived metrics
• bench(phase 2): full S1-S4 across all four arms, deterministic graders, judge calibration, report generator #943 phase 2: full S1-S4, all four arms, deterministic graders, judge calibration, report generator
• bench(phase 3): S5 memory-insight-knowledge lifecycle protocols with API-verified state #944 phase 3: S5 lifecycle protocols, API-verified state
• bench(phase 4): published benchmarks page, results history, regression baseline mode, smoke workflow #945 phase 4: published benchmarks page, results history, regression baseline mode, smoke workflow (closes this epic)

Phase 1 implementation is underway on branch feat/930-bench-phase1-pilot.

[cjimti]

Added to the series: #949 — a claude-cli adapter so subscription users can run real benchmark episodes via claude -p with no metered API key (raw anthropic adapter stays canonical for published/regression numbers).

[cjimti]

Phase 1 pilot results (#942 exit criterion): the A2-A0 gap is real and large.

Manifest: model claude-sonnet-5 (pinned, disclosed; the model is not the subject), k=3, dataset seed 930, task set f36a24207790, platform v1.102.0-5-g1984dd2e, DataHub v1.6.0 quickstart on OpenSearch, 60 episodes total, zero harness failures, all efficiency metrics from the platform audit API (delivery: sync).

metric	a0 (raw toolkit tools)	a2 (semantic-first platform)
S1 discovery accuracy / pass^3	100% / 100%	100% / 100%
S3 knowledge traps accuracy / pass^3	60% / 60%	100% / 100%
S3 median / p90 tool calls	16 / 30 (budget exhausted)	8 / 13
S3 median wall clock	67s	25s
arm output tokens	88k	33k

The distinctive result is the two pure policy-revenue traps (net-revenue definition living only in the knowledge layer): 0/6 on a0 with the full 30-call budget exhausted on every attempt, 6/6 on a2 with pass^3. This is the BIRD external-knowledge gap reproduced with automatic retrieval (enrichment + knowledge pages) in place of hand-curated evidence. Enrichment delivered ~7.3k tokens of context across the a2 S3 suite.

Honest caveats for the record:

• On easy discovery (S1) a capable model needs no help: both arms 100%, and a2 pays a small friction cost (median 8 calls vs 5; most a2 S1 tool errors are SEARCH_REQUIRED gate refusals while the model adopts the workflow). The gate's value shows on knowledge-dependent work, not trivial lookups.
• Two of five a0 S3 tasks were solved without the knowledge layer (the model inferred cents from magnitudes and computed net revenue spontaneously on one ranking task); trap classes vary in strength. Phase 2's larger suite should grade caveat rubric items via the judge to separate "right number" from "right number for the right reason".
• Single machine, single model, pilot-scale task set; no CIs yet (phase 2).
• Total model spend for the pilot including calibration: about $7.40.

Runbook notes from standing a2 up: modern datahub CLIs dropped put --file, so the seed emitter now writes MCP-format aspects ({"json": ...}) and make bench-seed-datahub uses datahub ingest with a generated recipe; the suite report now surfaces enrichment_tokens_full (dedup is zero in full mode). Fix branch: fix/942-datahub-ingest-format.