Stale-context bench

A reproducible harness for one failure mode of tool-using coding agents:

A confident, always-loaded context summary that already answers the question suppresses the verification search. When a project decision has quietly changed, the agent emits the stale value having read zero source files.

Results and write-up: https://research.meetless.ai/stale-context/

Everything here is fabricated fixture data — a fictional product called "Marlin". No real company information, no customer data.

---

What you can reproduce, and what you can't

Be clear on this before you start, because the distinction is the point.

claim	reproducible here?
The failure itself — floor arm scores 0/6 having read no files	Yes. A folder of notes, a summary file, and a plain claude -p run.
The mitigation comparison — verify, Memory Bank, ADR, spec, RAG, TierMem, CUPMem	Yes. Same shape: fixtures plus a model.
The graded 20-fact fixture (five relationship traps)	Yes.
The 272-scenario external replication ported from STALE	Yes, with the caveat below on the source dataset.
The mla push rows	No — and not because of a missing file. That arm is a stand-in: a fixed block of current facts placed into the prompt, representing what a proactive-injection tool would supply. It does not run any product. It measures the mechanism — current facts present in trusted context — and should be read that way and no further.

If you want to evaluate a real memory/injection product, wire it as a new arm and compare it against mf-floor. Do not treat the mf-mla-push numbers as a benchmark of a shipping integration.

Requirements

• Node 20+
• A working claude CLI on PATH (for the Anthropic arms), and/or OPENAI_API_KEY / GEMINI_API_KEY for the cross-vendor arms.
• Model calls cost money. A full matrix is many hundreds of runs. Start with one arm.

Quick start

# 1. the failure, on one model, one arm
node run.mjs --arm mf-floor --trials 1 --model claude-haiku-4-5

# 2. the control that proves the summary is the cause (same data on disk, no asserted facts)
node run.mjs --arm mf-nosummary --trials 1 --model claude-haiku-4-5

# 3. the graded 20-fact fixture (scored /20, five trap tiers)
node gen-hard.mjs --arm floor --out arms/hard-floor
node run.mjs --arm hard-floor --trials 1 --model claude-haiku-4-5

# 4. cross-vendor
OPENAI_API_KEY=... node openai-run.mjs --arm mf-floor --model gpt-5.4 --trials 1

node matrix.mjs --trials 3 runs the whole grid. Read metrics.mjs for cost/latency aggregation.

Layout

path	what it is
facts.mjs	the 6-fact model + keyword oracle (measures verification suppression)
facts-hard.mjs	the 20-fact graded model, five trap tiers (measures relationship resolution)
gen-multifact.mjs, gen-hard.mjs	deterministic fixture generators
arms/	generated fixtures, checked in so a run is inspectable without regenerating
run.mjs	isolated arm runner (claude -p in a clean config dir)
gemini-run.mjs, openai-run.mjs	cross-vendor REST harnesses with equivalent tools
oracle-*.mjs	deterministic graders
isolate.mjs, validate-isolation.mjs	the clean-room mechanism, and a validator that proves it
cupmem-adjudicate.mjs	builds the CUPMem-style adjudicated store (write-side stage)
stale-port/	the STALE port: extraction, gate, runner
agentbench-port/	path-staleness check over real repositories' context files
COMPETITORS.md	the real-world methods each arm represents, with sources

Two fixtures, two different questions

They are not interchangeable and their numbers must not be mixed.

• facts.mjs — a fluid deliverable over 6 superseded facts. Measures whether the agent looks at all. Saturates above the floor: every method that looks scores 6/6.
• facts-hard.mjs — 20 facts, 100 notes, five traps (recency, rejected proposal, scoped exception, multi-hop, near-duplicate draft). Measures whether it can resolve what it finds. Does not saturate — but does not reproduce the floor on a frontier model, which bulk-reads the corpus.

Isolation

Arms run as subprocesses with a clean CLAUDE_CONFIG_DIR (auth only, no CLAUDE.md, hooks stripped) and a clean HOME, so the operator's own global instructions cannot leak into a run. validate-isolation.mjs checks this three ways, including a two-sided canary.

Set your own canaries. The published negative sentinels are placeholders. Override with MEETLESS_LEAK_SENTINELS="a,b,c" using strings that exist in your environment and nowhere else — a sentinel a model could plausibly have seen in training proves nothing.

Data attribution

stale-port/scenarios.json is derived from STALE (arXiv 2605.06527), used under CC BY 4.0. It contains 272 of their 400 conflict scenarios, reduced to (stale value, current value, task) triples and re-cast into an on-disk setting. Their haystack_session field is discarded by design: the port's whole purpose is to move the evidence out of the context window and onto disk. Original dataset: https://huggingface.co/datasets/STALEproj/STALE.

Method arms reference published work — TierMem (arXiv 2602.17913), CUPMem (arXiv 2605.06527), AGENTS.md evaluation (arXiv 2602.11988). Those arms are our instantiation of each published policy, not the authors' code, and any result here is a statement about the policy under these conditions, not about their system on their benchmark.

Known limitations

• Small N. Trial counts are low; treat rankings as directional, not significant.
• The fixtures were authored with a frontier model, which is also under test — a real confound. The STALE port exists partly to escape it.
• The keyword oracles are deterministic but blunt: an answer that never names a value scores absent, which understates methods that answer indirectly.

License

Code: MIT (LICENSE). stale-port/scenarios.json: CC BY 4.0, per the attribution above.