v0.1.0 — two scenarios, four invariants, one SDK measured
Integrity scenarios for agent runtimes — deterministic fault injection against real SDKs, distilled from dated incidents in our own multi-machine fleet. First tagged release.
The class of failure it looks for
Most agent-runtime failures we have lived through were silent: every component reported success while an invariant was already dead. A benchmark that measures task success cannot see that class, so this one injects the fault and checks the invariant directly.
What is in v0.1.0
Two scenarios, four invariants (bench/s2_replay.py, bench/s3_concurrent_memory.py):
| ID | Invariant | Fault injected |
|---|---|---|
| ARIB-CONC-001 | N concurrent appends → N visible, 0 lost, 0 duplicated | 8 writers × 25 appends |
| ARIB-CONC-002 | close() is idempotent under concurrency |
2 concurrent close(), 20 trials |
| ARIB-CONC-003 | write-after-close is refused loudly, never silently committed or dropped | add_items() after close() |
| ARIB-REPLAY-001 | a redelivered batch is visible exactly once | same batch delivered twice |
Measured: openai-agents 0.19.2, four session backends documented as drop-in replacements for each other — 16 findings: 8 held, 6 violated, 2 not applicable. Raw dated JSON in results/, and the same run on CI Ubuntu produced an identical report — all 16 verdicts and every violation count, including the 20/20 close-race. Diff the two files yourself.
Self-test: seven mutants. The rule this repo lives by is that a check without a mutant that breaks it measures nothing. CI runs it on 3.10/3.12/3.13 on every push and weekly.
The verdict vocabulary, and why it has four values
A violated invariant is not automatically a bug: a store may legitimately document at-least-once semantics and push dedup to the caller. The benchmark's job is to make the actual semantics measurable and explicit — because agent code in the wild routinely assumes exactly-once and assumes closed means closed.
not_applicable exists because of an external review that killed one of our own findings: the first version of the report showed a red cell against SQLAlchemySession for a promise that backend never made (it has no close(); our harness supplied one). An abstention is now a first-class verdict, it does not enter the exit code, and a report made entirely of abstentions fails with "nothing was measured". An adapter's claim about itself is checked against the runtime object, so a false claim is a harness error rather than a free pass.
Honest limits
- Not covered: Windows, free-threaded builds, uvloop.
RedisSession/MongoDBSession/DaprSession/EncryptedSessionneed a live service and are deliberately not stubbed — a mock would produce a verdict about the mock. - Six more scenarios from the same incident log are designed and not built. A scenario ships only when it can be paired with a real incident and a real runtime.
- The two newest adapters are reproducible-and-machine-reviewed; no second pair of human eyes yet. The README says so per claim.
What's next
A second runtime family, then the six remaining scenarios. Every noticeable change — and any new result about someone else's library is one — ships as its own release, so this feed records exactly how much has actually been measured, and when.
Full Changelog: https://github.com/Palo-Alto-AI-Research-Lab/agent-runtime-integrity-bench/commits/v0.1.0