v0.2.0 — openai-agents 0.19.4: two findings fixed upstream, replay still stands

What this is. A fault-injection bench for agent session storage: it injects the fault and checks the invariant directly, because the failures worth catching here are the silent ones — every component reports success while the invariant is dead. Two scenarios, four invariants, four openai-agents session backends. No API key, no network, no model call.

What is new in v0.2.0 — the bench watched one of its own findings get fixed, and says so in the same format it used to report it.

• results/2026-08-10-openai-agents-0.19.4.json — 10 held, 4 violated, 2 not applicable (v0.1.0 reported 8 / 6 / 2 against 0.19.2). Both AsyncSQLiteSession close findings — the AttributeError in 20/20 trials, and the write silently committed to a resurrected connection — now hold.
• results/2026-08-10-openai-agents-0.19.2-control.json — the pinned old version, re-run on the same machine on the same day, still producing the original six violations. Without this control, "the SDK changed" would have been a guess about the machine. The only difference between the two files is the SDK version.
• Upstream credit is stated, not implied. Reported by @hsusul in openai/openai-agents-python#3983; fixed by #4109 (@chinmayv095, merged 2026-08-02); shipped in v0.19.3. We reproduced it deterministically and added the second failure mode to that thread. We did not write the fix.
• ARIB-REPLAY-001 is unchanged on all four backends. A redelivered batch is still visible twice, and no backend claims otherwise — add_items() carries no idempotency key. A finding that ages out and a finding that persists look identical in a blog post; they look different in a dated report.
• An animated demo at the top of the README, drawn from the verbatim stderr of a real run by .github/assets/make_demo.py — no hand-typed output — and every self-link repointed at the canonical account, because the old organisation URL under "Built by" had become a 404.

What is known to be missing. No open issues, which is not the same as no gaps, so they are named here: the 0.19.4 re-run is macOS only — the published cross-platform pair is still 0.19.2, so the Linux CI leg of the new numbers does not exist yet. The two newest adapters (AdvancedSQLiteSession, SQLAlchemySession) remain reproducible-and-machine-reviewed, with no second pair of human eyes; the README says so per claim. Windows, free-threaded builds and uvloop are unmeasured. RedisSession / MongoDBSession / DaprSession / EncryptedSession need a live service and are deliberately not stubbed — a mock would only produce a verdict about the mock. The selftest's mutants cover wrong verdicts, not hangs.

Next. A 0.19.4 run on CI Ubuntu so the newest numbers get the same two-platform backing as the old ones; a second runtime family, because a benchmark pointed at exactly one SDK has not yet shown that it measures the SDK rather than itself; and six more scenarios from the same incident log, each shipping only when it can be paired with a real incident and a real runtime.

v0.1.0 — two scenarios, four invariants, one SDK measured

Integrity scenarios for agent runtimes — deterministic fault injection against real SDKs, distilled from dated incidents in our own multi-machine fleet. First tagged release.

The class of failure it looks for

Most agent-runtime failures we have lived through were silent: every component reported success while an invariant was already dead. A benchmark that measures task success cannot see that class, so this one injects the fault and checks the invariant directly.

What is in v0.1.0

Two scenarios, four invariants (bench/s2_replay.py, bench/s3_concurrent_memory.py):

ID	Invariant	Fault injected
ARIB-CONC-001	N concurrent appends → N visible, 0 lost, 0 duplicated	8 writers × 25 appends
ARIB-CONC-002	close() is idempotent under concurrency	2 concurrent close(), 20 trials
ARIB-CONC-003	write-after-close is refused loudly, never silently committed or dropped	add_items() after close()
ARIB-REPLAY-001	a redelivered batch is visible exactly once	same batch delivered twice

Measured: openai-agents 0.19.2, four session backends documented as drop-in replacements for each other — 16 findings: 8 held, 6 violated, 2 not applicable. Raw dated JSON in results/, and the same run on CI Ubuntu produced an identical report — all 16 verdicts and every violation count, including the 20/20 close-race. Diff the two files yourself.

Self-test: seven mutants. The rule this repo lives by is that a check without a mutant that breaks it measures nothing. CI runs it on 3.10/3.12/3.13 on every push and weekly.

The verdict vocabulary, and why it has four values

A violated invariant is not automatically a bug: a store may legitimately document at-least-once semantics and push dedup to the caller. The benchmark's job is to make the actual semantics measurable and explicit — because agent code in the wild routinely assumes exactly-once and assumes closed means closed.

not_applicable exists because of an external review that killed one of our own findings: the first version of the report showed a red cell against SQLAlchemySession for a promise that backend never made (it has no close(); our harness supplied one). An abstention is now a first-class verdict, it does not enter the exit code, and a report made entirely of abstentions fails with "nothing was measured". An adapter's claim about itself is checked against the runtime object, so a false claim is a harness error rather than a free pass.

Honest limits

• Not covered: Windows, free-threaded builds, uvloop. RedisSession / MongoDBSession / DaprSession / EncryptedSession need a live service and are deliberately not stubbed — a mock would produce a verdict about the mock.
• Six more scenarios from the same incident log are designed and not built. A scenario ships only when it can be paired with a real incident and a real runtime.
• The two newest adapters are reproducible-and-machine-reviewed; no second pair of human eyes yet. The README says so per claim.

What's next

A second runtime family, then the six remaining scenarios. Every noticeable change — and any new result about someone else's library is one — ships as its own release, so this feed records exactly how much has actually been measured, and when.

Full Changelog: https://github.com/Palo-Alto-AI-Research-Lab/agent-runtime-integrity-bench/commits/v0.1.0