What this is. A fault-injection bench for agent session storage: it injects the fault and checks the invariant directly, because the failures worth catching here are the silent ones — every component reports success while the invariant is dead. Two scenarios, four invariants, four openai-agents session backends. No API key, no network, no model call.
What is new in v0.2.0 — the bench watched one of its own findings get fixed, and says so in the same format it used to report it.
results/2026-08-10-openai-agents-0.19.4.json— 10 held, 4 violated, 2 not applicable (v0.1.0 reported 8 / 6 / 2 against 0.19.2). BothAsyncSQLiteSessionclose findings — theAttributeErrorin 20/20 trials, and the write silently committed to a resurrected connection — now hold.results/2026-08-10-openai-agents-0.19.2-control.json— the pinned old version, re-run on the same machine on the same day, still producing the original six violations. Without this control, "the SDK changed" would have been a guess about the machine. The only difference between the two files is the SDK version.- Upstream credit is stated, not implied. Reported by @hsusul in openai/openai-agents-python#3983; fixed by #4109 (@chinmayv095, merged 2026-08-02); shipped in v0.19.3. We reproduced it deterministically and added the second failure mode to that thread. We did not write the fix.
- ARIB-REPLAY-001 is unchanged on all four backends. A redelivered batch is still visible twice, and no backend claims otherwise —
add_items()carries no idempotency key. A finding that ages out and a finding that persists look identical in a blog post; they look different in a dated report. - An animated demo at the top of the README, drawn from the verbatim stderr of a real run by
.github/assets/make_demo.py— no hand-typed output — and every self-link repointed at the canonical account, because the old organisation URL under "Built by" had become a 404.
What is known to be missing. No open issues, which is not the same as no gaps, so they are named here: the 0.19.4 re-run is macOS only — the published cross-platform pair is still 0.19.2, so the Linux CI leg of the new numbers does not exist yet. The two newest adapters (AdvancedSQLiteSession, SQLAlchemySession) remain reproducible-and-machine-reviewed, with no second pair of human eyes; the README says so per claim. Windows, free-threaded builds and uvloop are unmeasured. RedisSession / MongoDBSession / DaprSession / EncryptedSession need a live service and are deliberately not stubbed — a mock would only produce a verdict about the mock. The selftest's mutants cover wrong verdicts, not hangs.
Next. A 0.19.4 run on CI Ubuntu so the newest numbers get the same two-platform backing as the old ones; a second runtime family, because a benchmark pointed at exactly one SDK has not yet shown that it measures the SDK rather than itself; and six more scenarios from the same incident log, each shipping only when it can be paired with a real incident and a real runtime.