feat: real cross-backend integration tests (fresh + resume) + pi harness - #165
Merged
Conversation
Fake fixtures can't prove a backend actually works or that history survives a
cross-backend hop — that's the gap that let the codex sandbox bug ship. Add
opt-in integration tests that drive the REAL backends.
backends.integration.test.ts (CODEOID_INTEGRATION=1, backends from
CODEOID_INTEGRATION_BACKENDS, default "claude,codex") — through the real
SessionManager verbs:
- FRESH: each enabled backend takes a brand-new turn and replies.
- RESUME: a conversation started on backend S is forked onto a DIFFERENT
backend T; T must answer a question about a fact only established during
S's turn (exercises the seedFromHistory path fork/switch use).
Verified locally: claude + codex fresh both reply; claude→codex AND
codex→claude resume both carry the fact (5 pass, ~43s).
provider-pi.integration.test.ts (CODEOID_PI_INTEGRATION=1) — the pi
counterpart to provider-codex.integration.test.ts: P1 a fresh text turn runs
to completion, P2 pi runs a shell tool end-to-end. Skips unless pi resolves
and is opted in; pi additionally needs a provider key/login to run a turn.
Both files skip cleanly with no opt-in (9 skip, 0 fail in the offline suite:
1347 pass, tsc + biome clean).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
🤖 Gemini code reviewIntroduces robust cross-backend and Pi provider integration tests that run against real binaries and SDKs when opted in via environment variables. The tests cover fresh turns and conversational resume/handoff between different backends. The implementation is clean and carefully gated to ensure CI remains hermetic. Findings: 🔴 0 · 🟠 0 · 🟡 0 · 🟢 1 Tokens spent · ⬆️ Input: 5,259 · ⬇️ Output: 156 · Σ Total: 10,102 |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #165 +/- ##
=======================================
Coverage 84.51% 84.51%
=======================================
Files 104 104
Lines 18319 18319
=======================================
Hits 15482 15482
Misses 2837 2837
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
…ady awaits idle Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follows the codex integration work. Fake fixtures can't prove a backend actually works, or that a conversation survives a hop to a different backend — that's the exact blind spot that let the codex sandbox bug ship green. This adds opt-in integration tests that drive the real backends.
backends.integration.test.ts— the headline promise, liveCODEOID_INTEGRATION=1, backends fromCODEOID_INTEGRATION_BACKENDS(defaultclaude,codex), through the realSessionManagerverbs:session.fork-ed onto a different backend T; T must answer a question about a fact only established during S's turn. This is codeoid's "branch claude, continue on codex" promise, and it exercises the exactseedFromHistorypath fork/switch rely on.Verified locally against claude (Agent SDK) + codex (@openai/codex@0.144.1):
Both resume directions carry the fact — so history flows across backends regardless of which one produced it.
provider-pi.integration.test.ts— the pi counterpartCODEOID_PI_INTEGRATION=1. The pi analogue ofprovider-codex.integration.test.ts, driving the realPiProvideragainst the actual pi binary (bundled@earendil-works/pi-coding-agent, system pi, orproviders.pi.command):turn_donewith a reply, no error.Honest caveat: pi needs a provider key/login to take a turn, and I have no pi auth on any box I can reach, so I could not execute P1/P2 here — they're written + gated and run wherever pi is authed. pi's turn mechanics remain covered offline by the extensive
provider-pi.test.ts(fake-pi), and pi is also picked up automatically by thebackendsmatrix above once you add it toCODEOID_INTEGRATION_BACKENDSon a pi-authed host.Safety
Both files skip cleanly with no opt-in — CI stays hermetic (no network, no keys). Offline suite:
1347 pass / 9 skip / 0 fail,tsc+biomeclean.🤖 Generated with Claude Code