Skip to content

feat: real cross-backend integration tests (fresh + resume) + pi harness - #165

Merged
saucam merged 2 commits into
mainfrom
test/backend-integration-fresh-and-resume
Jul 12, 2026
Merged

feat: real cross-backend integration tests (fresh + resume) + pi harness#165
saucam merged 2 commits into
mainfrom
test/backend-integration-fresh-and-resume

Conversation

@saucam

@saucam saucam commented Jul 12, 2026

Copy link
Copy Markdown
Collaborator

Follows the codex integration work. Fake fixtures can't prove a backend actually works, or that a conversation survives a hop to a different backend — that's the exact blind spot that let the codex sandbox bug ship green. This adds opt-in integration tests that drive the real backends.

backends.integration.test.ts — the headline promise, live

CODEOID_INTEGRATION=1, backends from CODEOID_INTEGRATION_BACKENDS (default claude,codex), through the real SessionManager verbs:

  • FRESH — each enabled backend takes a brand-new turn and replies.
  • RESUME — a conversation started on backend S is session.fork-ed onto a different backend T; T must answer a question about a fact only established during S's turn. This is codeoid's "branch claude, continue on codex" promise, and it exercises the exact seedFromHistory path fork/switch rely on.

Verified locally against claude (Agent SDK) + codex (@openai/codex@0.144.1):

FRESH: claude ✓   FRESH: codex ✓
RESUME: claude → codex ✓   RESUME: codex → claude ✓     (5 pass, ~43s)

Both resume directions carry the fact — so history flows across backends regardless of which one produced it.

provider-pi.integration.test.ts — the pi counterpart

CODEOID_PI_INTEGRATION=1. The pi analogue of provider-codex.integration.test.ts, driving the real PiProvider against the actual pi binary (bundled @earendil-works/pi-coding-agent, system pi, or providers.pi.command):

  • P1 — a fresh text turn runs to turn_done with a reply, no error.
  • P2 — pi runs a shell tool end-to-end and its output reaches codeoid.

Honest caveat: pi needs a provider key/login to take a turn, and I have no pi auth on any box I can reach, so I could not execute P1/P2 here — they're written + gated and run wherever pi is authed. pi's turn mechanics remain covered offline by the extensive provider-pi.test.ts (fake-pi), and pi is also picked up automatically by the backends matrix above once you add it to CODEOID_INTEGRATION_BACKENDS on a pi-authed host.

Safety

Both files skip cleanly with no opt-in — CI stays hermetic (no network, no keys). Offline suite: 1347 pass / 9 skip / 0 fail, tsc + biome clean.

🤖 Generated with Claude Code

Fake fixtures can't prove a backend actually works or that history survives a
cross-backend hop — that's the gap that let the codex sandbox bug ship. Add
opt-in integration tests that drive the REAL backends.

backends.integration.test.ts (CODEOID_INTEGRATION=1, backends from
CODEOID_INTEGRATION_BACKENDS, default "claude,codex") — through the real
SessionManager verbs:
  - FRESH: each enabled backend takes a brand-new turn and replies.
  - RESUME: a conversation started on backend S is forked onto a DIFFERENT
    backend T; T must answer a question about a fact only established during
    S's turn (exercises the seedFromHistory path fork/switch use).
  Verified locally: claude + codex fresh both reply; claude→codex AND
  codex→claude resume both carry the fact (5 pass, ~43s).

provider-pi.integration.test.ts (CODEOID_PI_INTEGRATION=1) — the pi
counterpart to provider-codex.integration.test.ts: P1 a fresh text turn runs
to completion, P2 pi runs a shell tool end-to-end. Skips unless pi resolves
and is opted in; pi additionally needs a provider key/login to run a turn.

Both files skip cleanly with no opt-in (9 skip, 0 fail in the offline suite:
1347 pass, tsc + biome clean).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Comment thread src/tests/backends.integration.test.ts
@github-actions

Copy link
Copy Markdown

🤖 Gemini code review

Introduces robust cross-backend and Pi provider integration tests that run against real binaries and SDKs when opted in via environment variables. The tests cover fresh turns and conversational resume/handoff between different backends. The implementation is clean and carefully gated to ensure CI remains hermetic.

Findings: 🔴 0 · 🟠 0 · 🟡 0 · 🟢 1


Tokens spent · ⬆️ Input: 5,259 · ⬇️ Output: 156 · Σ Total: 10,102
Total may be higher due to thinking token counts.

@codecov

codecov Bot commented Jul 12, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.51%. Comparing base (ccf28d3) to head (1a39384).
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #165   +/-   ##
=======================================
  Coverage   84.51%   84.51%           
=======================================
  Files         104      104           
  Lines       18319    18319           
=======================================
  Hits        15482    15482           
  Misses       2837     2837           
Flag Coverage Δ
daemon 84.51% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…ady awaits idle

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@saucam
saucam merged commit c6fea52 into main Jul 12, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant