test: live e2e suite against the real OpenRouter API + CI job - #3
Conversation
Replaces the single smoke test with five e2e tests mirroring upstream's
packages/agent/tests/e2e coverage: text/stream agreement, a real tool
round, approval pause/resume across two CallModel invocations, lifecycle
hooks firing on live traffic (with SessionEnd usage totals), and
conversation-state serialize/deserialize round-trip resuming a live
paused run. Gated on OPENROUTER_API_KEY + OPENROUTER_AGENT_E2E=1 so a
plain `go test ./...` stays free and hermetic; model overridable via
OPENROUTER_E2E_MODEL.
Running these live immediately caught two real port bugs the mocked
unit suite could not see, both fixed here:
- go-sdk v0.5.4's InputsUnion1 unmarshaller panics (reflect on slice
value) on assistant messages with array content — the exact shape
every live response echoes back on follow-up turns. Every live tool
round crashed. responseInputItemsWithError now flattens output_text
array content to the equivalent string form before decoding
(flattenMessageContentForUnion); remove once the pinned SDK handles
array content.
- previous_response_id was sent on the wire for follow-up/resume
requests; the live API rejects it on stateless requests ("expected
null, received string"), 400-ing every second turn. Upstream tracks
previousResponseId in state only and always sends full history — the
port now does the same. (One unit test pinned the old wire behavior
and was updated to pin the new one.)
The approval tests pin the first turn with ToolChoice=required so model
nondeterminism can't skip the pause being asserted.
CI: new e2e job — warns and exits 0 when OPENROUTER_API_KEY is missing
(forks), same pattern as upstream typescript-agent.
Verified live: full suite green across repeated runs (one unreproduced
intermittent failure of the serialization test in ~15 runs — likely
model nondeterminism; left as-is to gather CI signal). Unit suite,
gofmt, go vet green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Perry's Review
Verdict: 💬 Comments / questions
Risk: 🟡 Medium
Details
This PR adds a live e2e test suite and fixes two real bugs the mocked unit tests couldn't catch. Both fixes are correct, match upstream behavior, and are well-tested. CI is green (check, e2e, verify-port all pass). Would approve if the maintainer app were installed on this org.
Bug fix 1: previous_response_id removed from the wire
The port was sending previous_response_id on follow-up and resume requests, which the live Responses API rejects on stateless requests ("expected null, received string"). The fix removes all three wire-level assignments (CallModel, the run loop, and prepareResumeRequest) while keeping the field tracked in ConversationState for observability — exactly matching upstream, which stores previousResponseId in state only and never sends it. The unit test was correctly flipped from asserting presence to asserting absence.
Bug fix 2: flattenMessageContentForUnion
go-sdk v0.5.4's InputsUnion1 unmarshaller panics on array-content assistant messages (reflect: call of reflect.Value.Elem on slice Value). The workaround flattens output_text array content to the equivalent string form before decoding, which the SDK accepts. The function is well-scoped: it only transforms message-type items, bails cleanly on non-output_text blocks (returning original bytes rather than silently dropping data), and is documented as removable once the SDK handles array content. The only gap is the lack of a deterministic unit test — the function is currently exercised only by the live e2e suite, which costs API credits and is model-dependent.
E2E tests
The five live tests are well-designed: double-gated (API key plus an explicit env flag), ToolChoice=required to pin tool calls for approval-pause assertions, t.Cleanup for timeout safety, and model overridable via env var. Coverage goes beyond upstream's e2e suite (upstream has no live approval/resume, lifecycle hooks, or serialization round-trip e2e tests).
CI
The new e2e job correctly warns and exits 0 when the secret is missing (fork PRs), matching the upstream typescript-agent pattern. The 15-minute timeout is generous for five short-prompt tests.
Suggestion
▶ Prompt for agents: Add a unit test for flattenMessageContentForUnion with deterministic JSON inputs (array-content message, non-text block bail-out, non-message passthrough) so regressions are caught without spending live API credits.
Review suggestion on #3: the union-decode workaround was only exercised by the credit-gated live e2e suite. These hardcoded-shape tests cover flatten, all three passthrough/bail paths, and the end-to-end property the workaround exists for (an array-content assistant message decodes into InputsUnion1 without panicking), running free on every PR. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds real end-to-end CI testing: five live tests against the OpenRouter API mirroring upstream's
packages/agent/tests/e2ecoverage — text/stream agreement, a real tool round, approval pause/resume across twoCallModelinvocations, lifecycle hooks on live traffic (asserting SessionEnd usage totals), and conversation-state serialize→deserialize round-trip resuming a live paused run.The live suite immediately caught two real port bugs the mocked unit tests couldn't see, fixed in this PR:
InputsUnion1unmarshaller hitsreflect: call of reflect.Value.Elem on slice Valueon assistant messages with array content — the exact shape every live response echoes back on follow-up turns.responseInputItemsWithErrornow flattensoutput_textarray content to the equivalent string form before decoding (flattenMessageContentForUnion, documented as removable once the pinned SDK handles array content).previous_response_idwas sent on the wire for follow-up/resume requests; the live API rejects it on stateless requests ("expected null, received string"). Upstream trackspreviousResponseIdin state only and always sends full history — the port now matches. One unit test pinned the old wire behavior and was updated to pin the new one.Test-design notes: approval tests pin the first turn with
ToolChoice=requiredso model nondeterminism can't skip the pause being asserted. Live tests are double-gated (OPENROUTER_API_KEY+OPENROUTER_AGENT_E2E=1) so a plaingo test ./...stays free and hermetic even with a key exported. Model overridable viaOPENROUTER_E2E_MODEL.CI: new
e2ejob — warns and exits 0 when the secret is missing (fork PRs), same pattern as upstream typescript-agent. Costs a few cents per run.Verified live: repeated full-suite runs green (one unreproduced intermittent failure of the serialization test in ~15 runs — likely model nondeterminism; left as-is to gather CI signal). Unit suite, gofmt, go vet green.
🤖 Generated with Claude Code