Skip to content

test: live e2e suite against the real OpenRouter API + CI job - #3

Merged
LukasParke merged 2 commits into
mainfrom
ci/live-e2e
Jul 29, 2026
Merged

test: live e2e suite against the real OpenRouter API + CI job#3
LukasParke merged 2 commits into
mainfrom
ci/live-e2e

Conversation

@LukasParke

Copy link
Copy Markdown
Contributor

Adds real end-to-end CI testing: five live tests against the OpenRouter API mirroring upstream's packages/agent/tests/e2e coverage — text/stream agreement, a real tool round, approval pause/resume across two CallModel invocations, lifecycle hooks on live traffic (asserting SessionEnd usage totals), and conversation-state serialize→deserialize round-trip resuming a live paused run.

The live suite immediately caught two real port bugs the mocked unit tests couldn't see, fixed in this PR:

  1. Every live tool round panicked. go-sdk v0.5.4's InputsUnion1 unmarshaller hits reflect: call of reflect.Value.Elem on slice Value on assistant messages with array content — the exact shape every live response echoes back on follow-up turns. responseInputItemsWithError now flattens output_text array content to the equivalent string form before decoding (flattenMessageContentForUnion, documented as removable once the pinned SDK handles array content).
  2. Every second turn 400'd. previous_response_id was sent on the wire for follow-up/resume requests; the live API rejects it on stateless requests ("expected null, received string"). Upstream tracks previousResponseId in state only and always sends full history — the port now matches. One unit test pinned the old wire behavior and was updated to pin the new one.

Test-design notes: approval tests pin the first turn with ToolChoice=required so model nondeterminism can't skip the pause being asserted. Live tests are double-gated (OPENROUTER_API_KEY + OPENROUTER_AGENT_E2E=1) so a plain go test ./... stays free and hermetic even with a key exported. Model overridable via OPENROUTER_E2E_MODEL.

CI: new e2e job — warns and exits 0 when the secret is missing (fork PRs), same pattern as upstream typescript-agent. Costs a few cents per run.

Verified live: repeated full-suite runs green (one unreproduced intermittent failure of the serialization test in ~15 runs — likely model nondeterminism; left as-is to gather CI signal). Unit suite, gofmt, go vet green.

🤖 Generated with Claude Code

Replaces the single smoke test with five e2e tests mirroring upstream's
packages/agent/tests/e2e coverage: text/stream agreement, a real tool
round, approval pause/resume across two CallModel invocations, lifecycle
hooks firing on live traffic (with SessionEnd usage totals), and
conversation-state serialize/deserialize round-trip resuming a live
paused run. Gated on OPENROUTER_API_KEY + OPENROUTER_AGENT_E2E=1 so a
plain `go test ./...` stays free and hermetic; model overridable via
OPENROUTER_E2E_MODEL.

Running these live immediately caught two real port bugs the mocked
unit suite could not see, both fixed here:

- go-sdk v0.5.4's InputsUnion1 unmarshaller panics (reflect on slice
  value) on assistant messages with array content — the exact shape
  every live response echoes back on follow-up turns. Every live tool
  round crashed. responseInputItemsWithError now flattens output_text
  array content to the equivalent string form before decoding
  (flattenMessageContentForUnion); remove once the pinned SDK handles
  array content.

- previous_response_id was sent on the wire for follow-up/resume
  requests; the live API rejects it on stateless requests ("expected
  null, received string"), 400-ing every second turn. Upstream tracks
  previousResponseId in state only and always sends full history — the
  port now does the same. (One unit test pinned the old wire behavior
  and was updated to pin the new one.)

The approval tests pin the first turn with ToolChoice=required so model
nondeterminism can't skip the pause being asserted.

CI: new e2e job — warns and exits 0 when OPENROUTER_API_KEY is missing
(forks), same pattern as upstream typescript-agent.

Verified live: full suite green across repeated runs (one unreproduced
intermittent failure of the serialization test in ~15 runs — likely
model nondeterminism; left as-is to gather CI signal). Unit suite,
gofmt, go vet green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@perry-the-pr-reviewer perry-the-pr-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perry's Review

Verdict: 💬 Comments / questions
Risk: 🟡 Medium

Details

This PR adds a live e2e test suite and fixes two real bugs the mocked unit tests couldn't catch. Both fixes are correct, match upstream behavior, and are well-tested. CI is green (check, e2e, verify-port all pass). Would approve if the maintainer app were installed on this org.

Bug fix 1: previous_response_id removed from the wire

The port was sending previous_response_id on follow-up and resume requests, which the live Responses API rejects on stateless requests ("expected null, received string"). The fix removes all three wire-level assignments (CallModel, the run loop, and prepareResumeRequest) while keeping the field tracked in ConversationState for observability — exactly matching upstream, which stores previousResponseId in state only and never sends it. The unit test was correctly flipped from asserting presence to asserting absence.

Bug fix 2: flattenMessageContentForUnion

go-sdk v0.5.4's InputsUnion1 unmarshaller panics on array-content assistant messages (reflect: call of reflect.Value.Elem on slice Value). The workaround flattens output_text array content to the equivalent string form before decoding, which the SDK accepts. The function is well-scoped: it only transforms message-type items, bails cleanly on non-output_text blocks (returning original bytes rather than silently dropping data), and is documented as removable once the SDK handles array content. The only gap is the lack of a deterministic unit test — the function is currently exercised only by the live e2e suite, which costs API credits and is model-dependent.

E2E tests

The five live tests are well-designed: double-gated (API key plus an explicit env flag), ToolChoice=required to pin tool calls for approval-pause assertions, t.Cleanup for timeout safety, and model overridable via env var. Coverage goes beyond upstream's e2e suite (upstream has no live approval/resume, lifecycle hooks, or serialization round-trip e2e tests).

CI

The new e2e job correctly warns and exits 0 when the secret is missing (fork PRs), matching the upstream typescript-agent pattern. The 15-minute timeout is generous for five short-prompt tests.

Suggestion

Prompt for agents: Add a unit test for flattenMessageContentForUnion with deterministic JSON inputs (array-content message, non-text block bail-out, non-message passthrough) so regressions are caught without spending live API credits.

Comment thread model_result.go
Review suggestion on #3: the union-decode workaround was only exercised
by the credit-gated live e2e suite. These hardcoded-shape tests cover
flatten, all three passthrough/bail paths, and the end-to-end property
the workaround exists for (an array-content assistant message decodes
into InputsUnion1 without panicking), running free on every PR.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@LukasParke
LukasParke merged commit 4368f82 into main Jul 29, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant