test: live e2e suite against the real OpenRouter API + CI job - #21
Conversation
Replaces the single smoke test with five e2e tests mirroring upstream's packages/agent/tests/e2e coverage: text/stream agreement, a real tool round, approval pause/resume across two call_model invocations, lifecycle hooks firing on live traffic (with SessionEnd usage totals), and conversation-state serialize/deserialize round-trip resuming a live paused run. Skipped without OPENROUTER_API_KEY; model pinned to a small default, overridable via OPENROUTER_E2E_MODEL. Running these live immediately caught two real port bugs the mocked unit suite could not see, both fixed here: - model_result._send now normalizes input items at the transport boundary: response items echoed back from a live turn are SDK pydantic models (the request validator wants plain dicts), and internal items use upstream's camelCase callId while the generated SDK validates snake_case call_id. Every follow-up tool round against the live API failed validation before this; mocked clients accepted anything. (test_tool_terminal_empty_final updated: it pinned the pre-fix wire spelling.) - serialize_conversation_state now dumps SDK pydantic items to plain dicts (json.dumps default=dump); live states holding OutputFunctionCallItem raised TypeError before. The approval tests pin the tool call with tool_choice="required" so model nondeterminism can't skip the pause; the plain tool-loop test deliberately does not (required persists across turns, matching upstream, and would trip the 20-turn safety limit). CI: new e2e job — warns and exits 0 when OPENROUTER_API_KEY is missing (forks), same pattern as upstream typescript-agent. Verified: full e2e suite passed 3x consecutively live; unit suite, ruff, mypy all green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Perry's Review
Verdict: 💬 Comments / questions
Risk: 🟢 Low
Details
This PR adds a live e2e test suite against the real OpenRouter API (mirroring upstream typescript-agent's packages/agent/tests/e2e coverage) and fixes two real bugs the new suite caught:
-
_sendtransport normalization (model_result.py): Response items echoed back from a live turn are SDK pydantic models, and internal items use upstream's camelCasecallIdwhile the Python SDK validatescall_id. The fix normalizes both at the transport boundary only — internal state and stream events keep the upstream shapes. This is the right place for the conversion: it's contained, doesn't leak into state serialization or event handling, and the shallowcallId→call_idconversion is sufficient sincecallIdonly appears at the top level of input items. -
serialize_conversation_statewithdefault=dump(conversation_state.py): States built from live responses hold SDK pydantic items (e.g.OutputFunctionCallItem) thatdataclasses.asdictpasses through untouched, causingTypeErroronjson.dumps. Usingdumpas thedefaultfallback is consistent with how the existingjson_dumpshelper already handles pydantic models, and theexclude_none=Truefrommodel_dumpis safe for the round-trip sincedeserialize_conversation_stateuses.get()withNonedefaults.
Test design is thoughtful: approval tests pin the first turn with tool_choice="required" to avoid model nondeterminism skipping the pause; the plain tool-loop test deliberately doesn't (since required persists and would trip the 20-turn safety limit). The MemoryState helper and _client factory are clean.
CI: The e2e job follows the upstream pattern (warn + exit 0 when OPENROUTER_API_KEY is missing), so fork PRs and local dev aren't blocked. All three CI checks are green (check, e2e, verify-port).
One suggestion (inline): the default=dump serialization fix is only exercised by the e2e test, which doesn't run without an API key (fork PRs, local dev, most CI environments). A unit test that puts a pydantic model into ConversationState.messages and asserts serialize_conversation_state produces valid JSON would guard against regressions in the unit test job.
- Unit tests (103), ruff, and CI all green.
- No security concerns — no hardcoded secrets, API key injected via GitHub secrets.
- Blast radius is minimal:
_sendis only called from_send_and_track, and the normalization doesn't touch internal state.
Review suggestion on #21: the pydantic-item serialization fix was only exercised by the credit-gated e2e suite. This deterministic unit test puts a pydantic model into state.messages and asserts valid JSON out, so the regression is caught by the check job on every PR. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds real end-to-end CI testing: five live tests against the OpenRouter API mirroring upstream's
packages/agent/tests/e2ecoverage — text/stream agreement, a real tool round, approval pause/resume across twocall_modelinvocations, lifecycle hooks on live traffic (asserting SessionEnd usage totals), and conversation-state serialize→deserialize round-trip resuming a live paused run.The live suite immediately caught two real port bugs the mocked unit tests couldn't see, fixed in this PR:
callIdbut the generated SDK validatescall_id, and response items echoed back from a live turn are SDK pydantic models where the validator wants plain dicts._sendnow normalizes both at the transport boundary only — internal state/stream events keep the upstream shape. One unit test pinned the broken wire spelling and was updated.serialize_conversation_stateraisedTypeErroron live states holding SDK pydantic items (OutputFunctionCallItemis not JSON serializable); now dumps them to plain dicts.Test-design notes: approval tests pin the first turn with
tool_choice="required"so model nondeterminism can't skip the pause being asserted; the plain tool-loop test deliberately doesn't (required persists across turns, matching upstream, and would trip the 20-turn safety limit). Model defaults to a small/cheap one, overridable viaOPENROUTER_E2E_MODEL. Suite skips entirely withoutOPENROUTER_API_KEY.CI: new
e2ejob — warns and exits 0 when the secret is missing (fork PRs), same pattern as upstream typescript-agent. Costs a few cents per run.Verified: full e2e suite passed 3× consecutively live; unit suite, ruff, mypy green.
🤖 Generated with Claude Code