Developer-session e2e: chat-theater recordings, focus timeline, one-mp4 films#962
Merged
Conversation
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
executor-marketing | 0c1d824 | Commit Preview URL Branch Preview URL |
Jun 12 2026, 05:50 PM |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
executor-cloud | 0c1d824 | Jun 12 2026, 05:50 PM |
@executor-js/cli
@executor-js/config
@executor-js/execution
@executor-js/sdk
@executor-js/plugin-file-secrets
@executor-js/plugin-graphql
@executor-js/plugin-keychain
@executor-js/plugin-mcp
@executor-js/plugin-onepassword
@executor-js/plugin-openapi
@executor-js/codemode-core
@executor-js/runtime-quickjs
executor
commit: |
RhysSullivan
force-pushed
the
claude/addspec-auth-derivation
branch
from
98c87fd to
1ab4043
Compare
RhysSullivan
force-pushed
the
claude/e2e-developer-session
branch
from
b6d2abd to
3e1f4ea
Compare
RhysSullivan
changed the base branch from
claude/addspec-auth-derivation
to
main
RhysSullivan
force-pushed
the
claude/e2e-developer-session
branch
2 times, most recently
from
ad5a637 to
aa3e919
Compare
This was referenced Jun 12, 2026
Contributor
Cloudflare previewTorn down — the PR is closed. |
RhysSullivan
force-pushed
the
claude/e2e-developer-session
branch
from
aa3e919 to
9486dc2
Compare
A scenario now runs the way a developer actually tests the product: chat with the real OpenCode TUI in a recorded PTY, the agent registers an API and drops a connect link in the chat (real MCP tool execution), a browser opens the link and pastes a real emulator-minted key while the terminal session stays open, and the chat resumes to verify the connection with a live send confirmed by the provider emulator's request ledger. - e2e/src/clients/replay-brain.ts: local OpenAI-wire chat-completions server (SSE + tool calls) scripted per scenario by transcript inspection, so approval pauses and OpenCode side requests don't derail it. Tool names resolve by suffix against whatever the agent offers. - clients/opencode.ts: makeOpenCodeHome optionally wires a 'replay' provider (openai-compatible) at the brain and pre-allows permissions. - scenarios/connect-handoff-session.test.ts: one continuous terminal.cast timeline with the browser hop forked concurrently; chat-brain.json transcript written as a run artifact even on failure.
…rence The developer-session recording no longer needs a third-party agent binary or a faked LLM: the 'agent' is a chat presentation (agent-chat-tui.ts, a renderer in the recorded PTY) driven by chat-theater.ts while the scenario makes REAL mcporter MCP calls — OAuth, execute, approval pause/resume all genuine, each on-screen tool spinner bracketing the actual call it narrates. Runs anywhere bun runs (no opencode capability), ~28s on cloud. replay-brain.ts stays for the distinct tier where the third-party client's own protocol behavior is under test (OpenCode/Claude Code); real-inference evals remain a separate axis (performance distributions, not pass/fail scenarios).
Runs from chat-theater scenarios produce a terminal.cast (the chat) AND a session.mp4 (the browser hop). The run page previously picked one (video ?? cast), hiding the chat that led to the browser. Now both render under a single 'session' tab in story order — 1 · the chat, in the terminal; 2 · the browser hop, mid-chat — with autoplay reserved for video-only runs.
e2e/desk/run.sh builds a Linux image (Xvfb + openbox + xterm + headed Chromium + ffmpeg) and runs a scenario inside it with E2E_DESK=1: the chat renderer opens in a visible UTF-8 xterm (events over a FIFO instead of a PTY), the browser hop is a real headed window raised over it, and a single x11grab films the whole screen for the whole run — no splicing. The film replaces session.mp4 in the scenario's run dir; the scenario file itself is unchanged (chat-theater switches transports on E2E_DESK). First run builds the image and fills a node_modules volume; subsequent films of the connect-handoff session take ~2 minutes end to end.
bun scripts/film.ts runs/<target>/<slug> turns a chat-theater run's two real recordings into one video that plays like a screen capture of a developer tabbing between full-screen windows: the cast (rendered without idle compression so cast time == video time) is cut at the browser hop — located by the scenario's narrator line, falling back to the largest output gap — and the browser recording fills the gap. Both segments are the genuine recordings; only the cut is editorial, exactly like tabbing. The film registers itself in result.json and the viewer plays film.mp4 as the session when present. Runs anywhere (agg + ffmpeg) — no Docker, no display server; the Desk remains for when actual window layering matters.
Acting on a window IS focusing it: a Playwright step marks the browser focused, a chat-theater event marks the terminal focused. Surfaces record these transitions (plus each recording's start anchor) into the run's timeline.json as a side effect of normal driving — no scenario changes, no declared focus anywhere. film.ts now builds its cut list from that timeline (any number of hops, each act mapped into its recording's own clock via the anchors), keeping the narrator-line/largest-gap heuristic only as a fallback for runs without one. The film is the operations, edited by where they happened.
The emulator serves its own OpenAPI document, so the scenario stops hand-writing a Resend subset and an explicit auth template — addSpec gets just the spec URL, exactly what an agent would send, and the platform derives base URL and the paste-a-token method from the spec itself.
chat.tool now takes the call itself — tool name, the REAL input (the sandbox code the scenario executes, shown as the indented preview block agent TUIs draw), and a result-line renderer — instead of a prose label. The renderer paints name → input block → live spinner → ✓ with the real duration and the real result's first line. The recording reads like an actual agent session: what was called, with what, how long it took, what came back.
film.mp4 was a manual post-step, so any re-run left the viewer showing the parts (cast + browser video) instead of the spliced session. The runner now invokes scripts/film.ts after every successful run that has both terminal.cast and session.mp4 — best-effort, so missing agg/ffmpeg degrades back to the parts rather than failing the run. Every developer- session run's page now plays the single film by default.
Every REST emulator now serves its own OpenAPI document; the connect scenarios register integrations from those live spec URLs instead of hand-written specs.
RhysSullivan
force-pushed
the
claude/e2e-developer-session
branch
from
9486dc2 to
0c1d824
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Watchable e2e runs that read like a developer actually using the product: chat with an agent in a terminal, get a connect link, paste a key in the browser, come back and verify — all real calls, recorded.
What's in here
e2e/src/clients/chat-theater.ts+agent-chat-tui.ts): the default for product-flow recordings. A chat renderer runs in the recorded PTY; every tool call on screen brackets a REAL mcporter MCP call (OAuth, execute, approval resume). Tool calls render like an agent TUI — tool name, the call's real input block, live spinner for the call's real duration, the real result line. No inference, no third-party binary, ~28s per run.e2e/src/clients/replay-brain.ts): a scripted OpenAI-wire chat-completions server for the distinct tier where a third-party CLIENT's behavior is under test (the real OpenCode binary chats through it; scripted by transcript inspection, not turn counting).e2e/src/timeline.ts): acting on a window IS focusing it — Playwright steps mark the browser focused, chat events mark the terminal focused. Surfaces emittimeline.jsonas a side effect of normal driving.e2e/scripts/film.ts: cuts the run's two real recordings into ONE mp4 that plays like a screen capture of tabbing between full-screen windows, using the focus timeline as the edit decision list. No display server needed (agg + ffmpeg).e2e/desk/): optional Docker stage that films a scenario on a real virtual desktop (visible xterm + headed Chromium, single x11grab) for when actual window layering matters. Scenario files are identical across all presentations.film.mp4plays as the session when present.e2e/scenarios/connect-handoff-session.test.ts): the connect-handoff developer session — adds the emulated Resend API by its served spec URL only (emulators.dev now serves OpenAPI documents from every REST emulator), pastes an emulator-minted key through the real add-account UI, and proves the connection with a send confirmed by the emulator's request ledger.Relationship to #957
Was stacked on #957 (addSpec auth derivation), which has merged — this branch is now based directly on main. The connect-handoff developer-session scenario is the end-to-end regression gate for that fix.
Split into a stack
This PR was split — it now carries only the recording framework above. The rest of the original work follows in:
Stack