Repository navigation
live view and control
An admin watches a workspace's browser in the admin UI's Live tab and can take it over. This area is
the server's half: the signed link and socket, the VNC proxy and its gate, control (a person holds
the browser and agent calls that would change it are refused), await_control and paste.
- An admin opens the Live tab, sees the screen, and clicks and types into it.
- An agent is stuck on a captcha. The admin takes control, solves it and hands back; the agent reads what they did.
- An admin asks for control while an agent's call runs. The call finishes, then control is theirs.
LIVE-1. The admin sign-in, roles and signed links are in auth-and-roles.md. The
monitor, the session lock and end_browser are in
workspaces-and-sessions.md. The capability table, surfaces and error
shape are in surfaces.md. How the Live tab draws is in admin-ui.md.
The spellings and precedence of the live settings are in config.md.
LIVE-2. GET /admin/workspaces/{key}/live, behind the admin door, answers 404 with the first failing
reason, in this order: the live view is switched off (live.enabled is false), this workspace has no browser open, this browser is headless: it has no live view, this browser has no live view (the Grid reported no VNC). GET /admin/workspaces and its stream carry live_enabled
beside workspaces.
LIVE-3. Otherwise the mint answers {"url", "heartbeat", "grace"}: the socket's URL, the presence
interval (10 seconds) and live.grace. With a server token the URL is a signed link for this one
socket and viewer, and it expires exactly 60 seconds after the mint (AUTH-31). It is checked only
at connect, so an open view is not cut off. With no token the URL is the bare path and the viewer
is open.
LIVE-4. The socket, /admin/workspaces/{key}/live/socket, is a WebSocket route mounted beside the custom
HTTP routes. It admits a link whose exp, sig and by verify, the viewer being by, else
whoever passes the admin door by header, the viewer being that principal; with no token it admits
anyone as open. It accepts the upgrade first and refuses by close code, because a browser sees a
refused upgrade only as 1006. A request admitted by neither closes 4401 with the reason AUTH-32
gives. A workspace that has lost its live view since the mint closes 4004 with LIVE-2's reason.
LIVE-5. After admitting, the socket reads the record and the password file once, registers a
tunnel under the browser's Grid id before it dials (so a browser that ends during the handshake
closes it, LIVE-14), and runs it. The tunnel dials the hub's /session/<id>/se/vnc, built from
grid.url with the scheme swapped to ws or wss, never through the ingress, and accepts hub
messages up to 16 MiB. GET {prefix}/live.js serves the built noVNC client with no credential,
no-cache and an ETag, and is a 404 when the UI build lacks it.
LIVE-6. live.password_file names the Grid's VNC password. It is read as bytes at every
connect, stripped of whitespace, and only its first 8 bytes are used, so a rotated Secret needs no
restart. A configured file that cannot be read closes the viewer 4003 the VNC password file cannot be read and warns with the path. With no file, a node that asks for a password fails with this Grid asks for a VNC password and live.password_file is not set.
LIVE-7. Toward the node the proxy is an RFB 3.8 client, and does its handshake first. Another
version, zero security types (the node's reason is passed on), neither type 1 nor type 2 offered,
and a failed password each close the viewer 4003 with the node's words. With type 2 offered and a
password configured the proxy answers the challenge with DES; otherwise it takes type 1.
LIVE-8. Toward the viewer the proxy is an RFB 3.8 server offering only security type None, so
the page never holds the password. Another version or choice closes 4003. The node is always told
the viewer is shared, so a new viewer never drops the others. Both handshakes have 15 seconds
together: a silent node closes 4002 the Grid's VNC server did not answer, a silent viewer 4003 the viewer did not finish the VNC handshake.
LIVE-9. The viewer's stream is parsed into whole RFB client messages, each forwarded or dropped
whole by an allowlist. Picture messages go in every mode. Input messages go only while the tunnel's
mode is control. A mode change applies from the next whole message.
| Type | Message | Length in bytes | Kind |
|---|---|---|---|
| 0, 3, 150 | SetPixelFormat, FramebufferUpdateRequest, EnableContinuousUpdates | 20, 10, 10 | picture |
| 2 | SetEncodings | 4 + 4n, n the u16 at offset 2 | picture |
| 248 | ClientFence | 9 + the u8 at offset 8 | picture |
| 4, 5 | KeyEvent, PointerEvent | 8, 6 | input |
| 6 | ClientCutText | 8 + the absolute s32 at offset 4 | input |
| 255 | QEMU extended key event, subtype 0 only | 12 | input |
| 250, 251 | xvp, SetDesktopSize | 4; 8 + 16 times the u8 at offset 6 | never forwarded |
LIVE-10. SetEncodings goes out without ExtendedMouseButtons and with its count fixed, so a
pointer event is always 6 bytes and gate and node agree where messages end. Any other type, and type
255 with another subtype, closes 4005 unknown VNC message. A ClientCutText over 1 MiB closes 4007 VNC message too large, checked before its body is buffered.
LIVE-11. The page sends JSON as text frames and noVNC sends binary. A frame over 1 KiB,
malformed JSON, an object of another type or a mode other than watch or control closes 4006 bad control message. A tunnel starts in watch and says so first,
{"type":"mode","mode":"watch"}, and says the mode now in force after every mode frame.
{"type":"presence"} resets the grace timer (LIVE-12). {"type":"mode","mode":"control"} asks for
control and is answered watch while the tunnel is not piping (control cannot be taken before
the screen shows), the mode in force when the viewer is already requesting or holding, and else as
LIVE-17 says. {"type":"mode","mode":"watch"} gives control up or withdraws the request.
LIVE-12. The page sends presence every 10 seconds while its tab is visible; a hidden tab goes quiet.
- Each tunnel has one timer of
live.graceseconds, started on accept and reset by a presence frame and by every input message forwarded in Control. A tunnel whose timer runs out ends. - At most once per heartbeat interval, presence or input touches the browser in the monitor, so capture's deadline does not close while a viewer keeps it alive.
- There is no cap on how long a visible tab keeps the view or on viewers per browser.
LIVE-13. A tunnel ends one way, the first recorded; both sockets close and the reason is cut to 123 bytes.
| Code | Reason | Cause |
|---|---|---|
| 4000 | nobody was watching |
the grace ran out |
| 4001 | the browser ended |
session.ended, or the Grid stops listing it |
| 4002 | the Grid closed the live view |
the hub closed or could not be dialled |
| 4003 | what the node or viewer did | a handshake or the password failed |
| 4004 | LIVE-2's reason | refused at connect |
| 4401 |
unauthorized, or the expired-link sentence |
refused at connect |
| 4005 | unknown VNC message |
LIVE-10 |
| 4006 | bad control message |
LIVE-11 |
| 4007 | VNC message too large |
LIVE-10 |
| 1011 | the live view failed |
|
| 1001 | the server is stopping |
|
| 1000 | the viewer closed |
LIVE-14. A viewer close of 1012 (uvicorn stopping) reads as the server stopping, any other as
the viewer leaving. When the viewer left or the hub closed, the Grid is asked once, within 3
seconds, whether it still lists the browser, and the ending is 4001 if not. One bus subscriber
ends a browser's tunnels 4001 on session.ended, and stopping the server ends every tunnel
1001. The registry is process memory, so a restart drops every tunnel and every holder of control.
LIVE-15. Only a person in a live-view tunnel can hold control: a viewer admitted by the admin door or by a link minted for it. No tool, route or flow step takes, grants, refuses or releases control. An agent, HTTP caller or flow is refused by it (LIVE-21 to LIVE-24) and can wait for it to end (LIVE-28 to LIVE-34).
LIVE-16. Control is a state of the workspace's current browser: one episode at a time, requested, then held, then released. An episode records who asked (every viewer that joined, in order, once), when it was requested, granted and released, whether the browser ended under it, and the pages the person reached.
LIVE-17. A viewer in Watch asks by LIVE-11. The first ask opens the episode as requested, and
the viewer is always told requested, even with no call running, because the grant waits for the
pages to be followed. A viewer that asks while an episode exists joins it, and is told control if
it is held, else requested. An ask on a browser other than the episode's releases the old episode
as ended and opens a new one. The gate shuts at the ask. From then until the release the workspace
is controlled for LIVE-21. A call that took its turn before the ask is not interrupted.
LIVE-18. Control is granted when the calls that queued on the browser's lock before the ask have had their turn, and the pages the person reaches are followed. The server marks the lock's queue at the ask and polls the mark every 0.1 second from the event loop, so no thread waits and no call ever waits on the mark.
Following holds the monitor's control reason for up to 15 seconds. A failure still grants, and
the hand-back says the pages could not be followed. The grant tells every requesting tunnel
{"type":"mode","mode":"control"} and opens its gate, and does not stop the running call. Control
is never refused to a viewer and nothing revokes it: a request is only delayed until the two
conditions hold, or withdrawn by the viewer going back to Watch.
LIVE-19. Control is held while any tunnel of the episode is requesting or holding. It releases when the last leaves: by switching to Watch, by its tunnel ending for any cause in LIVE-13 (the grace included), when the browser ends, or when the server stops. It has no time limit of its own.
LIVE-20. A release removes the gate, lets the control reason go, keeps the pages (at most 100,
newest, with a count of those dropped) and makes the episode the workspace's one unsaid hand-back,
replacing an earlier one. A request withdrawn before its grant is an episode never held (held_for
0).
LIVE-21. The gate is checked when a call begins, after the call has taken the session's turn, so
a call queued before the person asked is refused when its turn comes, and before the call is counted
or published. A refused call leaves no in-flight count and no call.finished. A single call, a
bound write and each browser-driving step of a flow all check.
LIVE-22. A call is refused with faults.Controlled unless its capability answers under control.
-
extract,outline,screenshot,print,network_bodyandawait_controlanswer. - Every other call that reaches the browser is refused,
execute_script,assert,save_site_data,write,frame,resizeanddialogincluded. - Calls that never touch the browser (the flow library,
keep_file) and resource reads are not checked. - An answering call whose
urlnames a page other than the browser's is refused as a navigation, and ascreenshotwithwidth,heightorfull_pageas a resize. Aurlthe browser is already on is not a navigation.
LIVE-23. open_session and end_browser take no turn, so they are checked directly, and only
when a browser is open. The admin's End is exempt: ending the session releases control. The person's
paste (LIVE-36) is exempt by its tool name. A flow step refused for control stops the run with the
refusal, whatever its onError says. A flow cannot call await_control.
LIVE-24. The refusal names who and since when. <who> is the viewers now requesting or holding,
joined with and, where admin and open read a person. The time is HH:MM UTC, of the grant
when held and of the ask when requested.
- Held:
<who> has control of this browser since 01:42 UTC. Call await_control (POST /browser/await-control), which returns when they hand back. - Requested:
<who> asked for control of this browser at 01:42 UTC. Call await_control (POST /browser/await-control), which returns when they hand back.
Over HTTP the refusal is a 409 in the usual error shape. Over MCP it is a tool error, Error calling tool '<name>': and the sentence, with no notice added.
LIVE-25. A tools/call result for a named workspace may carry a notice, read at the call's end
so a call in flight when a person asks carries it: a last text block and
structured_content["control"] (Claude Code shows a model only the structured content of a result
that has some; a result without any gains one holding only control). A failed call gets the hint
after its own sentence. HTTP results, a caller naming no workspace and a call that became a task
(its stub is untouched) carry none.
LIVE-26. control is {"state", "who", "since", "hint"}, since in UTC ISO form.
-
requested:<who> asked for control of this browser at 01:42 UTC; it is theirs once this call ends. Call await_control, which returns when they hand back. -
held: the held sentence of LIVE-24. -
handed_back:control of this browser was handed back at 01:57 UTC; the page may have changed. Call await_control for what they did, or look at the page before carrying on.
LIVE-27. A hand-back is said once per workspace, to the first MCP result after the release or to
an await_control that returns it. A new episode forgets one that was not said. await_control is
passed through whole, since its answer says control itself.
LIVE-28. await_control is one capability row: the MCP tool and POST /browser/await-control,
no arguments, read-only, answering under control, never a flow step, one body for both surfaces. It
waits on the workspace's episode until it releases. A blocking caller waits at most live.await_cap
seconds, then answers normally, not as an error: handed_back false, ended false, control with
who, since and state (requested or held), and still in <who>'s control; call again. The
HTTP long poll is blocking. An MCP call that became a task has no cap.
LIVE-29. With no episode it answers at once. With an unsaid hand-back it answers that hand-back,
which says it (LIVE-27). With none it answers handed_back false, ended false, control null,
the page and outline when a browser is open, and nobody has control of this browser: carry on.
LIVE-30. A hand-back answers handed_back true, control as {"who", "since", "until", "held_for"} (UTC ISO times, whole seconds), the page (url, title) and outline read live,
and <who> handed back at 01:57 UTC after 5m12s; <pages>. The duration reads 12s, 5m12s or
1h05m.
LIVE-31. visited lists the pages the person reached, oldest first, as {"url", "title"?},
with dropped counting earlier ones past 100. The last keeps its whole URL, credentials cut;
earlier ones keep origin, path and hash route. It is scrubbed as a resource is, and absent when the
pages could not be followed, which the message says.
LIVE-32. A browser that ended under a person, or is gone when the hand-back is read, answers
ended true with no page or outline, and the browser ended while <who> had control: call open_session. A request withdrawn before its grant answers handed_back true, held_for 0 and
<who> withdrew the request for control at 01:57 UTC before it was granted: nothing changed.
LIVE-33. The page and outline are read as one call that answers under control, never reopening a
browser. It slides the workspace's TTL and writes no page to History. A read that fails leaves them
out and gives page_error in a failed call's words; the hand-back is still said. A server that
stops releases every waiter, which reads nothing and answers handed_back false, control null or
{"who","since"}, and the server is stopping: call await_control again once it is back.
LIVE-34. Over MCP the tool supports background tasks (optional, polled each second, Docket in memory).
A client on the older protocol gets the blocking call, which reports progress at once and every 15
seconds as waiting for <who> to hand back control (PROT-15). Cancelling a wait changes nothing
for the person. A task's worker learns the caller's workspace as PROT-17 says.
LIVE-35. While control is held, the monitor's control reason subscribes browsingContext.load,
historyUpdated and contextCreated on the browser's connection, and lets them go with the
control. Control never counts as capture: a workspace owing only control has no capture buffer and
capturing stays false.
- A page is a
load(after redirects) orhistoryUpdatedin a top-level context: the one the hold learned, or one created with no parent. - A frame's load, a tab open before the hold, a URL with no origin and the same URL twice in a context are not pages.
- A title is one
document.titleevaluation within the BiDi reply timeout, left out on failure.
LIVE-36. POST /admin/workspaces/{key}/live/paste with {"text": "..."} types the text at the
element the person focused and answers {"typed": n}, the characters typed. It is the admin's, like
End: no capability, no tool. A body without a string text is a 400. The viewer admitted must hold
the workspace's episode as held in a tunnel of its own, else it is 409 paste needs Control: switch the live view to Control first.
LIVE-37. A paste never opens a browser: with none open it is a 404 this workspace has no browser open. It takes the session's turn, so it waits for a call in flight. It types with
send_keys at the caret without clearing, following the focused element into frames and open shadow
roots up to 10 deep, and leaves the agent in the top document. One trailing newline is dropped, the
text is cut to 10,000 characters, and newlines inside still type. It records no page. VNC keystrokes
would drop non-ASCII characters (measured on x11vnc), which is why paste goes through WebDriver.
LIVE-38. A paste is a 400 with a fixed sentence when the agent has a frame selected (the agent is working inside a frame, and a paste would move it out of it: type this in Control instead), when
nothing is focused (nothing in the browser has focus: click where the text goes, then paste), when
the focus is nested too deep, and when the browser cannot type a character (ChromeDriver refuses
those outside the Basic Multilingual Plane). Any other failure is the paste failed in the browser
with its own status.
LIVE-39. The kubed.selenium_flow.live logger writes one INFO line per event, live: then the viewer,
the workspace and the session by its start time (session started 2026-10-27T14:02:11Z, or an unknown session):
- opened:
alex opened the live view of w1 (…) in watch; - control asked for;
- a mode now in force:
alex switched to control in w1 (…); - a paste:
alex pasted 42 characters into w1 (…); - left:
alex left the live view of w1 (…) after 12m04s in watch: nobody was watching.
LIVE-40. A refused connect is a WARNING live: refused <who or someone> the live view of w1: <reason>
(AUTH-32), written in place of left by a tunnel that ends before both handshakes finish. An
unreadable password file and an unexpected failure (by its type) warn too.
The viewer is admin for an admin credential, else the principal's username, subject or oidc, or
open with no token; control characters become ?, cut to 64 characters. No line holds a frame, a
keystroke, pasted text, the password, the link's query, a page's URL or title, or a Grid id.
LIVE-41. The live section holds enabled (default true), password_file (default none, a
blank value reads as none), grace (default 30, at least 15) and await_cap (default 90, at least
30). Spellings and precedence are in config.md.
Not built. The reasoning, and the questions still open, are in the multiplayer design.
LIVE-42. (proposed) Agents that share a session may contend for its control. An agent asks for control, waits, and is handed it, as a person is (LIVE-16 to LIVE-20). Replaces, when built, LIVE-15's "Only a person in a live-view tunnel can hold control" and "No tool, route or flow step takes, grants, refuses or releases control". Who wins a race, whether a holder can refuse, the timeout that hands control back and what the waiting agent is told are open.
The action pages are generated from the server's tool schemas, so they describe the version that shipped.
selenium-flow · MIT
Start here
Guides
Lifecycle
Going places
Doing things
Getting things out
Site data
Console and network