Skip to content

live view and control

Dr K edited this page Nov 14, 2026 · 4 revisions

Live view and control

An admin watches a workspace's browser in the admin UI's Live tab and can take it over. This area is the server's half: the signed link and socket, the VNC proxy and its gate, control (a person holds the browser and agent calls that would change it are refused), await_control and paste.

Use cases

  • An admin opens the Live tab, sees the screen, and clicks and types into it.
  • An agent is stuck on a captcha. The admin takes control, solves it and hands back; the agent reads what they did.
  • An admin asks for control while an agent's call runs. The call finishes, then control is theirs.

LIVE-1. The admin sign-in, roles and signed links are in auth-and-roles.md. The monitor, the session lock and end_browser are in workspaces-and-sessions.md. The capability table, surfaces and error shape are in surfaces.md. How the Live tab draws is in admin-ui.md. The spellings and precedence of the live settings are in config.md.

The link and the socket

LIVE-2. GET /admin/workspaces/{key}/live, behind the admin door, answers 404 with the first failing reason, in this order: the live view is switched off (live.enabled is false), this workspace has no browser open, this browser is headless: it has no live view, this browser has no live view (the Grid reported no VNC). GET /admin/workspaces and its stream carry live_enabled beside workspaces.

LIVE-3. Otherwise the mint answers {"url", "heartbeat", "grace"}: the socket's URL, the presence interval (10 seconds) and live.grace. With a server token the URL is a signed link for this one socket and viewer, and it expires exactly 60 seconds after the mint (AUTH-31). It is checked only at connect, so an open view is not cut off. With no token the URL is the bare path and the viewer is open.

LIVE-4. The socket, /admin/workspaces/{key}/live/socket, is a WebSocket route mounted beside the custom HTTP routes. It admits a link whose exp, sig and by verify, the viewer being by, else whoever passes the admin door by header, the viewer being that principal; with no token it admits anyone as open. It accepts the upgrade first and refuses by close code, because a browser sees a refused upgrade only as 1006. A request admitted by neither closes 4401 with the reason AUTH-32 gives. A workspace that has lost its live view since the mint closes 4004 with LIVE-2's reason.

LIVE-5. After admitting, the socket reads the record and the password file once, registers a tunnel under the browser's Grid id before it dials (so a browser that ends during the handshake closes it, LIVE-14), and runs it. The tunnel dials the hub's /session/<id>/se/vnc, built from grid.url with the scheme swapped to ws or wss, never through the ingress, and accepts hub messages up to 16 MiB. GET {prefix}/live.js serves the built noVNC client with no credential, no-cache and an ETag, and is a 404 when the UI build lacks it.

The proxy

LIVE-6. live.password_file names the Grid's VNC password. It is read as bytes at every connect, stripped of whitespace, and only its first 8 bytes are used, so a rotated Secret needs no restart. A configured file that cannot be read closes the viewer 4003 the VNC password file cannot be read and warns with the path. With no file, a node that asks for a password fails with this Grid asks for a VNC password and live.password_file is not set.

LIVE-7. Toward the node the proxy is an RFB 3.8 client, and does its handshake first. Another version, zero security types (the node's reason is passed on), neither type 1 nor type 2 offered, and a failed password each close the viewer 4003 with the node's words. With type 2 offered and a password configured the proxy answers the challenge with DES; otherwise it takes type 1.

LIVE-8. Toward the viewer the proxy is an RFB 3.8 server offering only security type None, so the page never holds the password. Another version or choice closes 4003. The node is always told the viewer is shared, so a new viewer never drops the others. Both handshakes have 15 seconds together: a silent node closes 4002 the Grid's VNC server did not answer, a silent viewer 4003 the viewer did not finish the VNC handshake.

The gate

LIVE-9. The viewer's stream is parsed into whole RFB client messages, each forwarded or dropped whole by an allowlist. Picture messages go in every mode. Input messages go only while the tunnel's mode is control. A mode change applies from the next whole message.

Type Message Length in bytes Kind
0, 3, 150 SetPixelFormat, FramebufferUpdateRequest, EnableContinuousUpdates 20, 10, 10 picture
2 SetEncodings 4 + 4n, n the u16 at offset 2 picture
248 ClientFence 9 + the u8 at offset 8 picture
4, 5 KeyEvent, PointerEvent 8, 6 input
6 ClientCutText 8 + the absolute s32 at offset 4 input
255 QEMU extended key event, subtype 0 only 12 input
250, 251 xvp, SetDesktopSize 4; 8 + 16 times the u8 at offset 6 never forwarded

LIVE-10. SetEncodings goes out without ExtendedMouseButtons and with its count fixed, so a pointer event is always 6 bytes and gate and node agree where messages end. Any other type, and type 255 with another subtype, closes 4005 unknown VNC message. A ClientCutText over 1 MiB closes 4007 VNC message too large, checked before its body is buffered.

The viewer's text frames

LIVE-11. The page sends JSON as text frames and noVNC sends binary. A frame over 1 KiB, malformed JSON, an object of another type or a mode other than watch or control closes 4006 bad control message. A tunnel starts in watch and says so first, {"type":"mode","mode":"watch"}, and says the mode now in force after every mode frame. {"type":"presence"} resets the grace timer (LIVE-12). {"type":"mode","mode":"control"} asks for control and is answered watch while the tunnel is not piping (control cannot be taken before the screen shows), the mode in force when the viewer is already requesting or holding, and else as LIVE-17 says. {"type":"mode","mode":"watch"} gives control up or withdraws the request.

Presence and teardown

LIVE-12. The page sends presence every 10 seconds while its tab is visible; a hidden tab goes quiet.

  • Each tunnel has one timer of live.grace seconds, started on accept and reset by a presence frame and by every input message forwarded in Control. A tunnel whose timer runs out ends.
  • At most once per heartbeat interval, presence or input touches the browser in the monitor, so capture's deadline does not close while a viewer keeps it alive.
  • There is no cap on how long a visible tab keeps the view or on viewers per browser.

LIVE-13. A tunnel ends one way, the first recorded; both sockets close and the reason is cut to 123 bytes.

Code Reason Cause
4000 nobody was watching the grace ran out
4001 the browser ended session.ended, or the Grid stops listing it
4002 the Grid closed the live view the hub closed or could not be dialled
4003 what the node or viewer did a handshake or the password failed
4004 LIVE-2's reason refused at connect
4401 unauthorized, or the expired-link sentence refused at connect
4005 unknown VNC message LIVE-10
4006 bad control message LIVE-11
4007 VNC message too large LIVE-10
1011 the live view failed
1001 the server is stopping
1000 the viewer closed

LIVE-14. A viewer close of 1012 (uvicorn stopping) reads as the server stopping, any other as the viewer leaving. When the viewer left or the hub closed, the Grid is asked once, within 3 seconds, whether it still lists the browser, and the ending is 4001 if not. One bus subscriber ends a browser's tunnels 4001 on session.ended, and stopping the server ends every tunnel 1001. The registry is process memory, so a restart drops every tunnel and every holder of control.

Control

LIVE-15. Only a person in a live-view tunnel can hold control: a viewer admitted by the admin door or by a link minted for it. No tool, route or flow step takes, grants, refuses or releases control. An agent, HTTP caller or flow is refused by it (LIVE-21 to LIVE-24) and can wait for it to end (LIVE-28 to LIVE-34).

LIVE-16. Control is a state of the workspace's current browser: one episode at a time, requested, then held, then released. An episode records who asked (every viewer that joined, in order, once), when it was requested, granted and released, whether the browser ended under it, and the pages the person reached.

LIVE-17. A viewer in Watch asks by LIVE-11. The first ask opens the episode as requested, and the viewer is always told requested, even with no call running, because the grant waits for the pages to be followed. A viewer that asks while an episode exists joins it, and is told control if it is held, else requested. An ask on a browser other than the episode's releases the old episode as ended and opens a new one. The gate shuts at the ask. From then until the release the workspace is controlled for LIVE-21. A call that took its turn before the ask is not interrupted.

LIVE-18. Control is granted when the calls that queued on the browser's lock before the ask have had their turn, and the pages the person reaches are followed. The server marks the lock's queue at the ask and polls the mark every 0.1 second from the event loop, so no thread waits and no call ever waits on the mark.

Following holds the monitor's control reason for up to 15 seconds. A failure still grants, and the hand-back says the pages could not be followed. The grant tells every requesting tunnel {"type":"mode","mode":"control"} and opens its gate, and does not stop the running call. Control is never refused to a viewer and nothing revokes it: a request is only delayed until the two conditions hold, or withdrawn by the viewer going back to Watch.

LIVE-19. Control is held while any tunnel of the episode is requesting or holding. It releases when the last leaves: by switching to Watch, by its tunnel ending for any cause in LIVE-13 (the grace included), when the browser ends, or when the server stops. It has no time limit of its own.

LIVE-20. A release removes the gate, lets the control reason go, keeps the pages (at most 100, newest, with a count of those dropped) and makes the episode the workspace's one unsaid hand-back, replacing an earlier one. A request withdrawn before its grant is an episode never held (held_for 0).

What a call gets back

LIVE-21. The gate is checked when a call begins, after the call has taken the session's turn, so a call queued before the person asked is refused when its turn comes, and before the call is counted or published. A refused call leaves no in-flight count and no call.finished. A single call, a bound write and each browser-driving step of a flow all check.

LIVE-22. A call is refused with faults.Controlled unless its capability answers under control.

  • extract, outline, screenshot, print, network_body and await_control answer.
  • Every other call that reaches the browser is refused, execute_script, assert, save_site_data, write, frame, resize and dialog included.
  • Calls that never touch the browser (the flow library, keep_file) and resource reads are not checked.
  • An answering call whose url names a page other than the browser's is refused as a navigation, and a screenshot with width, height or full_page as a resize. A url the browser is already on is not a navigation.

LIVE-23. open_session and end_browser take no turn, so they are checked directly, and only when a browser is open. The admin's End is exempt: ending the session releases control. The person's paste (LIVE-36) is exempt by its tool name. A flow step refused for control stops the run with the refusal, whatever its onError says. A flow cannot call await_control.

LIVE-24. The refusal names who and since when. <who> is the viewers now requesting or holding, joined with and, where admin and open read a person. The time is HH:MM UTC, of the grant when held and of the ask when requested.

  • Held: <who> has control of this browser since 01:42 UTC. Call await_control (POST /browser/await-control), which returns when they hand back.
  • Requested: <who> asked for control of this browser at 01:42 UTC. Call await_control (POST /browser/await-control), which returns when they hand back.

Over HTTP the refusal is a 409 in the usual error shape. Over MCP it is a tool error, Error calling tool '<name>': and the sentence, with no notice added.

The notice on an MCP result

LIVE-25. A tools/call result for a named workspace may carry a notice, read at the call's end so a call in flight when a person asks carries it: a last text block and structured_content["control"] (Claude Code shows a model only the structured content of a result that has some; a result without any gains one holding only control). A failed call gets the hint after its own sentence. HTTP results, a caller naming no workspace and a call that became a task (its stub is untouched) carry none.

LIVE-26. control is {"state", "who", "since", "hint"}, since in UTC ISO form.

  • requested: <who> asked for control of this browser at 01:42 UTC; it is theirs once this call ends. Call await_control, which returns when they hand back.
  • held: the held sentence of LIVE-24.
  • handed_back: control of this browser was handed back at 01:57 UTC; the page may have changed. Call await_control for what they did, or look at the page before carrying on.

LIVE-27. A hand-back is said once per workspace, to the first MCP result after the release or to an await_control that returns it. A new episode forgets one that was not said. await_control is passed through whole, since its answer says control itself.

await_control

LIVE-28. await_control is one capability row: the MCP tool and POST /browser/await-control, no arguments, read-only, answering under control, never a flow step, one body for both surfaces. It waits on the workspace's episode until it releases. A blocking caller waits at most live.await_cap seconds, then answers normally, not as an error: handed_back false, ended false, control with who, since and state (requested or held), and still in <who>'s control; call again. The HTTP long poll is blocking. An MCP call that became a task has no cap.

LIVE-29. With no episode it answers at once. With an unsaid hand-back it answers that hand-back, which says it (LIVE-27). With none it answers handed_back false, ended false, control null, the page and outline when a browser is open, and nobody has control of this browser: carry on.

LIVE-30. A hand-back answers handed_back true, control as {"who", "since", "until", "held_for"} (UTC ISO times, whole seconds), the page (url, title) and outline read live, and <who> handed back at 01:57 UTC after 5m12s; <pages>. The duration reads 12s, 5m12s or 1h05m.

LIVE-31. visited lists the pages the person reached, oldest first, as {"url", "title"?}, with dropped counting earlier ones past 100. The last keeps its whole URL, credentials cut; earlier ones keep origin, path and hash route. It is scrubbed as a resource is, and absent when the pages could not be followed, which the message says.

LIVE-32. A browser that ended under a person, or is gone when the hand-back is read, answers ended true with no page or outline, and the browser ended while <who> had control: call open_session. A request withdrawn before its grant answers handed_back true, held_for 0 and <who> withdrew the request for control at 01:57 UTC before it was granted: nothing changed.

LIVE-33. The page and outline are read as one call that answers under control, never reopening a browser. It slides the workspace's TTL and writes no page to History. A read that fails leaves them out and gives page_error in a failed call's words; the hand-back is still said. A server that stops releases every waiter, which reads nothing and answers handed_back false, control null or {"who","since"}, and the server is stopping: call await_control again once it is back.

LIVE-34. Over MCP the tool supports background tasks (optional, polled each second, Docket in memory). A client on the older protocol gets the blocking call, which reports progress at once and every 15 seconds as waiting for <who> to hand back control (PROT-15). Cancelling a wait changes nothing for the person. A task's worker learns the caller's workspace as PROT-17 says.

LIVE-35. While control is held, the monitor's control reason subscribes browsingContext.load, historyUpdated and contextCreated on the browser's connection, and lets them go with the control. Control never counts as capture: a workspace owing only control has no capture buffer and capturing stays false.

  • A page is a load (after redirects) or historyUpdated in a top-level context: the one the hold learned, or one created with no parent.
  • A frame's load, a tab open before the hold, a URL with no origin and the same URL twice in a context are not pages.
  • A title is one document.title evaluation within the BiDi reply timeout, left out on failure.

Paste

LIVE-36. POST /admin/workspaces/{key}/live/paste with {"text": "..."} types the text at the element the person focused and answers {"typed": n}, the characters typed. It is the admin's, like End: no capability, no tool. A body without a string text is a 400. The viewer admitted must hold the workspace's episode as held in a tunnel of its own, else it is 409 paste needs Control: switch the live view to Control first.

LIVE-37. A paste never opens a browser: with none open it is a 404 this workspace has no browser open. It takes the session's turn, so it waits for a call in flight. It types with send_keys at the caret without clearing, following the focused element into frames and open shadow roots up to 10 deep, and leaves the agent in the top document. One trailing newline is dropped, the text is cut to 10,000 characters, and newlines inside still type. It records no page. VNC keystrokes would drop non-ASCII characters (measured on x11vnc), which is why paste goes through WebDriver.

LIVE-38. A paste is a 400 with a fixed sentence when the agent has a frame selected (the agent is working inside a frame, and a paste would move it out of it: type this in Control instead), when nothing is focused (nothing in the browser has focus: click where the text goes, then paste), when the focus is nested too deep, and when the browser cannot type a character (ChromeDriver refuses those outside the Basic Multilingual Plane). Any other failure is the paste failed in the browser with its own status.

The audit line

LIVE-39. The kubed.selenium_flow.live logger writes one INFO line per event, live: then the viewer, the workspace and the session by its start time (session started 2026-10-27T14:02:11Z, or an unknown session):

  • opened: alex opened the live view of w1 (…) in watch;
  • control asked for;
  • a mode now in force: alex switched to control in w1 (…);
  • a paste: alex pasted 42 characters into w1 (…);
  • left: alex left the live view of w1 (…) after 12m04s in watch: nobody was watching.

LIVE-40. A refused connect is a WARNING live: refused <who or someone> the live view of w1: <reason> (AUTH-32), written in place of left by a tunnel that ends before both handshakes finish. An unreadable password file and an unexpected failure (by its type) warn too.

The viewer is admin for an admin credential, else the principal's username, subject or oidc, or open with no token; control characters become ?, cut to 64 characters. No line holds a frame, a keystroke, pasted text, the password, the link's query, a page's URL or title, or a Grid id.

Settings

LIVE-41. The live section holds enabled (default true), password_file (default none, a blank value reads as none), grace (default 30, at least 15) and await_cap (default 90, at least 30). Spellings and precedence are in config.md.

Proposed: several sessions in a workspace

Not built. The reasoning, and the questions still open, are in the multiplayer design.

LIVE-42. (proposed) Agents that share a session may contend for its control. An agent asks for control, waits, and is handed it, as a person is (LIVE-16 to LIVE-20). Replaces, when built, LIVE-15's "Only a person in a live-view tunnel can hold control" and "No tool, route or flow step takes, grants, refuses or releases control". Who wins a race, whether a holder can refuse, the timeout that hands control back and what the waiting agent is told are open.

Clone this wiki locally