ADR 0044 Drive View With Camera

ADR 0044: Drive view with camera

Status

Accepted (2026-10-16); amends ADR 0026 (the drive page) and ADR 0032 (the camera player is shared). Amended by ADR 0045 (proposed, 2026-10-17): the stage plays the camera relay over WHEP; the relay never repeats or invents frames, so the freeze detection and the hold are unchanged.

Context

The drive page (ADR 0026) shows a small map, the session strip, the joystick, the speed limit, the blade switch and the emergency stop. Driving by the map alone is driving blind: the map knows where the mower is, not what is in front of it. The live camera (ADR 0032) exists on the mower page, but switching pages while driving ends the session, and two viewers of one mower cannot run at once, because a camera session carries a single viewer uid.

A camera picture brings its own risks. A stream over 4G freezes, and a frozen frame looks exactly like a live one: a driver who trusts it steers into whatever moved since. Phones are held sideways to drive, where a page that scrolls hides the stop. And the picture must not swallow touches meant for the stick floating over it.

Decision

One stage, shared. The camera's picture becomes CameraStage.svelte: it owns the CameraViewer (one session per mounted stage), the tiles, the stereo guard and the freeze detection. CameraPanel wraps it with the picker, notes, diagnostics and options; the drive page uses it with compact (fills its box, minimal captions). Exactly one camera session per drive page visit: switching to the map or leaving the page unmounts the stage and leaves the channel. A viewer shared across pages was considered and left out: it would keep a session (and 4G minutes) alive on pages that do not show it.

The view from the capabilities. driveView(caps, stored, cameraError) in lib/drive/layout.ts: the map without a camera capability (or none) and after a camera error during the visit (with the note "Kamera nicht verfügbar – Karte"), otherwise the stored choice, the camera by default. A "Kamera | Karte" switch appears only with a camera capability. The view, the map thumbnail's visibility and the drive camera's layout (front left by default, "Both" on request) are remembered per browser, every storage access guarded.

Three layouts. Phones upright: the camera 16:9 on top, a one-line readout (state lamp, battery, speed, video latency), a 240 px stick, speed limit and the session Start/Stop, the session details folded; the stop bar fixed at the bottom. Phones held sideways ((orientation: landscape) and (max-height: 520px)): a full-bleed cockpit that covers the screen and never scrolls, the picture (or the map) edge to edge, a translucent overlay joystick for the right thumb (Joystick overlay, size through --md-stick-size), the emergency stop upright in a rail on the left with the session Start/Stop above it (CommandBar/StopBar rail). Desktop: the picture fills the centre, the map is a corner thumbnail, the console column stays on the right. The emergency stop and the session Stop are fully visible and reachable in all three; the picture's video and canvas never take pointer events. Overlays over video use theme-independent glass tokens (--md-hud-*).

The camera light over the picture (amendment, 2026-10-16). With the light capability the camera view carries the light switch as a glass chip (LightChip), the same rules, permission (can_control) and command (set_light) as the tile on the mower page (lib/mower/light.ts). Upright it is a small chip in the tools row at the top right of the picture; sideways and on the desktop it takes its own row at the right edge under the view switch, which puts it right above the overlay stick for the right thumb and clear of the readout, the stop rail, the session Stop and the stick. It stays over a frozen picture and does not touch the pilot (the controller keeps the drive session open for SetLight). While the mower is driven, a light that was switched gets no success toast (the chip shows the state; the toast would land on the stick); failures still toast. No chip in the map view.

A frozen picture holds the mower. The stage reads every shown tile's frame counter five times a second (FreezeWatch): no new frame for more than 1 s, or a picture that went away after it was live, is frozen. The picture turns grey under a hatched "Bild steht" plate, so it never looks live, and the stage reports stalled. The drive page then holds the pilot: one forced zero frame at once, then zero frames at the idle cadence for every stick sample, never with blades, and the blades are disarmed; the session stays open so driving can continue. When the picture runs again a stick nobody holds is free at once; a stick held through the freeze stays latched at zero until it is centred or let go and touched again, so the mower never jumps to a deflection made blind. A camera that is still connecting does not hold the mower (nothing is shown that could pass for live); the driver can switch to the map.

Unchanged: the dead-man timer of the Device-Gateway stays the only safety authority, and the session still ends on leaving the page, hiding the tab and losing the socket. Turning the phone lets go of a held stick (one zero frame) rather than carrying it over to the other layout. A lost drive link still ends a joystick session as mower_refused; the drive channel has no link_lost reason, so the end labels need none.

Freeze detection on presented frames (amendment, 2026-10-17). On Safari the picture showed "Bild steht" about 2 s after it started, on Chrome not. The owner confirmed the picture really stands there (the iOS app, WebKit as well, showed none at all), so the detector was right that time; but the review showed that the counters it relied on are not reliable for a WebRTC MediaStream: the playback time runs on as a wall clock while the picture stands (measured in Chromium and Firefox: currentTime kept rising after the source stopped sending frames), Firefox reports getVideoPlaybackQuality().totalVideoFrames as 0 for a MediaStream (so it fell back to that clock and could never see a freeze), and what WebKit counts there is not specified to follow presented frames. Decision: a video's counter is the number of frames the browser actually presented, counted with requestVideoFrameCallback (Chrome, Edge, Safari 15.4+, Firefox 132+), with the decoded frames only where that API is missing and the playback time never. The threshold follows each tile's frame interval: three typical intervals, at least 1.5 s and at most 3 s, so a docked mower's steady 1 to 5 fps over 4G is not a freeze while a stream that stops is caught within 1.5 s (3 s at the slowest); a gap that counted as frozen never raises the threshold. The Agora SDK's own rates (receiveFrameRate, decodeFrameRate, renderFrameRate) were considered as a further signal and left out of the decision: they are averaged over seconds and the bytes still arrive when the decoder is stuck, so they would hide exactly the freeze that matters; they appear in the debug overlay instead. Why WebKit stops rendering this stream is found on the device with the debug overlay (?camdebug=1: element events, presented/decoded/received frames, PLI/FIR, codec, decoder, encoded transform) and its experiments (?camcodec=h264|vp8, ?camclone=1, ?camparam=NAME=value); a live picture the browser pauses by itself is resumed (at most once a second). The e2e suite plays the test pattern through a real <video> (canvas.captureStream(), ?camfakevideo=1) and runs the camera freeze and the drive view with camera in WebKit too.

Debug overlay removed (amendment, 2026-10-18). The camera relay works on the owner's devices, so the owner removed the debug overlay (?camdebug=1, ?camdebug=0, "Kopieren") and its experiment switches (?camcodec=, ?camclone=1, ?camparam=, ?camfakevideo=1) from the web app; the mentions above and under "Consequences" are history. Unchanged: the freeze detection on presented frames with its adaptive threshold, "Bild steht" with the drive view's hold and latch, the resume of a live picture the browser paused, and the "Diagnose" line. The e2e suite now checks the freeze of a real <video> through the camera relay's fake source and its freeze test hook instead of ?camfakevideo=1, and reads the stage's data-phase and data-frozen instead of the overlay. The console statistics behind localStorage.mowdeck.cameraDebug went the same way; the viewer removes that key and mowdeck.cameraExperiments from browsers that still carry them.

Manufacturer independence

Generic, the same for every manufacturer: the camera capability as the only switch for the camera view, manual_drive, drive_reverse and links as before, CameraStage, the freeze detection on frame counters of the page's own <video> or canvas elements, the pilot's hold and latch, the three layouts, the HUD tokens, the overlay joystick, the stop rail and the light chip (the light capability only). The web app reads no vendor field to decide anything here. The simulator's fake camera (a test pattern with a data-freeze hook) and its camera and light options (default on, off for a mower without a camera or light) carry the tests, which import no manufacturer package.

Adapter work, once per manufacturer: none beyond the camera session of ADR 0032. A manufacturer whose player is not Agora adds a player that fills the tile's <video> element; the freeze detection then works unchanged.

Consequences

Easier: driving with sight of what is in front of the mower, on the phone held sideways as on the desktop, with the stops always in reach. A stalled stream can no longer pass for a live one, and it stops the mower instead of letting it drive on blind.

Harder: the camera view costs video data (4G minutes) for as long as the drive page shows it; the map view and leaving the page end it. A flaky stream holds the mower often; the driver then switches to the map. The hold depends on the browser reporting presented frames (requestVideoFrameCallback), with the decoded frames (getVideoPlaybackQuality) as the fallback; a browser with neither cannot check a video and trusts the tile status.

To check on hardware: Safari on macOS and iOS and the iOS app with ?camdebug=1 (does the picture keep running, and if not, which counter stops first), the threshold with a slow stream of the docked mower, and the freeze detection and the latency readout with the LUBA 2 AWD X's Agora stream over 4G (how often a real stream stalls for more than a second), and the sideways layout on the owner's iPhone in the Capacitor shell (safe areas, the stop rail under the thumb).

---

Mirrored from docs/adr/0044-drive-view-with-camera.md at commit 20288b7586dddd62e476bdf3a7d8fc1bb920ed7d. Edit the file in the repository, not this wiki page.