ADR 0033 Vision Assisted Recovery With Approval

ADR 0033: Vision-assisted recovery with human approval

Status

Accepted (2026-10-10). Amended by ADR 0035: frames of a real camera through the capture worker, automatic analyses on errors.

Context

The project plan lists "Ai Recover und Find Dock: Kamerabild an Vision-Modell, Freigabe per Push" for the KI phase, a later Vision service, and "Wetter, Karten, KI: je ein Adapter-Port". Its safety rules apply to every automated movement: a dead-man switch (a drive command expires after about a second), blades off during transfers, an emergency stop that is always visible, and sensor stops that are never bypassed. The drive channel of ADR 0026 already turns a session recorded by Fleet (fleet.drive.start, can_drive) into ManualDrive frames that the Device-Gateway's controller checks one by one (ADR 0012).

A vision model can look at a snapshot of a stuck mower and suggest how to free it, but it can be wrong, it can miss a child or a pond, and it costs money per request. The owner decided that the service must work with any vision-capable model (cloud or local), that every user configures a provider in the web app, and that an installation may additionally offer its own model.

Decision

A separate Vision service asks a vision model for a short maneuver, a user with can_drive approves it, and the service drives it through the existing drive channel with blades off; the controller stays the safety authority.

1. Provider port. VisionProvider.analyse(images, context, task) -> Assessment. The real adapter uses LiteLLM with a model string (anthropic/claude-sonnet-5-5, openai/gpt-5-mini, gemini/gemini-2.5-flash, mistral/pixtral-large-latest, ollama/llava, openai/<model> plus a base URL for any OpenAI-compatible endpoint); a rule based fake/... provider serves development, demos and tests. A native SDK adapter can be added behind the same port. The answer is structured output: a JSON schema as response_format where the model supports it (Claude gets its native structured outputs), otherwise the schema in the prompt and a tolerant parser; either way Pydantic validates situation, confidence, at most six steps {linear, angular in -1..1, seconds <= 3}, rationale, risk and abort_reasons, with one retry on an invalid answer. The schema has no blade field. Every call is accounted (model, family, tokens, LiteLLM's cost estimate, latency, outcome) in provider_requests.
2. Who pays. A user's own provider (model string, base URL, write-only API key and extra headers encrypted with AES-GCM under MOWDECK_VISION_SECRETS_KEY, user and account as associated data) needs the plan feature ai_recover. The installation default (MOWDECK_VISION_DEFAULT_*, off by default) needs ai_recover_hosted. Resolution per request: own enabled provider, else the installation default when enabled and in the plan, else 409 vision-provider-missing. The hosted feature is a sibling key, not ai_recover.hosted: plan keys are hierarchical, so a plan listing ai_recover would otherwise always grant the hosted model too.
3. Cost control. A per-user daily cap of analyses (MOWDECK_VISION_USER_DAILY_REQUESTS, both providers), a per-account daily cap for the hosted model, an hourly limit of provider tests, and 429 vision-quota beyond them. Images are stored only for the proposal's TTL (MOWDECK_VISION_PROPOSAL_TTL_SECONDS, default 300) unless MOWDECK_VISION_KEEP_IMAGES_DAYS > 0; images and keys are never logged.
4. Scene. Images come from the app (a snapshot of the camera panel's video, JSON base64 or multipart) or, without one, from the mower's gateway (gateway.mower.<id>.photo, core NATS; an optional PhotoCapture.capture_photo() of mowdeck-control, implemented by the simulator from its map). The SceneContext holds the last state, the map around the mower in the mower frame (forward and left metres) with the dock's distance and bearing, recent commands and earlier rounds; no names.
5. Proposal lifecycle. POST /mowers/{id}/recover (can_drive, ai_recover) creates a proposal: analysing then proposed, rejected_by_ai (any risk flag, confidence below MOWDECK_VISION_MIN_CONFIDENCE, no step) or failed. A proposed one publishes vision.proposal.created; Notifications turns it into vision_proposal for the account's users with can_drive on the mower (Web Push and the inbox, deep link to the proposal). At most one open proposal per mower.
6. Approval is mandatory. POST /proposals/{id}/approve (can_drive, one approval, refused after expires_at) records the approver and drives the steps on the approver's behalf: fleet.drive.start with on_behalf_of the approver and the new origin: vision (Fleet checks can_drive again and records the session with its origin), then frames at 5 Hz with ttl_ms 1000 and blade_on = false, then a zero frame and fleet.drive.stop (completed or aborted). POST /reject rejects a waiting proposal or cancels a running one.
7. Abort conditions (checked between frames): an emergency stop by anyone (seen on command.mower.<id>), a sensor stop (any error that is not drive_recoverable, or the lift flag), a new error, another activity, a refused frame, no state for 3 s, leaving the geofence (mapped areas plus MOWDECK_VISION_GEOFENCE_MARGIN_M, or the dock surroundings; never a no-go zone), a cancellation. After an emergency or sensor stop no further frame is sent, not even a zero frame. Execution does not start without a fresh state, a map and a position inside the geofence.
8. Stuck is not a sensor stop. A new optional MowerState.drive_recoverable marks errors that driving may clear (stuck, wheel slip); the controller admits drive frames only for those, so the plan's "sensor stops are never bypassed" still holds for lift, tilt and collision.
9. Verification and rounds. After the steps the outcome is checked (no error, moved at least MOWDECK_VISION_MIN_MOVE_M or turned as planned): recovered or not_recovered; up to MOWDECK_VISION_MAX_ROUNDS (3) rounds per assist session, each a fresh analysis with the earlier rounds in the context and a new approval. find_dock drives only an approach and then sends a normal return_to_dock through fleet.commands.issue (origin vision); it never drives into the dock.
10. Audit. Proposals (requester, model, answer, context, approver, decisions, outcome, drive session, poses) and proposal_steps (timestamps, frames, outcome) are kept; Fleet's drive sessions show origin: vision.

Consequences

• Nothing moves without a human decision; the controller's dead-man timer and blade rules apply to every frame, and a crashed Vision instance stops the mower within one second.
• Images go to the chosen model provider; for cloud models they leave the installation (the settings page says so), local models (Ollama, OpenAI-compatible endpoints) keep them inside.
• LiteLLM makes the image about twice as large as the other services (about 710 MB) and its import takes seconds; the service imports it lazily and warms it up in a thread. Its price map is the bundled one (no request at startup), so cost estimates follow the LiteLLM version.
• Each Vision instance follows telemetry, maps, commands and results of every mower (ephemeral consumers); fine for the expected fleet sizes, a shard filter can follow.
• The hosted model costs the operator money per analysis; the caps bound it, Billing does not meter it yet.
• Not covered yet: realtime push of proposal updates (clients poll), Web Push actions to approve from the notification itself.

Addendum: stereo frames (2026-10-11)

The LUBA 2 AWD X has a front left and a front right camera, and the camera panel can show both side by side (ADR 0032, consequences). AI assist uses the pair:

• Capture in the browser. With both front cameras live and the user's preference stereo_frames on (per user and account, GET/PUT /vision/me/preferences, default on, a switch on the AI provider page), the web app grabs both frames of the same moment (two grabs started together; each in its own requestVideoFrameCallback, timed by the frame's receive time; a pair more than 200 ms apart is grabbed again, then only the left frame is sent).
• API. ImageIn gains optional camera, side (left, right) and captured_at; single-frame requests are unchanged. Two uploaded frames with both sides from two cameras within MOWDECK_VISION_STEREO_MAX_SKEW_MS (500 ms) are a stereo pair; with the preference off only the left frame is analysed.
• Prompt. Both images, left first, each after a caption with its camera and capture time, plus stereo: true in the context and an explicit hint: the left and right front camera of the same moment; use the parallax for a rough distance estimate; do not invent a distance when unsure (the camera spacing is not calibrated).
• Record. proposals.stereo and proposals.frames (camera, side, capture time per image; migration 0004), both images stored for the proposal's lifetime; ProposalOut.stereo and frames; the web app shows "Stereo" and the two images side by side. The cost accounting already counts the tokens of both images (about twice the image share); the detail of the cost row says images=2 stereo=true.
• Light. Unchanged: the light is only switched for the service's own photo (a request without images), so light_used keeps its meaning and a stereo request never switches it.
• Later. Depth estimation from the pair after a one-time calibration (a stated obstacle distance) stays an optional later step; until then distances are the model's rough estimate at most.

Addendum: server-side camera frames and automatic analysis (2026-10-11)

Real Mammotion mowers have no still image command (PyMammotion 0.10.9 has none, see ADR 0035). The Device-Gateway now answers photo requests of camera-capable adapters with a frame from the capture worker (vision-capture, headless Chromium with the Agora Web SDK, optional), so Vision gets a picture without an open browser. Analyses can also start by themselves when a mower enters a recoverable error and a user opted in (trigger = error, notification vision_auto_proposal); the approval rules of this ADR apply unchanged.

Addendum: return to dock and the map image (2026-10-18)

A field report: a LUBA stood paused (a vendor trap error MowDeck did not map yet) in the same lawn as its dock, 15 m away. find_dock answered "not recommended": the model read drive_recoverable: false (it is false for every state without an error) as a sensor stop, one camera frame was unusable, the dock lay outside the 8 m map excerpt and the answer schema had no dock command at all, although a plain return to dock was the right call.

• Action vocabulary. The answer gains action: return_to_dock (the generic dock command of the mower control interface, ADR 0012; the mower drives home on its mapped areas and transfer paths by itself), drive (the steps as before) or none. The prompt names return_to_dock the lowest-risk option and prefers it whenever it applies, also for recover.
• Route check (pure, manufacturer independent). Return to dock applies when the mower stands in a mapped area (or on a transfer path) that is the dock's area (the dock within 3 m of its boundary) or is linked to it by mapped transfer paths, and no lift, tilt or other safety stop (an error that is not drive recoverable, or one naming a lift, tilt, blade or motor fault) is active and the mower is not docked, charging, returning, offline or updating.
• Context. dock_route (mower area, dock area, route, distance, applicable and why), transfer paths in the map excerpt, recent_track (last RTK positions), sensor_stop, and drive_recoverable only with activity error.
• Map image. A 512 x 512 PNG rendered in Vision with Pillow from the generic map model, heading up, with mower, dock, areas, paths, no-go zones, the recent track, a scale bar and a north arrow; one extra image after the camera frames (it never counts against the frame limit).
• Fallback. When the model recommends nothing (or something the rules reject) without a risk flag and the route check allows it, the proposal becomes return_to_dock with fallback: true, labelled as such in the app and the notification. Like every proposal it runs only after a human approves; before the command is sent the route check runs again on the newest state and map, and a lift in between aborts it. Risk flags still mean nothing is offered.

---

Mirrored from docs/adr/0033-vision-assisted-recovery-with-approval.md at commit 8f3f7119bf4d91d9f8099b4597d37bec6990bf2f. Edit the file in the repository, not this wiki page.