Reference Vision

vision API

AI assisted recovery: proposals, approvals and the provider settings.

Base path /api/v1/vision behind the gateway. Authentication, rate limits and the error format are the same for every service: see Scripting MowDeck. Rendered from apps/web/openapi/vision.json (OpenAPI 3.1), which you can also load into any OpenAPI tool.

Operations

vision

GET /api/v1/vision/me/preferences

My Preferences

The caller's AI assist preferences (the defaults until saved): stereo_frames and the opt-in to automatic AI analyses on errors (off until saved) with the installation's limits (cooldown per mower, analyses per mower and day).

Status	Meaning	Body
200	Successful Response	application/json PreferencesOut

PUT /api/v1/vision/me/preferences

Put My Preferences

Change the caller's AI assist preferences; a field left out keeps its value.

auto_analysis: when a mower of the account enters a recoverable error (stuck, blocked, outside the work area, wheel slip), the service takes a photo, analyses it with the caller's provider (or the installation's) and asks for approval like a manual request; nothing is executed automatically. Enabling needs the plan feature ai_recover (403 feature-not-in-plan; nothing is saved then).

Request body (required): application/json PreferencesIn

Status	Meaning	Body
200	Successful Response	application/json PreferencesOut
422	Validation Error	application/json HTTPValidationError

GET /api/v1/vision/me/provider

My Provider

The caller's own provider (the key only as api_key_set) and the provider a request would use now (effective).

Status	Meaning	Body
200	Successful Response	application/json ProviderOut

PUT /api/v1/vision/me/provider

Put My Provider

Save the caller's provider. 422 for an unknown family, a refused base URL or bad headers; 409 vision-secrets-unavailable when a key is sent but cannot be stored.

Request body (required): application/json ProviderIn

Status	Meaning	Body
200	Successful Response	application/json ProviderOut
422	Validation Error	application/json HTTPValidationError

DELETE /api/v1/vision/me/provider

Delete My Provider

Delete the caller's provider (404 when there is none).

Status	Meaning	Body
204	Successful Response	no body

POST /api/v1/vision/me/provider/test

Test My Provider

One minimal vision call with a tiny generated image: model, latency, tokens and the estimated cost. 429 vision-quota beyond the hourly limit.

Status	Meaning	Body
200	Successful Response	application/json ProviderTestOut

GET /api/v1/vision/mowers/{mower_id}/proposals

Mower Proposals

Proposals of the mower, newest first.

Parameter	In	Type	Required	Notes
mower_id	path	string (uuid)	yes
limit	query	integer	no	min 1; max 100; default 20
before	query	string (date-time) or null	no

Status	Meaning	Body
200	Successful Response	application/json ProposalPageOut
422	Validation Error	application/json HTTPValidationError

POST /api/v1/vision/mowers/{mower_id}/recover

Recover

Start AI assist for the mower: task recover or find_dock, optional camera snapshots (base64, each optionally with camera, side and captured_at; a left and a right frame of the same moment are analysed as a stereo pair when the caller's preference stereo_frames is on, else only the left one). The analysis runs in the background; poll GET /proposals/{id}. Without images the mower's gateway is asked for one. 403 feature-not-in-plan without ai_recover, 409 vision-provider-missing without a usable provider, 409 when a proposal for the mower is open, 429 vision-quota beyond the daily cap.

Parameter	In	Type	Required	Notes
mower_id	path	string (uuid)	yes

Request body (required): application/json RecoverIn

Status	Meaning	Body
202	Successful Response	application/json ProposalOut
422	Validation Error	application/json HTTPValidationError

POST /api/v1/vision/mowers/{mower_id}/recover/upload

Recover Upload

recover with the snapshots as multipart/form-data files (field images).

Parameter	In	Type	Required	Notes
mower_id	path	string (uuid)	yes

Request body: multipart/form-data Body_recover_upload_api_v1_vision_mowers__mower_id__recover_upload_post

Status	Meaning	Body
202	Successful Response	application/json ProposalOut
422	Validation Error	application/json HTTPValidationError

GET /api/v1/vision/proposals/{proposal_id}

Get Proposal

One proposal with its step audit; poll while final is false.

Parameter	In	Type	Required	Notes
proposal_id	path	string (uuid)	yes

Status	Meaning	Body
200	Successful Response	application/json ProposalOut
422	Validation Error	application/json HTTPValidationError

POST /api/v1/vision/proposals/{proposal_id}/approve

Approve

Approve: the chosen option runs on the caller's behalf (blades off, dead-man 1 s). The optional body picks the option (model or fallback; omitted: the default action). One approval per proposal; 409 vision-proposal-expired after expires_at; 422 for an option the proposal does not offer.

Parameter	In	Type	Required	Notes
proposal_id	path	string (uuid)	yes

Request body: application/json ApproveIn or null

Status	Meaning	Body
200	Successful Response	application/json ProposalOut
422	Validation Error	application/json HTTPValidationError

GET /api/v1/vision/proposals/{proposal_id}/images/{index}

Proposal Image

An image the model saw; 404 once it was purged (after the proposal TTL).

Parameter	In	Type	Required	Notes
index	path	integer	yes	min 0; max 7
proposal_id	path	string (uuid)	yes

Status	Meaning	Body
200	Successful Response	image/jpeg any, image/png any
422	Validation Error	application/json HTTPValidationError

POST /api/v1/vision/proposals/{proposal_id}/reject

Reject

Reject a waiting proposal, or stop a running execution at once.

Parameter	In	Type	Required	Notes
proposal_id	path	string (uuid)	yes

Status	Meaning	Body
200	Successful Response	application/json ProposalOut
422	Validation Error	application/json HTTPValidationError

POST /api/v1/vision/proposals/{proposal_id}/retry

Retry

Another round after one that did not help (can_retry), with fresh snapshots.

Parameter	In	Type	Required	Notes
proposal_id	path	string (uuid)	yes

Request body (required): application/json RetryIn

Status	Meaning	Body
202	Successful Response	application/json ProposalOut
422	Validation Error	application/json HTTPValidationError

GET /api/v1/vision/providers

Providers

The model families users can configure and whether the installation's model may be used by the caller's account.

Status	Meaning	Body
200	Successful Response	application/json ProvidersOut

GET /api/v1/vision/status

Vision Status

What the operator configured: whether a hosted default exists and its model string (never a key), whether users can store keys, the base URL policy.

Status	Meaning	Body
200	Successful Response	application/json StatusOut

Schemas

ApproveIn

POST /proposals/{id}/approve: which option runs (the body is optional).

Field	Type	Required	Notes
option	"model" | "fallback" or null	no	model: the model's recommendation (also a low-confidence one offered next to the fallback); fallback: the rule based return to dock. Omitted: the default option (action). An option the proposal does not offer answers 422.

AssessmentOut

What the model thinks (without the steps, which are listed separately).

Field	Type	Required	Notes
abort_reasons	array of string	yes
action	ProposalAction	no	What the model recommended: drive, return_to_dock, resume or none.; default "drive"
confidence	number	yes
rationale	string	yes
risk	array of RiskFlag	yes
situation	string	yes

Body_recover_upload_api_v1_vision_mowers__mower_id__recover_upload_post

Field	Type	Required	Notes
images	array of string or null	no
task	VisionTask	no	default "recover"

FamilyOut

A model family the settings page offers.

Field	Type	Required	Notes
default_api_base	string or null	yes
examples	array of string	yes
key	string	yes
key_env	string or null	yes	Variable LiteLLM reads for the operator's key.
label	string	yes
local	boolean	yes	Runs on own hardware: images stay local.
needs_api_base	boolean	yes
needs_key	boolean	yes
prefix	string	yes	Model strings look like <prefix>/<model>.

FrameOut

What one image of the proposal was (fetch it from images/{index}).

Field	Type	Required	Notes
camera	string or null	no
captured_at	string (date-time) or null	no
index	integer	yes
side	"left" | "right" or null	no
source	"upload" | "mower" | "map"	yes	upload: sent by the app; mower: the mower's own photo; map: the top-down map rendering around the mower (heading up) that Vision adds.

HTTPValidationError

Field	Type	Required	Notes
detail	array of ValidationError	no

ImageIn

A camera snapshot, base64 encoded (JPEG or PNG), optionally with the camera that took it. Two frames with side left and right captured at most MOWDECK_VISION_STEREO_MAX_SKEW_MS apart form a stereo pair.

Field	Type	Required	Notes
camera	string or null	no	The camera's display name, e.g. Front left.
captured_at	string (date-time) or null	no	When the frame was grabbed (with a time zone).
data	string	yes	Base64 (standard) of the image bytes.; max length 14000000
media_type	"image/jpeg" | "image/png"	yes
side	"left" | "right" or null	no	left or right: the frame is one of the two front cameras (stereo).

JobAreaOut

One planned area of the job, in driving order.

Field	Type	Required	Notes
id	string	yes
name	string or null	no
status	"done" | "current" | "next" | "planned"	yes

JobOut

The mowing job the mower was running when the scene was analysed (what resume continues).

Field	Type	Required	Notes
areas	array of JobAreaOut	yes	Planned areas in driving order.
current_area	string or null	no
entry_distance_m	number or null	no	Distance to the entry of the way to the next area.
mower_at_entry	boolean	no	The mower stood at the entry of the way to the next area.; default false
next_area	string or null	no
progress_percent	integer or null	no
resumable	boolean	yes
status	"running" | "paused" | "paused_by_error" | "returning" | "spot_run"	yes

LastTestOut

The outcome of the last provider test.

Field	Type	Required	Notes
at	string (date-time)	yes
cost_usd	number or null	no
detail	string or null	no
input_tokens	integer or null	no
latency_ms	integer or null	no
ok	boolean	yes
output_tokens	integer or null	no

OptionOut

One option a waiting proposal offers; the user approves one of them.

Field	Type	Required	Notes
action	ProposalAction	yes
confidence	number or null	no	The model's confidence (null for the fallback).
default	boolean	yes	What an approval without an option runs (action).
low_confidence	boolean	yes	The model recommends it with a confidence below the threshold.
option	"model" | "fallback"	yes	model: the model's recommendation; fallback: the rule based return to dock. Pass it to approve.
steps	array of StepOut	no	The drive steps (blades off) of this option.

PreferencesIn

PUT /me/preferences: a field left out (null) keeps its value.

Field	Type	Required	Notes
auto_analysis	boolean or null	no	Opt in to (or out of) automatic AI analysis on errors.
stereo_frames	boolean or null	no	Send both front camera frames when both views are live.

PreferencesOut

GET /me/preferences: the caller's AI assist preferences and the installation's limits of automatic analyses.

Field	Type	Required	Notes
auto_analysis	boolean	yes	Automatic AI analysis on errors for the caller (opt-in, default off).
auto_analysis_available	boolean	yes	False when the installation switched automatic analyses off.
auto_cooldown_minutes	integer	yes	No automatic analysis for a mower within this time after the previous one.
auto_daily_per_mower	integer	yes	Automatic analyses per mower and UTC day.
stereo_frames	boolean	yes	Send both front camera frames when both views are live (a stereo pair: two images, about twice the image share of the cost).
updated_at	string (date-time) or null	yes	Null while the defaults apply.

PrincipalKind

What kind of subject a principal is.

Type: "user" | "subuser" | "api_token" | "service"

PrincipalRef

A compact reference to a principal, embedded in commands and audit records.

Field	Type	Required	Notes
display_name	string or null	no
kind	PrincipalKind	yes	Kind of subject.
subject	string (uuid)	yes	Stable subject id (JWT sub).
token_id	string (uuid) or null	no	Token id (JWT jti), if known.

ProposalAction

What a proposal does once a human approves it.

Type: "drive" | "return_to_dock" | "resume" | "none"

ProposalOut

A proposal of AI assist (poll it while final is false).

Field	Type	Required	Notes
action	ProposalAction	no	The default option, what an approval without a choice executes (after an approval: what runs): drive (the steps, blades off), return_to_dock (the normal dock command; no steps) or resume (the steps, if any, then the paused job continues with resume_mow). none only on rejected rows.; default "drive"
alternative_action	ProposalAction or null	no	The model's low-confidence recommendation next to the fallback action; approve it with option model.
analysed_at	string (date-time) or null	yes
approved_at	string (date-time) or null	yes
approved_by	PrincipalRef or null	yes
approved_option	"model" | "fallback" or null	no	Which option the approval ran.
assessment	AssessmentOut or null	yes
can_retry	boolean	yes
chain_id	string (uuid)	yes
created_at	string (date-time)	yes
decided_at	string (date-time) or null	yes
decided_by	PrincipalRef or null	yes	Who rejected or cancelled it.
detail	string or null	yes	Why it was rejected, failed or how it ended.
drive_session_id	string (uuid) or null	yes
expires_at	string (date-time) or null	yes	Approval is refused after this time.
fallback	boolean	no	action is the rule based return to dock fallback, not the model's recommendation: it applies (mower in a mapped area linked to the dock, no safety stop). It still needs an approval.; default false
final	boolean	yes
finished_at	string (date-time) or null	yes
frames	array of FrameOut	yes	What each image was, in the order the model saw them (left first in stereo).
hosted	boolean	yes
id	string (uuid)	yes
image_count	integer	yes
job	JobOut or null	no	The mowing job as the model saw it (with action resume the job that continues); null when no job was known.
light_used	boolean	yes	The photo was taken with the mower's light switched on (it was too dark).
low_confidence	boolean	no	The model's recommendation is offered although its confidence is below the threshold: as action, or as alternative_action next to the fallback.; default false
model_refused	string or null	no	Why the model's low-confidence recommendation is not offered (it failed a safety check: not applicable, step limits, position, geofence or no-go zone).
mower_id	string (uuid)	yes
options	array of OptionOut	no	While proposed: the options to choose from, the default first.
photo_problem	VisionPhotoProblem or null	no	The mower's camera delivered no photo, and why: weak_wifi or weak_mobile (with signal_dbm), offline or no_frame. With state failed and trigger error the automatic analysis could not run; otherwise the model saw only the map and uploaded images.
provider_family	string or null	yes
provider_model	string or null	yes
requested_by	PrincipalRef	yes
return_command_id	string (uuid) or null	yes	The command sent after approval: return_to_dock, or resume_mow for action resume.
round	integer	yes
rounds_left	integer	yes
signal_dbm	integer or null	no	With a weak signal: the mower's signal in dBm.
started_at	string (date-time) or null	yes
state	ProposalState	yes
steps	array of StepOut	yes
stereo	boolean	yes	The model saw a stereo pair: the left and right front camera, same moment.
task	VisionTask	yes
trigger	VisionTrigger	no	manual (a user asked) or error (started automatically when the mower entered a recoverable error; the requester is the user who opted in).; default "manual"
trigger_error_code	string or null	no	With trigger error: the error code that started it.
updated_at	string (date-time)	yes

ProposalPageOut

GET /mowers/{id}/proposals: newest first; pass next_before as before.

Field	Type	Required	Notes
items	array of ProposalOut	yes
next_before	string (date-time) or null	yes

ProposalState

Lifecycle of a proposal.

Type: "analysing" | "proposed" | "rejected_by_ai" | "failed" | "rejected" | "expired" | "executing" | "verifying" | "recovered" | "not_recovered" | "aborted"

ProviderIn

PUT /me/provider. api_key and headers are write-only: omit them to keep the stored values, send an empty string (or object) to delete them.

Field	Type	Required	Notes
api_base	string or null	no
api_key	string or null	no
enabled	boolean	no	default true
headers	map of string or null	no
model	string	yes	min length 3; max length 200

ProviderOut

GET /me/provider: the caller's provider (never the key) and the effective choice.

Field	Type	Required	Notes
api_base	string or null	yes
api_key_set	boolean	yes
configured	boolean	yes
effective	"own" | "hosted" | "none"	yes	Which provider a request would use now.
enabled	boolean	yes
family	string or null	yes
header_names	array of string	yes
hosted_available	boolean	yes
hosted_model	string or null	yes
last_test	LastTestOut or null	yes
model	string or null	yes
secrets_available	boolean	yes
updated_at	string (date-time) or null	yes

ProvidersOut

GET /providers: families, and whether the installation's model may be used.

Field	Type	Required	Notes
families	array of FamilyOut	yes
hosted_available	boolean	yes
hosted_model	string or null	yes
secrets_available	boolean	yes	Whether API keys can be stored here.

ProviderTestOut

POST /me/provider/test: one minimal vision call with a tiny generated image.

Field	Type	Required	Notes
cost_usd	number or null	yes	LiteLLM's estimate; null when unknown.
detail	string or null	yes
family	string	yes
input_tokens	integer or null	yes
latency_ms	integer	yes
model	string	yes
ok	boolean	yes
output_tokens	integer or null	yes

RecoverIn

POST /mowers/{id}/recover as JSON. Without images the service asks the mower's gateway for one (capture_photo) and otherwise analyses the context alone.

Field	Type	Required	Notes
images	array of ImageIn	no	max items 8
task	VisionTask	no	default "recover"

RetryIn

POST /proposals/{id}/retry: optional fresh snapshots.

Field	Type	Required	Notes
images	array of ImageIn	no	max items 8

RiskFlag

Hazards the model saw; any flag means the maneuver is never executed.

Type: "people_visible" | "animals_visible" | "drop_off" | "water" | "road"

StatusOut

GET /status: what the operator configured (never a key).

Field	Type	Required	Notes
allow_fake_provider	boolean	yes
hosted_default_enabled	boolean	yes
hosted_default_model	string or null	yes	Only when the default is enabled.
secrets_available	boolean	yes
user_api_base_policy	string	yes

StepOut

One drive step (blades are always off).

Field	Type	Required	Notes
angular	number	yes	Turn right (-1) to left (1).
detail	string or null	no
ended_at	string (date-time) or null	no
frames	integer	no	default 0
index	integer	yes
linear	number	yes	Backward (-1) to forward (1).
outcome	StepOutcome	yes
seconds	number	yes
started_at	string (date-time) or null	no

StepOutcome

What happened to one drive step.

Type: "pending" | "done" | "aborted" | "skipped"

ValidationError

Field	Type	Required	Notes
ctx	object	no
input	any	no
loc	array of string or integer	yes
msg	string	yes
type	string	yes

VisionPhotoProblem

Why an analysis got no camera photo from the mower (the most likely cause, judged from the mower's state at the time and the gateway's answer).

Type: "weak_wifi" | "weak_mobile" | "offline" | "no_frame"

VisionTask

What AI assist is asked to do.

Type: "recover" | "find_dock"

VisionTrigger

Who started an analysis.

Type: "manual" | "error"

---

Mirrored from docs/reference/api/vision.md at commit 4c3787f99a906065181b05e960a83f0aca64bbe4. Edit the file in the repository, not this wiki page.