RFC: Agentic Workflows for OpenNVR — design and implementation plan for review

[xpertvoip]

We are planning a new feature, Agentic Workflows, and want the community's review before we build it. This post is the full design and implementation plan. A PDF with the screen mockups is attached below. Comments on everything are welcome, but section 12 lists the specific questions we would most like answered.

Summary

A Workflows page where a site builds automations from a short vertical flow: a trigger, a few steps, an outcome. One step kind, AI, is where the judgement lives: it looks at a picture or investigates with tools, and returns a typed decision the flow branches on. Site rules, kept in one place, make every AI step accurate for this site. People and plain rules move things; AI informs and recommends. Everything runs on the box.

Concept	What it is
Workflow	The skeleton: trigger, steps, outcome. Three to seven steps, read top to bottom. Deterministic, versioned, every run recorded. Drawn in a visual builder, written as YAML, or drafted from a sentence.
AI step	The judgement. Look answers one question from a picture or text. Investigate works toward a goal with tools. Both return a decision from a list, fields, a reason and a confidence.
Site rules	The knowledge. "Cleaners come at 5 am." "Deliveries use the back gate." Written once, read by every AI step. Where most accuracy fixes go.

1. Why this shape

Before designing, we studied eleven products: automation builders (n8n, Dify), no-code agent builders (Zapier Agents, Lindy, Relevance AI, Microsoft Copilot Studio), developer frameworks (OpenAI AgentKit, LangGraph and Anthropic's "Building effective agents"), voice-agent platforms (Retell, Vapi), and the camera platforms we compete with (Coram AI, Spot AI, plus what Frigate and Home Assistant users build by hand).

They have converged on one shape, and the design follows it:

1. The agent lives inside a workflow, not instead of one. n8n, Dify, Copilot Studio, Zapier and AgentKit all put a deterministic flow around an agent node.
2. An agent is goal + instructions + tools + memory, plus a typed output so the next step can route on it.
3. Approval is set per tool, not per workflow. Read tools run freely; tools with outside impact pause for a person.
4. Standing rules belong to the site, not to one workflow (Lindy's "skills", Relevance's instructions).
5. Generic canvases die; opinionated builders live. OpenAI is retiring its free-form Agent Builder in November 2026 and Vapi retired its flow canvas, while narrow, domain-specific builders are thriving. So this is a short vertical flow with a small vocabulary for cameras and sites, not an n8n clone.
6. What fails in production is the invisible part: untraced tool calls, untracked prompt changes, no evals. So every run is a trace, every AI step is versioned, and "try it on recent events" is in the editor.
7. Voice is the same pattern over a phone line (Retell's conversation nodes with typed post-call outcomes).

2. The product

Workflows is a new item in the left pane, in the pinned NVR group after Cameras. A workflow has one trigger, then steps, then outcomes. AI, If, Ask a person and Call split the flow into one path per outcome. Paths never loop back. An Always section runs after whichever path ran.

Step	What it does	Outcomes	Comes from
Trigger	An app's alert; a contracted event such as a gate decision or plate read; a person or vehicle in a zone for N seconds; a camera going offline; a schedule; a button; a message to the camera agent	fires	apps, detector, core
AI	Look: a prompt over a picture, frames or text. Investigate: a goal pursued with tools	one per decision, plus unsure	core, local models
If	A plain rule: time window, site mode, camera, a value from an earlier step, membership of a list	true · false	core
For each	Repeats the steps inside it for each item: cameras in a group, plates in a list, events in a window	done	core
Ask a person	A question with reply buttons to named people or a role; waits; default on timeout	each reply · no answer	core, inbox and notifications
Call	A voice agent phones someone with a goal and script, returns typed outcomes and a transcript (phase 3)	each outcome · no answer	core, Twilio or SIP
Action	Alert, notify, log, save clip, PTZ preset, webhook, Home Assistant, site mode, an app's action such as open the gate, wait	done	core, apps

States. Off, Test (everything runs, actions are recorded as "would have"), On. New workflows start in Test. A workflow whose step lost its provider shows Needs attention and stops.

Limits on every workflow, with defaults: cooldown 1 minute per camera, one run at a time per camera (or per plate or track), 30 runs an hour, 60 AI calls an hour, chain depth 3. Repeat triggers while a run waits for a person join it as "seen again".

Three ways to author. Add steps in the visual builder (a vertical flow of cards with a side panel for the selected step; decisions fan out into one lane per outcome); write YAML; or type a sentence and let the camera agent draft the steps for review. Templates are YAML files in the repository with prerequisites the editor checks.

3. The AI step

One step, two modes, one output shape.

Look

• Prompt in plain language; click to insert {plate}, {camera} and other values from earlier steps.
• Input: the event's details, plus the camera picture now, a few seconds of frames (for movement), or text such as an intercom transcript.
• Runs the site's vision model through KAI-C for pictures, the local language model for text. One call, usually one to three seconds.
• Use it for most cases: is someone at the barrier, is this a delivery, is the door open, helmet and vest.

Investigate

• Goal: one sentence the agent works toward.
• Instructions for this step; Site rules are added automatically.
• Tools picked from the list below, each with its mode. The model chooses which to use and in what order, up to the limits.
• Memory: this run (default), or this camera's recent decisions, read-only.
• Limits: tool calls (default 6), seconds (45), and a share of the site's AI budget.
• Use it when one picture is not enough: who is this person, has this car been here, what happened before, why is the camera down.

Output, in both modes

• Decision: one of the options the author lists (yes / no; delivery / cab / visitor / unknown; ignore / watch / alert). The model is held to the list by schema-constrained output. Below the confidence floor the decision is unsure, which always has its own path.
• Fields, optional and named by the author: colour, company, who, flat. Available to later steps as {colour}.
• Reason: one line, shown to people and used in messages.
• Confidence: 0 to 1.

If the model is down, slow or over budget, the step answers unsure and the run continues. An AI step never crashes a workflow.

4. Tools

Tools are what Investigate can do. They are the camera agent's existing tools plus a few from core, each with a description written for a model and a mode set by site policy. Installing an app or model that declares tools adds them.

Tool	What it does	Default mode	Exists today
camera_snapshot, describe_camera	A fresh picture, described or answered against a question	auto	yes, camera agent
describe_event, describe_window	Describe a stored event's best frame, or a time window	auto	yes
detect_objects	What and how many a camera sees now	auto	yes
recent_events, recent_plates	What happened on this or nearby cameras recently	auto	yes
recognize_faces	Match against the face gallery	auto	yes; carries the face data scope
search_footage, search_history	Similar scenes or the same person across cameras; past events by text	auto	yes, semantic search
site_lookup	Read a Site rules list: expected visitors, contractors, staff hours, watchlist	auto	new
camera_health	Online state, last frame time, stream errors; a ping	auto	partly; ping is new
ask_person	A question with reply buttons; the run waits	ask	new
notify, alert	A message or an inbox alert, up to the severity cap	auto	yes
save_clip, ptz_preset	Protect footage; point a PTZ camera at a preset	auto	yes
speak, call	Play a message on a speaker; phone someone	ask	phase 3
open_gate, unlock, set_site_mode, siren	Moves things	not available to AI	as Actions only

Modes are a site policy: an admin sets each tool to auto, ask or not available once, on the Site rules page. A workflow may make a tool stricter for one step but never looser. An app action declared with confirm=True arrives as "not available to AI" and can only be placed as an Action after an Ask a person or an If on the same path.

5. A worked example

Night intrusion triage, as the builder shows it:

flowchart TB
  T["Trigger: person in the yard, 5 s<br/>yard-east, yard-west"] --> I["If: site armed, 22:00 to 06:00"]
  I --> A["AI, Investigate: is this a threat?<br/>tools: describe_camera, recent_events,<br/>site_lookup, search_footage, ask_person<br/>decision: ignore / watch / alert"]
  A -- ignore --> L["Action: log with reason"]
  A -- watch --> N["Action: notify guard, low"]
  A -- alert --> AL["Action: alert, medium<br/>clip + frames"]
  A -- unsure --> P["Ask a person: guard, threat?<br/>Yes / No, 60 s"]
  P -- Yes --> AH["Action: alert, high"]

Loading

The same workflow as YAML:

name: Night intrusion triage
state: test
limits: { cooldown: 2m, one_at_a_time: track, per_hour: 20, ai_per_hour: 40 }
when:
  detect.person_in_zone: { cameras: [yard-east, yard-west], min_seconds: 5 }
if:
  - site_mode: [armed_home, armed_away]
  - time: { from: "22:00", to: "06:00" }
then:
  - ai:
      mode: investigate
      goal: Decide whether the person in the yard is a threat, and who they probably are.
      instructions: Treat anyone near the fence line or the loading bay as suspicious.
      tools: [describe_camera, recent_events, site_lookup, search_footage, ask_person]
      decision: [ignore, watch, alert]
      fields: { who: [staff, contractor, unknown] }
      sure: 0.75
      limits: { tool_calls: 6, seconds: 45 }
      ignore: [ log: "{reason}" ]
      watch:  [ notify: { who: role:guard, severity: low, message: "{reason}" } ]
      alert:  [ alert: { severity: medium, title: "Intruder on {camera}", attach: [clip, frames] } ]
      unsure:
        - ask_person: { who: role:guard, message: "Possible intruder on {camera}. Threat?", replies: [Yes, No], timeout: 60s }
          Yes: [ alert: { severity: high, title: "Intruder on {camera}" } ]

A run of it, as the Runs view shows it:

Time	Step	Result
02:14:07	Person in the yard	yard-east, track 8812, 6 s
+0.0 s	If: site armed, 22:00 to 06:00	true
+2.1 s	describe_camera	"One adult walking along the fence line toward the loading bay, carrying a long object, no high-visibility clothing."
+2.2 s	site_lookup: staff hours	Nobody expected on site. Last contractor sign-out 19:40.
+2.4 s	recent_events	Loitering alert at the back lane, 01:58, same direction.
+5.9 s	search_footage	Same person at the back lane camera at 01:57 and 02:03.
+7.0 s	Decision: alert	who: unknown, confidence 0.88. "Unknown adult approaching the loading bay with an object at 2 am, seen earlier at the back lane; nobody is expected on site."
+7.1 s	Alert, medium	would have alerted (Test); clip and 3 frames attached; 4 tool calls, 6.9 s, within limits

Each run can be marked right or wrong, and the marks become the AI step's accuracy score.

6. Templates

Six ship with the feature, two more in phase 3. Each is named after its pattern and lists what it needs.

Template	Pattern	Flow	Needs
Night intrusion triage	investigate, then route	Person in yard at night → Investigate → log / notify / alert / ask the guard	detector, vision model
Gate concierge	look, then approve	Unknown plate at gate → Look: delivery / cab / visitor / unknown → Ask the guard → open on Let in	plate app with barrier mode, gate-controller app
Safety gear check	look, then act	Person enters work zone → Look at 3 frames: helmet and vest? → notify supervisor if no	vision model
Camera offline triage	investigate, not vision	Camera offline 2 min → Investigate: health, ping, history, power schedule → power / network / camera → webhook to ticketing	local language model
Morning site report	for each, then write	Every day 07:00 → For each camera group → Look at the collected text: one-page report → email	language model, email integration
Investigate on request	investigate, then verify	Message to the camera agent → Investigate across cameras → Look: does every claim cite a clip? → reply with the report	semantic search, camera agent
Resident call-back (phase 3)	look, then call	Visitor at lobby intercom → Look at transcript: which flat? → Call the resident → unlock on Let in	voice stack, resident list, speaker app
Talk-down (phase 3)	speak, then watch	Intrusion decided "alert" → speak a warning → Look at 30 s of frames: leaving / staying → escalate	speaker app

7. Accuracy and safety

AI informs and recommends. People and plain rules move things.

Opening, unlocking, arming and sirens are never on an AI step's tool list. They are Actions that may only follow an Ask a person, a Call, or an If on the same path. An AI-decided alert is capped at medium until the step has 80% accuracy over 20 marked runs, then high. Critical, which triggers phone calls through alarm actions, always needs a person or a plain rule. The builder enforces all of this and offers the fix in one click.

Accuracy

• Try on recent events before the workflow runs: pick up to 20 events from the last days and see each decision, reason and trace.
• Test mode for every new workflow and every new version of an AI step.
• Marks and a score: right or wrong on each run; the score is on the step, on the list and in the go-live dialog.
• Versions: every change to a prompt, goal, tool list or site fact is a version; each run records which decided; roll back is one click.
• Site rules fix most errors: a wrong decision that comes from missing knowledge is fixed with one sentence, for every workflow at once.
• Verify pattern: a second Look step checks the first's claims against evidence.
• "Describe it" is offered only when the site's language model passes 80% of the drafting cases in the camera agent's eval harness.

Safety

• All AI runs on the box. No picture, plate or face leaves the site.
• Tool modes are a site policy set by an admin; a step can tighten them, never loosen.
• A picture can mislead a vision model, and a sign in view can try to. The design accepts this: a wrong decision costs a wrong alert, never an open gate. Detector-based apps stay on for the same cameras.
• Approvals are their own records with their own permission (workflows.approve); acknowledge-all cannot clear them; no answer means the step's default, deny unless changed.
• Limits on every step and workflow; chain depth 3; a workflow ignores alerts it caused itself.
• Face and plate tools carry the same data scopes apps declare; turning On a workflow that uses them needs an admin; run history follows the recordings retention policy, 30 days by default.
• Test mode performs no action, sends no notification and makes no call.

Permissions: workflows.view (admin, operator, viewer), workflows.manage (admin, operator; workflows that neither move things nor read face or plate data), workflows.approve (admin, operator), and admin for Site rules, the tool policy and turning On anything that moves things or reads face or plate data.

8. Architecture

Part	Where	What
Frontend	app/src/views/Workflows*.tsx	Nav item in NAV_GROUPS; list, builder (Steps and YAML tabs), Runs, Versions, Site rules; approval card in AlertsIncidents.tsx. New deps: CodeMirror 6 with YAML, the yaml package; Vitest for tests.
Core: workflows	server/workflows/	registry.py (steps and tools from core, app manifests and model tasks), schema.py, validator.py (the safety rule, caps, needs), engine.py (NATS queue-group consumer, runs persisted step by step, waits as due-time rows, limits, chain depth); routers workflows.py, approvals.py, site_rules.py.
SDK: agent library	opennvr_app_sdk/agent/	LLM clients (Ollama, OpenAI-compatible) lifted from the camera agent, with schema-constrained output; the tool loop with traces, limits and typed results; tool descriptions. Used by core's engine and by the camera agent.
Models and actions	existing services	Vision through KAI-C and the enrichment governor, below live detection. Language model: Ollama from the camera-agent overlay; without it, AI steps are unavailable and rule-based workflows still run. Alerts on opennvr.alerts.workflow.*, so the inbox, siren, alarm actions and notifier apply. App actions through a new internal route that audits workflow, version and owner.

How a run works

1. The engine subscribes, as one NATS queue group, to the subjects its On and Test workflows need: opennvr.alerts.>, the contracted opennvr.events.> subjects, Tier-0 tracks for zone triggers, and core's camera connection events. Schedules are due-time rows.
2. A matching event checks the workflow's limits and creates a run row with a correlation id and chain. That id goes on every alert, notification and action the run causes.
3. Each step's inputs, result and timing are written before the next step starts. A restart resumes runs that were waiting. The queue group and row-locked claims make a second replica or a separate engine container safe later.
4. An AI step calls the SDK agent library: Look is one schema-constrained call; Investigate is the tool loop with the step's tool list, the site facts, memory and limits. Every tool call is a trace row.
5. Actions that move things write an intent row, act, then write the result. A crash in between marks the step unknown and alerts, rather than acting twice. They never retry automatically.
6. In Test, Action, Ask a person and Call steps write "would have" rows and perform nothing; AI steps and read tools run for real so the trace is genuine.

Tables: workflows, workflow_versions, workflow_runs, workflow_run_steps, workflow_tool_calls, workflow_approvals, site_rules.

Endpoints under /api/v1: GET workflows/registry, GET workflows/schema, GET|POST workflows, PUT workflows/{id}, POST workflows/{id}/state, POST workflows/{id}/try, GET workflows/{id}/runs, POST workflows/runs/{id}/mark, POST workflows/draft, GET|POST approvals, GET|PUT site-rules, POST internal/apps/{id}/actions/{name}.

9. Implementation plan

Three phases, each shippable. Estimates assume one backend engineer, one frontend engineer, and the camera-agent owner for the SDK agent library.

Phase 1, about 9 weeks: Workflows and Look

• RFC-0004 recording the decisions on this page.
• Step registry from core, app manifests (emits, actions, published events) and model tasks; YAML schema; validator with the safety rule, caps, needs, located errors.
• Engine: queue-group consumer, step-by-step persistence, waits and schedules as due-time rows, limits, chain depth, Test mode, Needs attention. Triggers: alerts, contracted events, zone presence from Tier-0, camera offline, schedule, button.
• Steps: If, For each, Ask a person (approvals record, inbox card, notification with link), Action (alert, notify, log, save clip, PTZ, webhook, site mode, app actions through the internal route), Always.
• AI step, Look mode: SDK LLM clients with schema-constrained output; pictures through KAI-C; the severity cap; try on recent events; marks.
• Site rules: facts and lists, with site_lookup readable by Look as input.
• Tables in one Alembic revision; three permissions with backfill; audit entries for state changes and approvals.
• Frontend: nav item; list; builder with Steps and YAML tabs; step forms generated from the registry; Runs with marks; Versions; Site rules page; approval card. Vitest added to CI.
• Templates: gate concierge, safety gear check, barrier fault to maintenance.

Done when a site owner installs Gate concierge from the template, tries the Look step on ten recent gate events, runs it in Test for a day, turns it On, and an operator lets a delivery in from the inbox, with every step visible in Runs.

Phase 2, about 7 weeks: Investigate

• Lift the camera agent's tool loop into opennvr_app_sdk.agent: tool registry with model-facing descriptions, traces, limits, typed results; the camera agent switches to it.
• Tool descriptions for the ten existing tools; new site_lookup, camera_health with ping, ask_person on the approvals record.
• AI step, Investigate mode: goal, instructions, tools, memory, limits; tool-call traces; tool policy enforcement; accuracy score and the trust gate for severity.
• Tools from apps and models through the registry; message-to-agent trigger; verify pattern; Describe it, with drafting cases in the eval harness.
• Frontend: Investigate panel; tool-call timeline in Runs; accuracy on list and step; go-live dialog; tool policy on Site rules.
• Templates: night intrusion triage, camera offline triage, morning site report, investigate on request.

Done when Night intrusion triage reaches 80% marked accuracy over 20 Test runs on a real site, every run shows its tool calls, and an agent over budget returns unsure without an error.

Phase 3, about 8 weeks: Voice and reach

• Call step: Twilio or SIP line; Whisper and Piper from the camera-agent stack; the local language model holding a scripted conversation with typed outcomes; transcripts on runs.
• Speaker app with a speak action; talk-down and resident call-back templates.
• Approve and refuse inside push messages with signed one-time links.
• App SDK: apps declare tools and alert fields; workflows validate and apply in the SDK command line.
• The engine as its own container for large sites.

Done when the resident call-back runs end to end on a test line, its transcript is on the run, and a third-party app's tools appear in the Investigate panel with no change to OpenNVR.

Testing

• Validator: a corpus of workflows that must be refused, including every route from an AI decision to a step that moves things.
• Engine: a test that stops the engine at every step boundary and checks each run completes once and no actuator fires twice; a flood of identical triggers yields one run and one approval; two workflows that trigger each other stop at depth 3.
• AI step: schema-constrained output against a fake model; unsure on timeout, error and budget; tool policy cannot be loosened by a step.
• Approvals: acknowledge-all leaves them; a viewer cannot answer; a user without the camera cannot answer.
• Templates: labelled events and a minimum Look accuracy checked before release; each survives Steps → YAML → Steps unchanged.
• Load: 200 cameras of synthetic events with 50 workflows on one core process.

10. Decisions made

Decision	Why
Workflows lead; chat is a trigger	Every product studied keeps the agent inside a flow. An investigation started from chat is a workflow whose trigger is a message.
One AI step, two modes	One panel to learn. Look covers most cases cheaply; Investigate is there when one picture is not enough.
Tool modes are a site policy	Safety set once by an admin; a step may tighten but never loosen.
Local models only	Sovereignty is the product. No cloud models.
AI informs; people and rules move things	A wrong decision costs a wrong alert, never an open gate.
Severity cap tied to measured accuracy	False positives must not phone people at night; trust is earned with marks.
Agent loop in the SDK	Core's engine and the camera agent share one implementation; apps can reuse it.
Engine inside core, replica-safe	Fewer moving parts now; splitting out later is a deployment change.
Approvals as their own records in the inbox	One place to look, with their own permission, uncleared by acknowledge-all.
Site rules as a page	What the no-code products call skills or instructions; where most accuracy fixes go.
Call in phase 3, contract in the schema from phase 1	The strongest demo, but it needs the voice stack and a speaker app.

11. Out of scope for version 1

• Cloud AI of any kind.
• Loops, code steps and an expression language. Logic beyond the seven step kinds is an app, and the app's alerts, actions and tools then appear in workflows.
• Workflows calling other workflows.
• Moving the camera agent's alarms, monitors and reports onto the engine. They keep working.

12. What we would like reviewed

1. The safety rule. "AI informs and recommends; people and plain rules move things." Does any site you run need the AI to open a gate or sound a siren on its own? If so, under what evidence?
2. One AI step with Look and Investigate modes, rather than separate steps. Is the mode switch clear, or would two steps be easier to learn?
3. Tool modes as a site policy set by an admin, which a step may tighten but never loosen. Does that match how you would want to run a site?
4. The six first templates. Which would you use, and what is missing? Camera offline triage and the morning report are there to show that workflows are not only about vision.
5. The visual builder is a vertical flow with branches, not a free-form canvas. Is that enough for the workflows you have in mind?
6. Small models. Look should work on small local models; Investigate needs a more capable one. What hardware do you run, and would a "Look only" site still be worth it to you?
7. Phase order. Call (a voice agent phoning someone with a script and typed outcomes) is in phase 3. Should it come before Investigate?

If you have a workflow you build by hand today, in Home Assistant, Node-RED or a script, please paste it. The templates list should come from real sites.

[xpertvoip]

Corrections after internal review

We reviewed the plan against the code after posting it. Twelve things were weaker than the text said; none change the shape of the feature, but four of them change what phase 1 and phase 2 contain. The post above has been left as it was so the thread stays readable; the attached PDF is updated.

#	Finding	Change
1	"Person in a zone for N seconds" was listed as a trigger, but zones and dwell exist only inside the intrusion and loitering apps; Tier-0 publishes per-frame tracks with no zones.	Phase 1 uses those apps' alerts as the trigger (emits makes them triggers for free). A native zone trigger is a separate phase 2 item with its own estimate.
2	The "message to the camera agent" trigger and "reply in chat" had no path: the camera agent is a separate container and core cannot push into a conversation.	Phase 2 adds an internal route for the agent to start a run and poll its result. The investigate template depends on it.
3	Lifting the camera agent's tool loop (about 8,000 lines, entangled with routing and turn tracing) into the SDK was the riskiest item with the thinnest estimate.	Phase 2 builds a new, small loop in the SDK for the engine. The camera agent switches to it in phase 3, once it has run on real sites. Named owner. Phase 2 is now about 9 weeks, not 7.
4	Investigate on CPU-only boxes would exceed the 45 s limit; the 7 s trace in the example is a GPU number.	Limits default by hardware class (6 calls / 45 s on GPU, 4 calls / 120 s on CPU). Investigate is offered on a site only when its model passes the harness cases; Look is offered everywhere.
5	Workflow webhooks from inside core would bypass the per-destination egress policy that apps are held to.	Webhooks go only to admin-approved destinations through the same policy.
6	Workflow vision calls share KAI-C's per-camera budget with apps and could starve a plate read on a busy gate.	A workflow priority at KAI-C, so apps keep their budget.
7	Camera scope was not checked at save time.	The validator rejects cameras the saver cannot see.
8	Chain depth is lost when an app re-alerts on a workflow alert.	Stated; the hourly cap and cooldown contain a loop through an app until the SDK passes the chain through in phase 3.
9	Unbounded Site rules facts would crowd a small model's context.	Facts capped at about 300 words; longer material goes in lists, which tools read on demand.
10	The severity cap stops alarm actions (default threshold high) but not notification channels, which push at any severity.	Stated; Test mode is the answer for new workflows.
11	What the ask_person tool does in Test mode was unstated.	Same as the step: records "would have asked", takes the default.
12	Who may mark runs right or wrong was unstated, though marks gate severity.	workflows.manage or workflows.approve.

Two questions this adds for reviewers:

• On 1: is a native "in zone for N seconds" trigger worth building at all, or is "use the intrusion app" good enough? The app already does zones, dwell, and per-camera tuning.
• On 4: for those on CPU-only hardware, would a Look-only site still be useful to you?

[Emart29]

Thanks for this. the idea of AI informs, people and plain rules move things line is the right foundation, and the post-review corrections show the care. Comments from the agent/eval side, grounded in what the harness measured on the camera agent (#616, #617, #618, and the monitor-threshold PR).

1. Small models report success they didn't achieve

Across the live runs, the failure that worried me most wasn't wrong tool choice, it was confident false reports: qwen2.5:1.5b replied "I'm watching for more than 3 people" over a watch with no threshold; llama3.2:1b said it was watching after the camera lookup had refused its call; qwen2.5:1.5b answered "hi" with "I see the front door. It's 14:55." So:

• Q1 (autonomous actuators): no. I'd keep actuators off AI steps entirely, as proposed. The model's own account of what happened isn't evidence.
• The step result should come from what ran, not what the model says ran. The run-step trace should record tool results and state read back, and the decision should be checked against them where possible — the same split the eval harness makes between "args as asked" and "state as stored".

2. Self-reported confidence from a 1–3B model isn't calibrated

The unsure path hangs on a 0–1 confidence the model reports. Small models don't produce meaningful self-reported confidence, so the floor will mostly fire at random. Options, cheapest first:

• derive it from evidence the step already has (detector scores, whether the tools it needed actually returned);
• sample the decision k times (k=3 on Look is cheap) and use agreement as the confidence;
• calibrate the floor per workflow from marked runs.

Happy to measure all three with the harness before we pick one.

3. Schema-constrained output fixes shape, not content

Constrained decoding will stop malformed JSON, but the failure we measured is valid-shaped output with the wrong or missing content: qwen2.5:1.5b left target out of create_alarm 14/15 times and max_count out of the crowd watch 4/5. Decision enums are the safe case (the model can only pick from the list). Named fields need per-field validation and, where the trigger already carries the answer, a deterministic backstop — which is what fixed the crowd case.

4. Investigate on CPU (Q6)

Measured on 4 vCPU, no GPU: qwen3:1.7b ~9 s per tool-calling turn, llama3.2:3b ~40 s. An Investigate with 4–5 tool calls is 40 s+ on CPU at best. More important: with the router and grounding guard off, models chose the right tool themselves far less often (qwen3:1.7b 10/45 before the fixes; only llama3.2:3b did most of the work itself, 30/45). Investigate removes exactly those backstops. So:

• Yes to a Look-only mode on CPU-only hardware; Investigate gated on an accelerator or a measured model.
• Keep each Investigate tool list small (≤5): fewer tools measurably improves selection in small models.
• Hardware-class limits (correction added cross platform docker-compose.yml file for ease #4) should come from measurement — the benchmark data in MODELS_AND_LATENCY.md is a start.

5. Accuracy gating needs an interval, not a ratio

"Accuracy target over 20 marked runs": 18/20 = 90% has a 95% Wilson interval of roughly 70–97%. I'd gate severity on the lower bound (the harness already reports Wilson intervals), otherwise a lucky 20 promotes an unreliable workflow.

6. Evaluation as a release gate

• Each template should ship with a labelled case set, run through the harness on every prompt/goal version change — a regression gate, not just one-click rollback after the fact.
• The 80% bar for "describe it" drafting should be measured the same way, with the validator as the hard gate on what gets saved.

7. Q2 (one AI step, two modes)

One step in the UI is fine. I'd keep them separate in traces, metrics and limits: their cost and failure profiles differ by an order of magnitude.

Where I can help

I'd like to own the AI-step side of Phase 2: the SDK tool loop (with by-style provenance, limits and per-call tracing, as in #613), the workflow eval suite and accuracy/confidence statistics, and hardware-class limits from measured runs. Phase 1's Look step and template case sets I can start on whenever you want.

[sipworld]

Hi @Emart29 Please own this and proceed. Thanks

[Emart29]

Read the full PDF and i am glad to see the harness as the gate for Investigate and Describe it, and the verify pattern already covers part of my point 1.

One data point on the CPU limits (4 calls / 120 s): on 4 vCPU, qwen3:1.7b runs ~9 s per model turn, so it fits; llama3.2:3b runs ~40 s, so 4 calls plus the answer is 160 s+. But llama3.2:3b is the model that chose tools well on its own (30/45 vs 10/45 with router and guard off), and Investigate has neither. So on CPU the model that can pass the Investigate cases likely can't meet the limit, and the one that meets it likely can't pass. I'd expect Investigate to be effectively GPU-only at launch, which makes Look-on-CPU the important path. I can confirm that once the Investigate cases exist.

On the 80%/20-runs gate: at exactly 16/20 the 95% interval is roughly 58–92%.

I'm happy to write the Investigate and drafting cases for the harness (phase 2), and they'd tell us early whether any CPU model qualifies.

[sipworld]

Hi @Emart29 Impressive, Please go ahead and work it out

[Emart29]

Design note: opennvr_app_sdk.agent — the AI step library

For RFC #620 (Agentic Workflows), Phase 2. Owner: @Emart29. Status: draft for review.

What this is

The library core's workflow engine calls to run an AI step, and that the camera agent switches to in Phase 3. One job: given a model, a question or goal, some inputs and some tools, return a typed decision with its evidence and a trace, inside hard limits, and never raise.

It is not the engine (runs, persistence, approvals, triggers stay in server/workflows/), and it holds no tool implementations — the caller passes tools in.

Goals

1. Look and Investigate behind one output shape (Decision), as the RFC specifies.
2. Safety that holds in the library, not only in the validator: nothing that moves things can ever be handed to the loop.
3. Every model call and tool call is a trace row the engine writes straight to workflow_tool_calls.
4. A run can pause on an ask tool and resume after a core restart.
5. Confidence the engine can gate on, which is not the model's self-report.
6. Measurable from day one: the eval harness drives the library directly.

Non-goals for Phase 2: the camera agent's router and forced grounding (they stay camera-agent features on top), streaming, cloud models, multi-agent anything.

Layout

opennvr_app_sdk/agent/
  llm.py        LLMClient protocol; OllamaChat, OpenAIChat (lifted from the camera agent, plus `format`)
  tools.py      Tool, ToolResult, ToolMode, effective_mode()
  decision.py   DecisionSpec -> JSON schema; Decision
  limits.py     Limits, hardware-class presets, Budget
  trace.py      TraceRow (LLM calls and tool calls)
  loop.py       run_look(), run_investigate(), resume()

No new dependencies: httpx is already an SDK dependency.

API sketch

class LLMClient(Protocol):
    async def chat(self, *, messages: list[dict], tools: list[dict] | None = None,
                   format: dict | None = None,          # JSON schema -> constrained output
                   temperature: float = 0.2, max_tokens: int = 256) -> ChatResult: ...

@dataclass(frozen=True)
class Tool:
    name: str
    description: str                 # written for a small model; see "Tool descriptions"
    parameters: dict                 # JSON schema
    handler: Callable[[dict], Awaitable[ToolResult]]
    mode: Literal["auto", "ask", "unavailable"] = "auto"
    effect: Literal["read", "ask_person"] = "read"   # there is no "act"

@dataclass(frozen=True)
class DecisionSpec:
    options: list[str]                        # e.g. ["ignore", "watch", "alert"]; "unsure" is implicit
    fields: dict[str, list[str] | type] = field(default_factory=dict)  # {"who": ["staff", "contractor", "unknown"]}
    floor: float = 0.75                       # below this, the decision becomes "unsure"

@dataclass
class Decision:
    decision: str                    # one of options, or "unsure"
    fields: dict[str, Any]
    reason: str                      # one line, shown to people
    confidence: float                # computed (see Confidence), what the floor applies to
    confidence_source: str           # "agreement" | "evidence" | ...
    self_reported: float | None      # the model's own number, kept for calibration only
    unsure_reason: str | None        # "below_floor" | "llm_down" | "over_budget" | "invalid_output" | ...
    evidence: list[str]              # trace row ids of the tool results it used
    trace: list[TraceRow]
    usage: Usage                     # model calls, tool calls, seconds, tokens

async def run_look(client, spec, *, prompt, inputs, images=(), limits) -> Decision
async def run_investigate(client, spec, *, goal, instructions, site_facts, tools,
                          inputs, memory=(), limits) -> Decision | Suspended
async def resume(client, state: Suspended, answer: ToolResult) -> Decision | Suspended

The Investigate loop

1. Tool phase. Chat with the step's tools until the model stops calling tools or a limit is hit. Every call goes through one gate:
   • validate the arguments against the tool's schema; a malformed call goes back to the model once as an error result, and a second one ends the tool phase;
   • apply the effective mode, stricter(site policy, step override): unavailable is refused and traced, ask suspends (below);
   • run the handler with the remaining time budget; a handler that raises becomes a failed ToolResult;
   • trace the call as the model asked and record whether it worked.
2. Decision phase. One more call with no tools and format = the JSON schema built from the DecisionSpec: enum decision, enum or typed fields, a capped reason. Separating it from the tool phase is deliberate: constrained decoding is reliable on its own, and combining it with tool calls isn't consistent across runtimes. Whether one combined call works with small models is measured, not assumed (open question 2).
3. Validation. Re-check the result against the spec even though the schema enforced it. A missing or invalid field makes the decision unsure (invalid_output), with no guessing.

Look is step 2 and 3 only, with the event's inputs and images.

Prompt-injection hygiene, as RFC §7 requires. Values such as {plate}, OCR text and captions go in a labelled JSON block in the user turn. They never go in the system instructions.

Pause and resume (ask tools)

An ask tool such as ask_person can wait minutes for a guard, and the RFC says waiting runs survive a restart. So the loop returns instead of blocking:

Suspended(state=..., pending_call=ToolCall("ask_person", {...}))

state is plain JSON: the messages so far, the trace, the usage, and the limits remaining. The engine persists it, creates the approval, and later calls resume(client, state, answer). Time spent waiting for a person doesn't count against the step's seconds limit. In Test mode, the engine answers with the step's default straight away, as the RFC specifies.

Confidence

The RFC's example floor is sure: 0.75 on a model-reported 0–1 number. In the camera-agent eval, 1–3B models reported success they hadn't achieved (a watch "for more than 3 people" with no threshold; "I see the front door" with no tool call). Their self-reported confidence isn't something to gate on. Proposal:

• The default is agreement. Run the decision phase k times on the same transcript and use the share that agree as the confidence. k=3 on GPU; on CPU, k=1 plus the evidence check below, because each extra call costs about 9 s with qwen3:1.7b.
• Evidence check. If a tool the decision depends on failed or never ran, confidence is capped below the floor. For example, an alert while describe_camera returned "unavailable".
• The self-reported number is still recorded (self_reported), so the eval can measure whether it carries any signal.

Before the default is fixed, the Investigate eval cases measure which source separates right from wrong decisions (AUROC per model). That data picks the default, not this note.

Limits and budget

Limits(tool_calls, seconds, model_calls, tokens, budget_share), with the RFC's hardware-class presets: GPU 6 calls / 45 s, CPU 4 calls / 120 s. Hitting a limit ends the tool phase. The decision phase still runs if its own time fits, otherwise the step returns unsure (over_budget). The site's hourly AI budget is a Budget object the engine owns and passes in; the library only draws from it.

Measured context (camera agent, 4 vCPU, #617): qwen3:1.7b takes about 9 s per model turn, so 4 calls fit in 120 s. llama3.2:3b takes about 40 s, so it doesn't fit, yet it's the only small model that chose tools well on its own. The Investigate cases will show whether any CPU model both fits and qualifies.

Trace

One TraceRow per model call and per tool call, shaped for workflow_tool_calls:

id, kind (llm | tool), name, args_asked, result (capped), ok | failed | refused | suspended,
mode, started_at, ms, tokens_in, tokens_out

This is the camera agent's #613 trace with one simplification: there's no router or forced grounding in a workflow step, so every tool call is the model's own. A refusal by policy and a hit limit are rows too, so a run explains itself.

Safety, enforced in the library

• Tool.effect has no "act" value. Constructing a tool that moves things is a TypeError, so an actuator can't reach an AI step even if the validator has a bug.
• Effective mode only ever tightens: effective_mode(site, step) is the stricter of the two.
• A call to a tool not on the step's list is refused and traced, never dispatched by name.
• Never raises. The LLM being down, a timeout, the budget running out or invalid output all become Decision(decision="unsure", unsure_reason=...), matching the RFC's "an AI step never crashes a workflow".

Tool descriptions

The Phase 2 item "descriptions for the ten existing tools" is model-facing text, so it gets the same treatment as the create_monitor fix: one-line purpose, when not to use it, and argument descriptions that say what a value is not (e.g. target is never a camera). Each description change is measured on the Investigate cases before it lands.

Evaluation (the other half of this work)

• Investigate cases in the harness. Each case is an event, the tools available, fixture tool results (recorded outputs, so only the model varies), the expected decision and fields, and the tools a good run needs. Fixtures make the cases deterministic on the tool side and cheap enough to run on all ten models. A live mode runs the same cases against real tools.
• Drafting cases for "Describe it": a sentence, the expected workflow, scored by the validator first and then by field-level match.
• The gate. "Passes the Investigate cases" is defined on the Wilson lower bound of the pass rate, not the raw ratio.
• Metrics: decision accuracy, needed tools called, unsure rate, limit hits, p50/p95 latency, and confidence calibration.

Testing

Unit tests with a scripted fake LLMClient:

• schema enforcement, and the invalid-output path to unsure;
• each limit;
• mode tightening and refusals;
• the "act" TypeError;
• suspend → JSON round-trip → resume, including across a fresh process;
• a handler that raises, hangs, or returns an error string;
• never raising on transport errors.

PR sequence

1. agent/llm.py: the LLM clients lifted from the camera agent, plus format. Phase 1's Look step needs this too, so it may land first and be shared (open question 4).
2. tools.py, decision.py, limits.py, trace.py, run_look().
3. run_investigate(), suspend/resume.
4. Investigate and drafting cases in the harness, with fixture mode, run on the small-model set.
5. The confidence study, then the default source and floor.

Each PR is Apache-2.0 under sdk/, with a DCO sign-off.

Open questions

1. Where do the tool handlers run for the engine? The ten tools live in the camera agent today. Does core get its own implementations (KAI-C, event store, search), or does it call the camera agent over the internal route? This decides what "tools from the registry" means for Phase 2.
2. Combined or separate decision call. A separate no-tools call with format is the safe default; it's worth measuring whether one combined call is as reliable on small models, since it saves a model turn on CPU.
3. k-sampling on CPU. Is k=3 affordable at ~9 s a call, or is evidence plus k=1 the CPU default?
4. Who builds the LLM clients for Phase 1 Look? Building them once, here, avoids two copies. I can take it if the Phase 1 backend owner prefers.
5. Does the seconds limit pause while an ask waits? This note assumes yes.
6. "This camera's recent decisions" memory: how many, and summarised or raw? It competes with site facts for the context of a small model.

[xpertvoip]

Decision: the workflow file is a public contract

One addition to the design before Phase 2 builds on it.

A workflow is a YAML file. The visual builder edits it, "Describe it" drafts it, and the validator checks it. That makes anything that can produce the file a builder, so I want the file format treated as a public, versioned contract, like the event contracts (workflow.v1, additive-only, a changed field is a new version), and published with the SDK.

What that buys:

• Import and export in the product. A workflow's page gets Download as YAML. Upload a workflow runs the validator, shows what the workflow would do and which cameras and skills it needs, and asks the operator to confirm before anything runs, because a workflow can trigger actions, so an imported one gets the same review as an installed app.
• Portability between sites. Cameras are referenced by role (gate, yard), never by id, so a workflow built for one site maps onto another. site_facts already points this way; this makes it a rule.
• Builders outside OpenNVR. A standalone builder on opennvr.org that runs the same validator in the browser and exports YAML, so anyone can try it before installing anything. An AI assistant writing the file from a description, which is the headless path with the skill definitions. An integrator's tooling generating a workflow per customer.
• Sharing. Once import exists, workflows can be shared like apps: a catalog with verified entries, or a file on GitHub.

Three things this asks of the Phase 2 design:

1. The validator is a library in the SDK (opennvr_app_sdk.workflows.validate), the same code the engine, the builder and an external tool all run.
2. The file carries its contract version, the skills it needs, and nothing site-specific beyond camera roles.
3. The engine's tools from the registry (your open question 1) resolve by skill name at import time, and a missing skill is a validation error with the name, not a runtime failure.

@Emart29 none of this changes the AI step library; it fixes what the engine imports and exports around it. Shout if any of it fights the design.

[Emart29]

Draft: the AI step schema and executor (Phase 1 Look, Phase 2 Investigate)

For review before I build it, so the engine core (registry, validator, persistence, routers) can be built against it. Scope is the AI step only: how it reads in the workflow file, what the validator checks, what the executor returns, and what gets stored. Triggers, paths and actions are the engine's; where I assume something about them it's marked (engine).

It follows the RFC's YAML (§5) and the decision above that the workflow file is a public contract (workflow.v1). Everything maps onto opennvr_app_sdk.agent as shipped in 0.10.0, plus rule tables (#679).

1. In the workflow file

The RFC's AI step, with one new key, rules:

then:
  - ai:
      mode: investigate
      goal: Decide whether the person in the yard is a threat.
      instructions: Guards on patrol are not intruders. Alert only when someone who should not be there is on site.
      tools: [describe_camera, recognize_faces, recent_events]   # read / ask only, at most 5
      decision: [ignore, watch, alert]           # outcomes; unsure is always added
      fields: { who: [staff, contractor, intruder, unknown] }   # the facts the model reports
      rules:                                      # new: first match wins, no match = unsure
        - { when: { who: intruder }, then: alert }
        - { when: { who: [staff, contractor] }, then: ignore }
        - { then: watch }
      sure: 0.75
      limits: { tool_calls: 6, seconds: 45 }
      ignore: [ log: "{reason}" ]
      watch:  [ notify: { who: role:guard, severity: low, message: "{reason}" } ]
      alert:  [ alert: { severity: medium, title: "Intruder on {camera}" } ]
      unsure: [ ask_person: { who: role:guard, message: "Possible intruder on {camera}. Threat?", replies: [Yes, No], timeout: 60s } ]

• With rules, the model reports fields (plus a reason and a confidence) and never the decision: the first matching rule gives it, and no match is unsure. On the 40 Investigate cases this didn't raise accuracy (feat(sdk): rule tables in AI steps, tested on held-out cases #679), but an outcome can no longer contradict the step's own facts, and the trace says which rule fired.
• Without rules, the model picks from decision, as the RFC has it today.
• Look uses prompt and its input (picture, frames or text) instead of goal, instructions and tools.
• limits default to the site's hardware class (GPU 6 calls / 45 s, CPU 4 / 120 s). sure is the confidence floor, applied to agreement across draws, not to the model's own number (it didn't separate right from wrong in feat(camera-agent): Investigate eval cases, and the first CPU results #649).
• Placeholders ({plate}, {camera}, {reason}) expand only into the step's data block, never its instructions (RFC §7).
• Site facts aren't in the step: they come from Site rules (≤300 words) at run time.

2. What the validator refuses

The AI step's checks already live in the SDK (DecisionSpec, Tool), so opennvr_app_sdk.workflows.validate can call them, and the engine, the builder and an external tool reject the same steps for the same reasons. Happy to write this part of the validator.

• mode missing, or fields that don't belong to it (tools on Look, a picture input on Investigate).
• A tool that isn't a registered skill, more than 5 tools, or any actuator. Tools resolve by skill name at import, so a missing one is a validation error with its name. The SDK refuses an actuator on its own too (Tool has no "act" effect), so this holds even if the validator has a bug.
• A rule naming an unknown field, a value that field can't take, or an outcome not in decision.
• An outcome without a path, unsure included (engine).
• A camera referenced by id rather than role, per the contract decision (engine).

3. The executor

run_ai_step(step, context) -> StepResult | Suspended. It builds the SDK DecisionSpec and tools from the step, calls run_look or run_investigate, and never raises: a model that's down, a step out of time, invalid output, or facts that match no rule all come back as unsure with the reason (llm_down, over_budget, invalid_output, no_rule).

An ask tool returns Suspended. The engine persists its state (plain JSON), raises the approval, and calls resume() with the answer. Waiting time isn't counted against the step's seconds. In Test mode the engine answers with the step's default straight away.

4. What gets stored

workflow_run_steps, one row per AI step run:
outcome, fields (json), reason, confidence, confidence_source, self_reported, unsure_reason, rule (index of the rule that fired, or null), model, usage (model calls, tool calls, seconds, tokens), state (only while suspended)

workflow_tool_calls, one row per model call and per tool call, the SDK's TraceRow.as_row() unchanged:
id, kind (llm | tool), name, args_asked, result (capped), status (ok | failed | refused | suspended), mode, started_at, ms, tokens_in, tokens_out

(rule needs one small SDK addition: Decision doesn't record which rule fired yet.)

5. When a step is offered

• Investigate: only when the site's model passes the 40 Investigate cases (seen + held-out) at a Wilson lower bound ≥ 0.80 on its hardware class. Today no CPU model does: granite4:micro is 29/60 on the held-out cases (feat(sdk): rule tables in AI steps, tested on held-out cases #679).
• Look: the gate follows from Look cases on real images (the consented clips).

Open questions

1. Rules in Look: required, as in Investigate, or optional? I'd make them optional for Look, since a single "who is it" question often needs no rule, and required for Investigate.
2. Hardware class: read from GET /api/v1/system/hardware at run time, or fixed per site in settings?
3. Severity: I've assumed the severity cap lives on the path's alert action, not on the AI step. Right?
4. fields naming: the RFC uses fields for both "extra values for later steps" and, with rules, "the facts the rules read". One key for both, or should rule inputs be marked?