Tiled detection: measured performance, comparison with Frigate, and a camera-side proposal

[johnchia]

Sharing a working document from running this service against a 6-camera LightNVR
deployment, in case any of it is useful to others or worth pursuing upstream.

The implemented part is #8 (fixed-budget tiled detection). Everything from section 3
onward is analysis and a proposal, not code — posting it here rather than as an issue
because the interesting parts are architectural and I'd value being argued with,
particularly on the Frigate comparison and the camera-side proposal.

Measurements throughout are from an Intel i5-6500 / HD Graphics 530 with the OpenVINO
execution provider (#6), on 5 MP JPEG frames. Anything I inferred rather than measured
is flagged as such.

Contents

1. Fixed-budget tiled detection — how it works, measured grid sizes
2. Measured performance — per-request costs, 6-camera capacity, a baseline correction
3. Honest comparison with Frigate — where this loses, and where it doesn't
4. Proposal: push region selection to the camera
5. Running on EdgeTPU / Cortex-A72
6. Open items

---

1. Where we are: fixed-budget tiled detection

The original problem was resolution loss. A frame was downscaled twice before reaching the
model — once by the detection service (MAX_IMAGE_SIZE = 1024) and again by the backend's
letterbox to 640. Net linear scale on 1080p was 0.333; a 60 px person arrived at the model
as 20 px, right at the floor where YOLO stops firing.

#8 addresses this two ways:

1. Removed the 1024 px pre-resize from the /v1/detect path. The backend letterboxes
   anyway, so that resample was pure loss. This alone improved every request, tiled or not.
2. Fixed-budget tiling. Per request, run exactly T inferences. Content never changes
   T — it only changes which T regions get inspected.

How the tiling works

The grid is derived from min_object_px, not a fixed layout. The model stops firing
below ~24 px, so crop size is whatever makes the smallest interesting object arrive at 24 px:

crop size = model_input ÷ (MIN_MODEL_PX / min_object_px)
          = 640 ÷ (24/60) = 1600 px

Crops are laid on a 25 %-overlap stride with the last row/column flushed to the frame edge.
Tile 0 is always the full frame, so large and near objects are caught every cycle and
behaviour never regresses. The remaining T-1 rotate over the pool on a clock-derived
cursor — stateless, no per-stream server state, multi-worker safe:

cursor = int(now // tile_period) * n_rot % len(pool)

Measured grid sizes (from utils/tiling.py, min_object_px=60, 640 model):

Resolution	Tiles (K)	Crop size	Full-frame scale	Via a crop	Sweep at T=4
1080p	3	1600×1080	0.33× (60 px → 20 px)	0.40× (→ 24 px)	gapless, rotation never engages
5 MP	5	1600×1600	0.25× (60 px → 15 px)	0.40× (→ 24 px)	3 of 4 crops/sec, 2-cycle sweep
4K	7	1600×1600	0.17×	0.40×	2-cycle sweep

select_tiles caps n_rot at the pool size, so passing tiles=8 to a 1080p camera costs
nothing — it still issues 3.

Lowering min_object_px increases magnification and tile count: 5 MP at 40 px is 10 tiles,
at 24 px (native scale) it is 25 tiles and an 8-second sweep.

Known limitation: tile_period aliasing

The cursor is linear in the clock, so a caller whose period is an exact multiple of
tile_period samples the same residues forever — permanent blind spots, not a slower
sweep. Measured with a 4-tile pool and T=3: a caller at 2 s against tile_period=1
selects tiles 1 and 2 on every cycle; tiles 3 and 4 are never visited.

This matters because lightNVR's keyframe-gated fallback path fires at the GOP length, not at
detection_interval. tile_period must be set to the effective firing rate. Pinned by
test_commensurate_period_mismatch_starves_tiles.

---

2. Measured performance

Benchmarked on the PoC host — Intel i5-6500 (4 cores, no HT) / HD Graphics 530, OpenVINO EP
on GPU. The production host is newer, so these are a floor.

Important calibration: the 35 ms figure quoted in the original plan was measured against
test_image.jpg, which is 640×480. That fixture decodes in ~2 ms. Real 5 MP frames are
far more expensive, and the 35 ms number understated production cost badly.

CPU-side cost of one 5 MP request:

Stage	Cost
JPEG decode (2592×1944, q85)	71.7 ms
Letterbox full frame → 640	25.9 ms
Crop one tile (memcpy)	2.1 ms
Crop + letterbox one tile	19.3 ms
(the removed 1024 pre-resize, for reference)	84.9 ms

Backing the decode out of that 35 ms figure puts inference at ~30 ms on the HD 530.

Concurrency

Two findings that matter:

• The JPEG decode is lazy and lands inside the worker thread. Image.open() reads only
  the header; preprocess_image does not force a decode when no resize is needed. Tile 0 is
  the full frame, so the first thing to touch pixels is detect() — already inside
  asyncio.to_thread. The event loop only pays ~2 ms per crop. The architecture is sound.
• PIL releases the GIL during decode and resize, so CPU work parallelizes across cores.

Six cameras firing on the same tick (measured, 4 cores, ThreadPoolExecutor):

T	Solo CPU	6-camera burst wall-clock
1	86.7 ms	155.7 ms
2	123.0 ms	275.4 ms
4	151.6 ms	394.9 ms
6	172.6 ms	371.5 ms

CPU is not the constraint. The iGPU is — it is one serialized resource:

T	Inferences/s (6 cam @ 1 Hz)	iGPU duty @ 30 ms	Verdict
1	6	~18 %	lots of room
2	12	~36 %	comfortable
3	18	~54 %	fine
4	24	~72 %	works, little margin
5	30	~90 %	no margin for a slow frame
6	36	~108 %	queue grows without bound

Past 100 % the failure is not graceful: work arrives faster than it drains, the backlog grows
monotonically, and the 10 s client timeout (API_DETECTION_TIMEOUT_SECONDS, not
configurable) is hit within ~15–20 s of sustained load. The 7 s deadline limiter prevents
total collapse but does so by silently dropping tiles — coverage degrades while the logs look
healthy.

Recommended configuration

T is a per-request URL parameter, therefore per-camera. Do not set it globally.

close-range cameras   ...&tiles=1
distant-view cameras  ...&tiles=4&min_object_px=60&tile_period=1

Close-range cameras still benefit at tiles=1, because the 1024 px downscale was
removed regardless of tiling. Three tiled at T=4 plus three untiled is 15 inferences/s —
~45 % duty, roughly half the load of T=4 everywhere with nearly all the benefit.

Class filtering is already supported and costs nothing: &filter_classes=person,car will
suppress the static TV/light detections that otherwise re-fire every second. zones,
min_width and min_height also exist and run after the merge, so they compose correctly
with tiling.

To pin the true marginal cost per tile on the production host:

for t in 1 2 4 6; do
  curl -s -o /dev/null -w "tiles=$t  %{time_total}s\n" \
    -X POST "http://localhost:8000/api/v1/detect?backend=onnx&tiles=$t&min_object_px=60" \
    -F "file=@<a-real-5MP-frame>.jpg"
done

Subtract the T=1 time from the T=4 time and divide by 3.

---

3. Honest comparison with Frigate

Worth stating plainly, because it defines what the next architecture should target.

Frigate is motion-gated and adaptive: cheap CPU motion detection on every frame of a
low-res detect stream, motion contours grouped into regions, a detector-sized crop taken
around the motion, then object tracking with re-detection around each track.

	Frigate	This design
Where inferences land	on changed pixels	on a blind clock rotation
Effective frame rate	~5 fps+	1 Hz (lightNVR's integer-second gate)
Latency to first detect	same frame	up to a few seconds (rotation)
Tracking / N-hit confirmation	yes	none — every request independent
Cost under heavy scene activity	scales with motion; can saturate	flat by construction
Server state	stateful tracker per camera	stateless
Integration	replaces the NVR	drop-in HTTP service

Frigate would detect more, sooner, with fewer wasted inferences. Motion gating is simply
a better prior than blind rotation, and the ~5× frame-rate advantage compounds it.

What this design wins is the worst case rather than the average one. Frigate's compute is
scene-dependent — wind, rain, night-time headlights and insect swarms can saturate the
detector and cause dropped frames, which is why so much Frigate tuning effort goes into
motion masks. Fixed-budget tiling cannot be overwhelmed by scene activity. It is also
stateless, restart-safe and horizontally scalable, and it leaves lightNVR owning recording,
storage and UI.

A large part of the gap is transport, not algorithm. Frigate runs motion detection on an
already-decoded frame in shared memory. We pay a JPEG encode on the lightNVR side, an HTTP
round trip, and a 72 ms decode on ours — more than double the inference cost — per request.
Even a perfect tile-selection strategy is capped by receiving one frame per second over HTTP.

---

4. Proposal: push selection to the camera

The core observation

The camera has already computed the motion field and is throwing it away. H.264/H.265
inter-prediction evaluates a motion vector and residual cost for every macroblock. That
search is the most expensive part of encoding, it runs on dedicated silicon in the camera
SoC, and the results are discarded into the bitstream.

Firmware access exposes the encoder's internal state, which is richer than the vectors alone:

• Skip flags — blocks proven identical to the reference. A certified "nothing happened
  here", and the most useful signal because it eliminates rather than ranks.
• Intra-coded blocks inside a P-frame — prediction failed; something appeared that was
  not there before. Strong novelty signal.
• Residual magnitude (SAD/SATD) per block — a free change-energy map.

The ISP adds a second free layer: per-zone 3A statistics (luminance histograms, per-region
sharpness) computed for auto-exposure and auto-focus.

The bigger win is transport, not signal

Today the camera encodes a full 5 MP frame, ships it, and we spend 71.7 ms decoding it to
look at perhaps 5 % of the pixels. If the camera sends only the candidate crop at native
resolution, pre-sized to the detector input, that deletes the full-frame decode, the resize
work and most of the bandwidth. Per-request server cost drops from ~150 ms to roughly 30 ms
of nearly pure inference.

It also dissolves the 1 Hz ceiling, which exists only because lightNVR polls on an
integer-second gate. A camera that pushes on change is not polling at all.

Why this scales better than either alternative

Motion analysis is embarrassingly parallel across cameras, and every camera ships with its own
SoC. Six cameras is six free motion engines. Centralising that work on one server puts it on
the single resource that does not scale with camera count. Push selection to the edge, keep
only semantic classification central, and server load stops growing with resolution and starts
growing only with the number of interesting events.

Shape

Camera SoC
  ├─ encoder (already running) ──► skip flags / MV / residual map   [free]
  ├─ region builder: group blocks → dilate → pad to min size → rate limit
  └─ POST crop + metadata ──────────────┐
                                        │  keep-alive HTTP or MQTT
Detect service (stateless)  ◄───────────┘
  ├─ admission control: per-camera token bucket, max in-flight, drop-oldest
  ├─ batch collector: coalesce crops arriving within ~50 ms
  ├─ ONNX batch inference
  ├─ map crop-normalized boxes → frame-normalized via crop_rect
  └─ emit event ──► MQTT / webhook ──► lightNVR ingest
                                        │
  ▲ safety net: low-rate full-frame tile sweep (the existing pass-1 design)

The decisions that determine whether it works

Correlate on PTS, not wall clock. A detection at "timestamp T" is only useful if lightNVR
can tie it to a recording segment. Camera-clock versus NVR-clock drift will put boxes a few
hundred milliseconds off, which presents as a coordinate bug and gets debugged as one. Carry
the RTP/stream presentation timestamp through as the correlation key; treat wall clock as
human-readable decoration.

Metadata needs crop_rect and frame_dims. Not only to map boxes back — without frame
dimensions you cannot reason about real-world size, and min_width/min_height stop meaning
anything. A tight crop around a distant person, upscaled to 640, is pixel-wise
indistinguishable from a near person; that is exactly how a windblown leaf gets promoted to a
person-sized object.

Pad the crop; do not send the raw motion blob. A walking person fragments into several
motion blobs (arms, legs, torso against different backgrounds), so per-blob crops yield
fragments of people. Detectors trained on full scenes also degrade on context-free tight
crops. Group the blocks, dilate, then snap to at least detector-input size with margin.

Admission control is not optional. Once the camera drives the request rate, a windy tree
can DoS the detect service — precisely the failure mode fixed-budget tiling was built to
eliminate. A per-camera token bucket with drop-oldest keeps the bounded-cost property while
letting the camera decide which pixels get the budget. Content chooses the regions; policy
still caps the spend.

Batching is new headroom. Crops arrive pre-sized to model input, so coalescing everything
within ~50 ms into one batched ONNX call is natural, and batch inference on the iGPU is
materially better than N sequential invocations. Tiles today are sequential awaits and
cannot exploit this.

Keep the tile sweep as a fallback. Camera motion detection will miss things — slow
approach, objects entering during an auto-exposure step, anything already in frame at start.
Running the pass-1 rotating sweep on a slow timer (5–10 s) is a fixed-cost safety net beneath
the event-driven path. The two compose cleanly and no pass-1 work is wasted.

Known costs and risks

The last hop is where "no lightNVR changes" dies. Today lightNVR pulls: it decodes, POSTs,
and consumes the response synchronously inside its own thread. Here results arrive
asynchronously and out of band, and lightNVR has nowhere to receive them. An MQTT subscriber
or ingest endpoint plus correlation logic is the largest single work item in the architecture,
and it is in the codebase we have so far avoided touching.

Migration path that defers it: have the detect service keep a short result cache keyed by
(stream, pts window). Cameras push crops and the service detects continuously; lightNVR
keeps polling the existing endpoint and the service answers from cache instead of running
inference on the posted frame. This wastes the uploaded frame's bandwidth and inherits the
1 Hz ceiling, so it is not the destination — but it validates the entire camera-side and
service-side pipeline before touching lightNVR, and allows a clean fallback if camera-side
motion proves noisy.

Tracking forces state somewhere. Raw per-crop detections mean one event per frame per
object; something must collapse "same person, forty consecutive frames" into one event with a
start and end. That is the point at which per-stream state becomes unavoidable and the detect
service stops being stateless. Decide deliberately whether it lives in the detect service or
in lightNVR — doing it accidentally is how it ends up in both.

Encoder motion vectors are not optical flow. They are chosen to minimise bitrate, not to
describe motion, so in flat or textureless regions they are close to arbitrary, and sensor
noise in low light generates spurious ones. Auto-exposure steps, IR-cut filter switching and
camera shake produce whole-frame vector storms indistinguishable from real motion. Use them
as a cheap elimination filter ("these blocks are provably static") rather than as a detector.

Saliency proper is not free. The ISP does not compute it and the encoder does not
approximate it. It would mean running a small network on the camera's NPU — a real option on
most modern SoCs, but relocated compute rather than free compute.

Firmware modification is a significant practical lift. Cameras are locked, SDKs are
vendor-specific per SoC family (Hisilicon, Ambarella, Novatek), and forked firmware must be
maintained across models. It realistically only pencils out on a standardised camera model or
an open platform such as OpenIPC.

The version that needs no firmware at all

Most of the benefit is available today:

• ffmpeg exposes H.264 motion vectors during decode (AV_FRAME_DATA_MOTION_VECTORS), so
  lightNVR could extract a change map as a byproduct of decoding it already performs.
• Most cameras publish built-in VMD regions over ONVIF as a metadata stream, with no
  firmware access required.

Either would feed a real prior into the deferred change-priority work without forcing
per-stream state onto the detection server, and without a firmware programme.

---

5. Running on EdgeTPU / Cortex-A72

Sketched for a possible low-power deployment at 1 MP. Viable, but the constraint inverts.

The tiling layer is already backend-agnostic — it calls detector.detect() and never touches
backend internals. The edgetpu backend returns boxes normalised to its own input and its
preprocess_image does a plain resize(), which is linearly invertible per axis, so
normalised coordinates survive it. The geometry works on edgetpu unchanged.

The bottleneck moves from accelerator to CPU. Inference is ~10 ms and effectively free;
everything between inferences (decode, crop, resize) is CPU work, and an A72 core is roughly
5–10× slower than the i5 at pixel shuffling. Total CPU pixel work is approximately
decode(full frame) + Σ resize(crop area → model input) — at 25 % overlap the crops sum to
~1.7× frame area, so it is decode plus ~2 frames of resampling, near-constant regardless of
tile count. Tile count barely moves CPU cost, so T can be generous.

The need for tiling is unchanged, which is not obvious. What matters is the ratio of frame
to model input:

• 5 MP into a 640 YOLO: 2592/640 = 4.05× downscale
• 1 MP into a 320 EdgeTPU model: 1280/320 = 4.0× downscale

A 60 px object arrives at ~15 px either way.

What has to change:

• MIN_MODEL_PX = 24 is a YOLO assumption. SSD MobileNet's feature pyramid is far coarser;
  40–48 px is realistic. This must become a per-backend property, and it roughly triples tile
  count (~9 tiles on 1 MP) — acceptable, since tiles are cheap here.
• Aspect distortion stops being cosmetic. Squashing a 16:9 full frame into a square 320
  input deforms objects ~1.8×, and int8-quantised models are brittle where fp16 YOLO is not.
  Crops are square-ish and fine; tile 0 is the problem. This is where the deferred
  utils/letterbox.py extraction finally gains a real consumer.
• Threading changes. ORT sessions are thread-safe so the current code fires concurrent
  detect() calls freely. A TFLite Interpreter is not thread-safe, and one EdgeTPU is
  one serialised device — a single-worker queue or a lock around invoke is required.
• Scaled JPEG decode becomes worthwhile. libjpeg can decode at 1/2, 1/4 or 1/8 scale
  almost free by truncating the DCT. Tile 0 never needs full resolution. On an A72 this is
  likely the difference between comfortable and marginal.

Rough sizing: ~40–60 ms decode + ~30–50 ms resampling + 9 × 10 ms inference + Python
overhead ≈ 200–250 ms CPU per camera-request. At 1 Hz that is ~25 % of one A72 core, so a
quad-A72 should carry 6–10 cameras. The EdgeTPU is nowhere near its limit: 100 fps ÷ 9
tiles ≈ 11 full camera-requests/second.

The real risk is the detector, not the compute. SSD MobileNet at 300×300 is substantially
weaker than YOLO at 640, and tiling multiplies resolution but cannot add capability the
network lacks. EfficientDet-Lite or an int8 YOLOv8n EdgeTPU build would be needed, or the
magnification feeds a model that still cannot see what it is pointed at.

---

6. Open items

Still unverified on the shipped work: the actual per-stream firing rate — a stream
silently running at 0.5 Hz doubles every sweep time in section 1, and the keyframe-gated
path makes that easy to hit without noticing. Also worth confirming go2rtc is not serving a
substream to detect_objects_api_snapshot, since tiling a frame whose detail was already
discarded upstream invalidates the whole exercise. Both are config checks, not code.

What I would build next, in order:

1. Debug tile overlays on the return_image path (2–3 h). Tuning is currently blind —
   you can see that detections improved but not which tiles ran, where the seams fell, or
   whether an object sat in a tile that was skipped. This is a prerequisite for choosing
   ROI bands sensibly rather than by guesswork.
2. ROI / far-field bands (4–6 h). For a fixed camera, apparent object size varies with
   image row. Tiling only the horizon band collapses a 24-tile native-scale grid to ~6 and
   cuts the sweep from 8 s to 2 s, which is what makes min_object_px=24 practical rather
   than theoretical. Per-stream static config, so it adds no server state.
3. Docs for the per-stream URL recipe.

Change-priority — spending the budget where pixels moved — I would deprioritise, despite
it being the obvious next idea. It only pays off when the tile pool is large relative to T,
and ROI bands attack that same problem by shrinking the pool rather than scheduling it more
cleverly, without per-stream state, eviction or restart fragility. And if anything like the
camera-side proposal in section 4 happens, change-priority is obviated entirely — the camera
becomes the priority signal.

Extracting the ONNX letterbox into a shared util has no consumer while the deployment is
ONNX-only. It gains one under the EdgeTPU path in section 5, and not before.