use case event camera vision

Spikeforge — UC-3: Event-Camera (DVS) Vision

A fully scoped production use case. Extends the reference slice in use_case_streaming_timeseries.md; consumes the shared enablers from production_toolkit_plan.md; listed in the umbrella production_use_cases.md.

One-line summary. Classify asynchronous event-camera streams (gesture, eye-tracking, automotive) window-by-window on GPU/CPU with no neuromorphic chip.

What it demonstrates. The strongest near-term SNN fit: the input is spikes, so no artificial encoding is needed — an event stream is accumulated into polarity rasters, consumed natively, and served through the stateful runtime.

Design/spec only. Every claim about current code is grounded in a file/line reference so a Code-mode agent can execute this file-by-file. No implementation starts in this document.

---

1. Problem and users

What. Given an asynchronous stream of (x, y, polarity, t) events, produce a class label per time window (e.g. which gesture) or a detection, with bounded latency and a sparse compute profile, on GPU training / CPU-or-GPU inference with no neuromorphic hardware.

Users. Robotics and automotive perception teams; HCI/gesture product teams; DVS researchers who already have recorded event data and want a served SNN.

Why SNN here. Event cameras emit sparse per-pixel changes; a conventional frame pipeline must first densify them into frames. An SNN consumes sparse events directly, so compute scales with activity, not sensor resolution.

Success criteria (SLAs).

• Quality: on DVS128 Gesture (11 classes) top-1 accuracy ≥ 0.90 and macro-F1 ≥ 0.88 with a compact SNN; on CIFAR10-DVS, accuracy above a dataset-specific floor defined at kickoff.
• Latency: p99 per-window decision under a configured budget (target: ≤ 10 ms on CPU for a small topology; GPU for the larger variant).
• Sparsity: reported input spike density and per-stage sparsity recorded per run.
• Zero hardware: recorded-stream replay on CPU/GPU; no DVS camera required.

---

2. Reference architecture

flowchart LR
    A[Recorded event stream] --> B[Event to polarity raster]
    B --> C[Time-window sampler]
    C --> D[Encoding contract]
    D --> E[Stateful InferenceSession]
    E --> F[Gesture or detection head]
    E --> G[DeploymentBundle]
    G --> E
    F --> H[spikeforge-serve predict and stream]
    H --> I[spikeforge-clients SDK]
    H --> J[Prometheus metrics]
    K[spikeforge-io replay harness] --> H
    L[Offline training] --> G

Loading

---

3. Offline: data, encoding, model, training

3.1 Data

• Sources: DVS128 Gesture and CIFAR10-DVS public event datasets (event_datasets.md), plus a deterministic synthetic event generator for CI (paralleling synthetic.py).
• Contract: events (x, y, p, t) are accumulated into [T, 2, H, W] polarity rasters over T slice windows with a frozen slice width; train-fitted normalization and the raster geometry live in preprocessing.json in the bundle.
• Repo fit: event loading and the polarity raster live in spikeforge/events/ (event_sample.py, event_bridge.py); replay belongs to spikeforge-io (replay.py).

3.2 Encoding

• Primary coding: identity / event-native — the raster is already binary, so the "encoder" is the raster builder; where a coded path is needed, rate over the slice is the documented fallback. A frozen EncodeSpec records whichever path is used so the bundle stays self-describing.
• Input shape: [T, 2, H, W], matching the event raster contract and the simulator's [T, …] loop.

3.3 Model

• Topology: dvs128_gesture preset for the small path; builder.py for a compact convolutional spiking net; declared once as a TopologySpec.
• Head: softmax classifier over gesture classes; a detection variant swaps the head for a per-window score.
• Reuse: TrainingEngine, surrogate gradients, BPTT over event slices (checkpoint_mixin.py).

3.4 Evaluation

• Top-1 accuracy / macro-F1 / per-class recall on the held-out event split.
• Event-native metrics: input spike density, per-stage spike sparsity, and parity of readout vs spikes through the simulator (default tolerances from rewrite_drift.py).
• Validation: NIR drift (within_tolerance) and determinism (determinism.py); manifest per run (manifest.py).

---

4. Online: bundle, runtime, service

4.1 Deployment bundle

• model.spkf with manifest.json, weights.pt, encode_config.json, preprocessing.json (raster geometry + slice width), graph.nir.json, checksums/signature.
• Anchors: bundle.py, bundle_manifest.py.

4.2 Stateful runtime

• InferenceSession.load(bundle), .reset(), .step(frame) -> Prediction, .run_stream(frames); carried membrane state via StateTree so a gesture can span windows.
• Shares the per-step body with the closed loop (step.py).

4.3 Service

• spikeforge-serve: POST /v1/predict, POST /v1/reset, GET|WS /v1/stream, GET /health, GET /metrics, GET /v1/bundle (app.py, service.py).
• Session ids per recording/camera; bounded batching; concurrency cap; auth; stream backpressure for continuous event feeds.

4.4 Clients and I/O

• spikeforge-clients SDKs drive predict/stream (client.py).
• A recorded-stream replay harness in spikeforge-io feeds a stored event file into /v1/stream deterministically (replay.py, adapters.py). A live DVS-camera adapter is optional (W7) and out of the MVP.

4.5 Observability

• Prometheus over the registry (registry.py): latency histogram, throughput, queue depth, per-stage spike sparsity, class score distribution.
• Serving benchmark p50/p99, throughput at concurrency N, cold start, peak memory, wired into the regression gate (compare.py).

4.6 Compression option

• Optional structured prune of spiking conv channels via pruning.py plus weight-only quantization (quantize.py); drift from PruningReport recorded in the bundle.

4.7 Test-deploy matrix row

• Test-deploy the compact conv spiking net on reference, norse, and lava_loihi2 (CPU emulator) with a parity report (test_deploy.py); estimate: true everywhere and available: false with a reason when an SDK is absent.

---

5. Phased delivery

Phase	Deliverable	Depends on	Acceptance
P0	Event dataset adapter + synthetic generator + frozen raster spec	—	raster reproducible; geometry frozen
P1	dvs128_gesture/conv topology trained; metrics + manifest	P0	accuracy ≥ 0.90 / macro-F1 ≥ 0.88; NIR validates
P2	InferenceSession over event slices + tests	W1	streaming readout == closed-loop run within tolerance
P3	DeploymentBundle export/import + tamper check	W1	fresh-process rebuild is exact; tamper is refused
P4	spikeforge-serve /predict + /stream + /reset	W2, W3	parity with in-process; state persists across slices
P5	/metrics, serving benchmark, sparsity + latency CI gate	W6	p99 within budget; sparsity recorded in CI
P6	Replay harness in CI, container, promotion/rollback, drift	W7	replay reproduces a stored recording exactly

MVP = P0–P4. That is the smallest end-to-end slice that demonstrates the chip-less DVS story.

---

6. Dependencies and out of scope

Depends on: PT-W1/W2 (released spikeforge/serving/ + frozen EncodeSpec); PT-W3 spikeforge-serve; PT-W6 observability + serving benchmarks; PT-W7 I/O adapters (replay; live-camera adapter optional). Tracked by umbrella issue #12; reference implementation UC-1 (released in spikeforge 0.3.0).

Out of scope: live DVS camera drivers and SDKs (optional W7); measured power (estimate: true); full temporal ONNX (NIR is the temporal graph); automotive safety certification.

7. Risks

Risk	Mitigation
Raster geometry mismatch train vs serve	freeze geometry in preprocessing.json; raster builder is the single call site
Large event datasets overwhelm CI	keep the synthetic generator as the CI fixture; use a subset for the public adapter
Sparse conv lowering past linear chains	start with reference+norse; broaden Lava lowering as W-track work
Latency vs accuracy trade-off	two presets (small/large) with per-preset budgets

8. GitHub issue payload

• Title: [UC-3] Event-camera (DVS) vision
• Labels: enhancement, architecture
• Body: see plans/use_case_event_camera_vision.md — goal, reference architecture (event raster → InferenceSession → gesture head → spikeforge-serve → clients/io → /metrics), event-native encoding, dvs128_gesture/conv topology, acceptance (DVS128 Gesture accuracy ≥ 0.90, macro-F1 ≥ 0.88, p99 ≤ 10 ms on CPU), MVP phases P0–P4, dependencies PT-W3/W6/W7.