Gesture-Controlled Room

Gesture-Controlled Room is a local Python 3.12 host for a deliberate, safety-oriented room-control interaction: hold an open palm to wake, point and dwell on a fixed target, return to neutral, perform one action gesture, and send exactly one named command to an ESP32 over USB serial. OpenCV supplies the interface, MediaPipe supplies landmarks, and replay plus a realistic mock transport allow development without hardware.

flowchart LR
  Camera[Webcam / video / synthetic] --> Tracker[MediaPipe adapter]
  Tracker --> Gesture[Gesture + swipe]
  Tracker --> Pointing[Pointing + target dwell]
  Gesture --> FSM[Pure interaction FSM]
  Pointing --> FSM
  FSM --> Router[Named-command router]
  Router --> Transport[Mock or USB serial]
  Transport -->|ACK| FSM
  FSM --> Logs[JSONL events + CSV trials]
  FSM --> UI[OpenCV overlay]

Loading

Safety boundary

Python never generates IR timing or drives an IR LED. It sends validated, named commands only. The ESP32 owns IR codes, carrier timing, transmission, safe idle, and acknowledgement. Commands start disabled unless explicitly enabled with space; replay cannot access serial. An ACK proves only that firmware reported an IR transmission—it cannot prove the appliance physically responded without manual annotation or a future feedback sensor.

Features

• Open-palm wake hold, stable ray/zone target selection, neutral separation, action, ACK, and cooldown
• Fist, pinch, open palm, neutral, swipe-left, and swipe-right heuristics with diagnostics
• Explicit, deterministic monotonic-time FSM and exactly-once command emission
• Webcam, video, and synthetic frame sources; mock and threaded USB serial transports
• Normalized polygon calibration, OpenCV overlay, JSONL events, raw CSV trials, replay, and reports
• Deterministic fixtures and tests for safety-critical geometry, timing, routing, protocol, and failures

Repository layout

src/gesture_room contains the runtime; configs contains editable examples; tests contains unit/integration fixtures; data preserves raw/log/report outputs; docs contains design and integration details; scripts contains convenience entry points.

Windows PowerShell setup

Python 3.12 and uv are required. Install uv using its official installer if it is not already on PATH, then:

git clone <your-repository-url>
cd gesture-controlled-room
uv sync --extra dev
uv run gesture-room create-config
.\scripts\download_hand_model.ps1

The command creates configs/app.yaml, configs/targets.yaml, and configs/commands.yaml without overwriting existing files. The download script retrieves Google's maintained Hand Landmarker task asset once; runtime remains local. Configuration errors identify the invalid field. GESTURE_ROOM_SERIAL_PORT and GESTURE_ROOM_LOG_LEVEL override their YAML values.

Run

First, run the complete hardware-free safety path:

uv run gesture-room demo --synthetic

Use a webcam with a mock ESP32:

uv run gesture-room --config configs/app.yaml --targets configs/targets.yaml --commands configs/commands.yaml run --transport mock

Use a video file with the current vision pipeline and mock transport:

uv run gesture-room run --transport mock --source video --video .\data\raw\session.mp4

Find the ESP32 port and run real serial transport:

uv run gesture-room list-ports
$env:GESTURE_ROOM_SERIAL_PORT = "COM3"
uv run gesture-room serial-smoke-test
uv run gesture-room run --transport serial

Automatic discovery succeeds only when exactly one likely USB serial device is present. The default retry count is zero because replaying a toggle after a lost ACK could reverse the intended physical state.

Calibration and mappings

Keep the camera fixed, then draw one polygon per target:

uv run gesture-room calibrate-zones --names lamp fan --output configs/targets.yaml

Left click adds a vertex, right click or u removes one, Enter finishes a polygon, and Esc cancels without changing the file. Existing files receive timestamped backups. See calibration.

Target polygons use normalized coordinates. configs/commands.yaml maps each target and gesture to an uppercase named firmware command; it contains no appliance IR data. Use interaction.selection_mode: zone when fingertip zones are more stable than pointing rays.

Keyboard controls

Key	Action
q	Quit cleanly
space	Enable/disable command transmission
Esc	Cancel the active interaction
r	Reset recoverable state
l	Toggle session recording status
s	Save a screenshot
h	Toggle overlay help
c	Calibration reminder (run the dedicated command)

Logs, replay, and reports

Event logs are JSONL under data/logs; every command creates a raw trial row. Unknown experimental fields remain blank for later annotation.

uv run gesture-room replay .\tests\fixtures\sample_logs\replay.jsonl
uv run gesture-room evaluate-gestures .\data\processed\labels.csv --output .\data\reports\gesture-summary.json
uv run gesture-room analyze .\tests\fixtures\sample_logs\session.jsonl --output .\data\reports\sample
uv run gesture-room false-activation .\data\logs\idle-session.jsonl

Replay is mock-only by construction. Reports retain raw rows, include sample counts, avoid averages alone, and generate charts only with adequate data.

Quality checks

uv sync --extra dev
uv run ruff check .
uv run ruff format --check .
uv run mypy src
uv run pytest
uv run pytest --cov=gesture_room --cov-report=term-missing

Troubleshooting

• Camera error: close other camera applications, verify camera.index, and check Windows privacy permissions.
• No hand: improve front lighting, keep the full hand visible, and inspect confidence on the overlay.
• Unstable selection: fix the camera, enlarge polygons, or switch from ray to zone mode.
• No serial ACK: run list-ports, verify 115200 baud and protocol version 1, then use serial-smoke-test.
• MediaPipe import or wheel error: verify python --version is 3.12 and rebuild with uv sync.

See failure modes for the full diagnostic table.

Current limitations

Rules are heuristic rather than learned probabilities; camera placement and individual hand geometry affect results. The first implementation tracks one deterministic primary hand, uses fixed camera-visible target polygons, has no Wi-Fi transport, and does not confirm physical appliance state. The ESP32 firmware is intentionally outside this repository; its exact contract is documented in ESP32 integration and serial protocol.