Project Proposal

Project Proposal: SayHi — Gesture Control for Desktop

Project Overview: A camera-driven gesture layer for the desktop that lets users trigger and compose actions with their hands, addressing the problem that computer control assumes a keyboard and mouse within arm's reach, which excludes users with limited dexterity and breaks down in any hands-busy context.

1. Project Requirements

Functional Requirements

Recognition & Capture

• Capture camera frames at a configurable analysis rate.
• Detect hand landmarks (21 joints) and classify them into a defined gesture vocabulary.
• Distinguish left from right hand, treating each as a separate binding target.
• Report recognition confidence, and reject low-confidence or ambiguous poses rather than guessing.

Gesture–Action Mapping

• Create, edit, and delete bindings between a gesture-and-hand pair and an action.
• Support multiple action types: launch application, open URL, screenshot, keyboard shortcut, media control, hide application, minimise window, run system shortcut.
• Persist mappings across sessions, and migrate mapping files written by earlier versions.
• Restore factory defaults without destroying user configuration.

Macro Composition

• Compose an ordered sequence of steps (type text, press keys, wait, open app, media control) into a single bindable action.
• Reorder, edit, and delete individual steps.
• Validate required permissions up front, so a macro cannot fail part-way through.

Trigger Safety & Control

• Require a gesture to be held for a configurable duration before firing.
• Enforce a cooldown and a re-arm rule, so a sustained gesture fires exactly once.
• Tolerate brief detection dropouts without resetting the hold.
• Provide a global pause that suspends all actions while leaving recognition visible.

Feedback & Visibility

• Live camera view with detected hand skeleton, current gesture, and confidence.
• Floating hold-progress indicator and trigger confirmation, visible over other applications.
• Persistent status indicator showing active / paused / off.
• On-demand cheat sheet listing every gesture and its bound action, accessible by global hotkey.
• Debug view exposing per-finger geometry for threshold tuning.

Configuration

• User-adjustable timing (hold, cooldown, re-arm, dropout tolerance) and recognition thresholds.
• Configurable global hotkeys with conflict reporting.
• Live permission status with direct navigation to the relevant system settings.

Non-Functional Requirements

Security & Privacy

• All recognition performed on-device; no video, landmark, or usage data transmitted off the machine.
• Frames discarded immediately after analysis; no persistent storage of camera data.
• Explicit, revocable operating-system permission grants for camera, screen recording, and synthetic input.
• Stable code-signing identity, so permission grants are not silently invalidated between builds.

Performance & Reliability

• End-to-end input latency under 100 ms from gesture completion to action execution.
• Per-frame detection cost under 10 ms, so the analysis frame rate — not processing — bounds responsiveness.
• Zero unintended triggers during a five-minute session of ordinary hand movement and typing.
• Recognition degrades to an explicit "unknown" state rather than firing an incorrect action.

Usability

• Feedback surfaces remain visible across other applications, spaces, and full-screen contexts.
• Recognition tolerant of distance from the camera through hand-size normalisation.
• A new binding configurable in under 30 seconds without documentation.
• Overlay elements are click-through or repositionable, never obstructing the user's actual work.

Maintainability & Extensibility

• Adding a gesture requires one enum case and one classification rule; mapping UI, persistence, and feedback surfaces update automatically.
• Adding an action type requires one case and one executor branch.
• All recognition thresholds centralised in a single configuration type.

2. Initial Use Cases & Success Criteria

Use Case	Description	Success Criteria
UC1: First-Run Setup	A new user launches the app, enables gesture control, and grants camera access.	The permission prompt appears, the camera preview begins, and the user's hand is tracked with a visible skeleton overlay within five seconds of granting access.
UC2: Binding a Gesture	A user assigns "open palm, right hand" to launch a specific application.	The binding is written to persistent storage, appears in the mappings grid and the cheat sheet, and survives a restart of the application.
UC3: Hands-Free Triggering	A user working in another application performs a bound gesture without returning to the keyboard.	The floating hold indicator appears over the active application, the action executes on completion of the hold, and a confirmation identifies which action ran.
UC4: Macro Composition	A user builds a multi-step macro and binds it to a single gesture.	All steps execute in order in the correct target application; if a required permission is missing, the macro is refused before any step runs rather than failing midway.
UC5: Accidental-Trigger Prevention	A user gestures naturally while talking, then pauses gesture control to speak freely.	No bound action fires during ordinary hand movement; the pause gesture is still recognised while paused, and re-enables control without keyboard input.