-
Notifications
You must be signed in to change notification settings - Fork 0
Home
Production-grade gesture input for Flutter canvases using hand tracking, built with a strong focus on architecture, reliability, latency, and user experience.
Gesture Canvas Input is a web-first Flutter plugin that transforms hand-tracking data into a clean, platform-agnostic canvas input system.
The goal is not simply to detect gestures. The goal is to create an input architecture where a canvas can be controlled by different input sources—mouse, touch, stylus, or hand tracking—without changing canvas logic.
This project is intentionally designed as a quality-focused engineering exercise. Every feature must meet production-grade standards for reliability, testing, failure handling, and maintainability.
- Convert hand gestures into canvas interactions.
- Maintain complete separation between gesture recognition and canvas behavior.
- Deliver predictable low-latency interaction.
- Provide deterministic testing without camera hardware.
- Offer a simple and intuitive Flutter API.
- General-purpose gesture recognition.
- Full-body tracking.
- Sign language interpretation.
- Mobile-native implementations (initially web-focused).
- AI model training or customization.
The architecture is built around a single abstraction:
CanvasInputSourceEvery input method communicates with the canvas through this boundary.
Examples:
- MouseInputSource
- GestureInputSource
- Future TouchInputSource
- Future StylusInputSource
The canvas never receives raw hand landmarks, MediaPipe objects, or camera-specific data.
Camera
↓
MediaPipe Hand Landmarker
↓
Gesture Interpreter
↓
GestureInputSource
↓
CanvasInputController
↓
Canvas WidgetsThis separation allows:
- Hardware-independent testing
- Input source replacement
- Replay-based debugging
- Cleaner API design
Every operation must have explicit success and failure behavior.
Examples:
- Camera permission denied
- Camera unavailable
- Model loading failure
- Tracking loss
- Inference interruption
No silent failures.
Users should always understand what the system is doing.
Examples:
- Hand detected
- Tracking active
- Hover state
- Pinch active
- Tracking lost
All gesture behavior must be testable without camera hardware.
Recorded landmark streams should be replayable through the input boundary.
The public API is considered a product.
Requirements:
- Safe defaults
- Discoverable configuration
- Sensible naming
- Minimal setup
- Comprehensive documentation
Hand-tracking implementation details must never leak beyond:
GestureInputSourceRaw landmarks should remain internal implementation details.
- Create Flutter package structure
- Configure CI
- Configure linting
- Create example application
- CI passes on every commit
- Zero analyzer warnings
- Standard package structure
- Known limitations documented
- Implement CanvasInputSource
- Implement MouseInputSource
- Build example canvas
- Validate architecture
- Canvas remains input-agnostic
- Immutable events
- Deterministic tests
- No resource leaks
- Camera integration
- MediaPipe hand detection
- Web Worker inference
- Latency measurement
- Main UI thread remains responsive
- Proper cleanup of camera resources
- User-visible error states
- Measured latency reporting
- Gesture state machine
- Hysteresis handling
- Smoothing filters
- Velocity prediction
- Replay testing
- No threshold chatter
- Stable drag interactions
- Graceful tracking recovery
- Debug visualization tools
- GestureInputSource integration
- Pinch-to-drag interaction
- Two-hand zoom support
- Visual cursor feedback
- No accidental activation
- Tracking quality monitoring
- Safe fallback to mouse input
- Clear interaction states
- Inclusivity testing
- Calibration workflow
- Documentation
- Package publication
- Cross-environment validation
- User calibration under 10 seconds
- Complete README
- Production example application
| Metric | Target |
|---|---|
| Inference FPS | 30+ FPS |
| Perceived Latency | < 50 ms |
| UI Thread Blocking | 0 ms |
| Camera Cleanup | Deterministic |
| Gesture Chatter | None |
Validate:
- Input controller behavior
- Gesture state transitions
- Hysteresis logic
- Replay fixtures
Validate:
- Camera lifecycle
- Source switching
- Resource cleanup
Recorded landmark sequences should reproduce bugs and edge cases deterministically.
Examples:
- Pinch while moving
- Tracking loss during drag
- Multiple hands appearing
- Rapid gesture changes
Current project limitations include:
- Web-first implementation
- Dependency on MediaPipe hand tracking
- Performance varies by hardware
- Tracking quality affected by lighting conditions
Additional limitations will be documented as the project evolves.
Before submitting a pull request:
- Run all tests.
- Pass static analysis.
- Add documentation for public APIs.
- Maintain boundary separation.
- Add tests for new behavior.
- Avoid introducing undefined states.
Contributors should prioritize reliability, maintainability, and user experience over feature count.
The objective of this project is not to implement every possible gesture.
The objective is to build a small set of interactions that feel reliable, predictable, and production-ready.
Quality is achieved not by adding features, but by ensuring every shipped feature behaves correctly under real-world conditions.