GestureScreen v1.0.0

GestureScreen v1.0.0

Demo video

Play the complete demo in your browser · 1080p, 30 fps, 2:59

Download the high-quality MP4 (629 MiB). The same anonymized video is also in the main branch. Faces are pixelated, and camera location metadata was removed.

Core architecture

First public release of the offline gesture-controlled reader for the EmbedFire
STM32F429IGT6 Challenger V1, plus the Windows Gesture Studio capture tool.
Before using GestureScreen, it's recommended to use a collection tool to create
private training/validation/test sets for the user, then retrain and deploy
MobileNetV1 with the user's data — this will give better results.

Download

• GestureStudio.exe: self-contained Windows x64 application. Windows 11 was
  tested; Windows 10 remains an untested support target. SHA-256:
  37b710f9588a1a96ff9d27ff65e353177b31f50aa582852c6e4f49a8574bea56.
• The final firmware HEX with matching AXF/map, the six-class checkpoint, and
  source are in this tag. See SHA256SUMS for file hashes.

Final model and measured speed

The six-class MobileNetV1 0.25 × 96 RGB model was trained with supervised
labels. The frozen deployed int8 TFLite model scored 293/361 (81.16%) on
the owner's private validation split and 294/350 (84.00%) on the private
test split in offline top-1 classification.

With this final firmware on the STM32F429 board, 72 complete inferences took
153–182 ms each in wall time (median 158 ms, 95th percentile 176 ms).
Image preprocessing took another 25–40 ms.

Privacy and license

The dataset was recorded by the owner and contains personal imagery. No
dataset, raw frames, recording sessions, sample images, or annotations are
published. Original project work is MIT licensed; bundled third-party files
retain their own licenses. See PRIVACY.md and THIRD_PARTY.md.