When will gesture detection be launched on production platforms

[bugensui2022]

When will gesture detection be launched on production platforms，Gesture detection is cool, my ha needs it

[francescopace]

Hi @bugensui2022, gesture detection is definitely an area I want to explore in depth.
Version 2.x of ESPectre is focused on the ESPHome port and stabilizing the Home Assistant integration.
The next release (3.x) will be fully centered on HAR (Human Activity Recognition), with gesture detection as a key feature.

It would be amazing to clap your hands to turn on the lights or give a thumbs-up to increase the volume of your home theater.
Really cool possibilities.

What use case do you have in mind?

[jerome83136]

Hi all,
First: great project, really like it 🙂
Use case I have in mind: replace wakewords by gesture to trigger smart assistant

Scenario: I raise my hand toward the sky, it triggers my HA smart assistant in listening mode, I say "turn on the kitchen light"

But maybe I'm dreaming and/or my use case is out of purpose ?
Let me know 😉

[francescopace]

Hi @jerome83136, this is absolutely the kind of use case I have in mind, and yes, gesture recognition is in the roadmap.

Before I move there, I want to ship v3.0 first, hopefully soon, ideally by the end of August. That release is important because it introduces a much more solid foundation for data collection, validation, training, and testing, and that is exactly the framework I used to build the new ml detector.

Once v3.0 is out, the next step for gestures will be to collect dedicated datasets for that problem, and then train a small model specifically for gesture recognition rather than generic motion detection.

The huge challenge is making that model run within the very limited memory budget of the device. But that is also the kind of challenge I already managed to solve for motion detection, so the door is definitely open!

And let me ask you this: why raise your hand to activate voice recognition, when you could just clap and let ESPectre send the event to Home Assistant and turn on the lights in that room? :)

[jerome83136]

Hi,
Thank you for your answer.
I see your roadmap is already well defined and you prepare a lot of exciting great features in a very short time. Congrats and thank you for your hard work 💪

I proposed a single gesture for triggering the smart assistant (as an alternative to wake word) for the following reasons:

• if I clap to turn on the kitchen light; what happens if I want to turn on living room light by clapping ? Will espectre determine I am in the living room and then understand it should turn living room light on, rather than kitchen ?
• by using a single gesture for triggering smart assistant's listening mode; I can then ask it to perform complex actions by vocal order. Correct me if I'm wrong but I guess doing the same with gesture would require recording 1 gesture per action, which could become hard to maintain and limiting for the maximum number of actions
• also, maybe could we make the architecture more robust by moving the gesture recognition logic out of the esp, to a dedicated node running ML on GPU ? (Just suggesting)

Thank you for your feedback and have a great day