Gesture collector website

[gfwilliams]

http://forum.espruino.com/comments/15082797/

We wanted to do this for Nodeconf and never got around to it.

A website where you connect via Web Bluetooth and then perform a bunch of gestures. They all get saved online and then you can use that information to train a tensorflow model - either for you or based on everyone's data.

[brainfart-be]

I'm very interested in this gesture collector website and i would like to help to build it.
Unfortunately i don't know enough about bluetooth LE for now.
If someone can help me to get started about how i can collect data from the watch using bluetooth (with some useful ressource for exemple) that would be great!

[gfwilliams]

Wow, thanks!

To handle the Web Bluetooth it's easiest just to use something like the Puck.js library - there's a bit of info here: http://www.espruino.com/Web+Bluetooth

But basically you can just upload the code from here with Puck.write: https://github.com/gfwilliams/workshop-nodeconfeu2019/blob/master/step4.md#getting-more-data

Then listen for new lines of data coming back. It's basically this example here: http://www.espruino.com/Web+Bluetooth#two-way-communications

With setInterval(function(){Bluetooth.println(Puck.light());},100); replaced with the Bangle.on('gesture',gotGesture); code from the workshop (using Bluetooth.println rather than print)...

[gfwilliams]

... but let me know if you need any more help - I could knock a simple test app up pretty quick if needed

[brainfart-be]

Thank you for the ressources, i will take a look later today

[brainfart-be]

I think i got it, thanks for those.
I will try to hack a simple basic ui using vue (or vanilla if you prefer i don't really care)

What's are your though about the storage of those data ?

I will not handle that for now, just display collected data in the browser.

[gfwilliams]

From my point of view I'm trying to use vanilla for stuff.

I don't have great thoughts about data storage. I guess given we're doing a bunch of stuff on GitHub it might be possible to just store the data in a file there? The Google Colab for training the AI is hosted there so I guess it'd be really fancy if it could update the colab files itself?

[brainfart-be]

This is how i imagine the tool. https://bangle.dimitri-gigot.com/gesture.html
Obviously it's all ugly and poorly coded, i'm just trying/learning stuff.

What do you think about it ? Is this how you imagine it ?

[gfwilliams]

Thanks! I just tried it and I'm not 100% sure if it's doing this right now as I only managed to record one gesture, but the idea is for each gesture you'll need to get lots of examples of it in order to train the network - so you need to be able to record each one.

Ideally there'd be a way to store those somewhere so you didn't have to do everything each time.

If possible we could have some kind of common repository for gestures - if everyone could contribute their own movements then we might have more of a chance of training a model that worked reliably for different people.

[brainfart-be]

After choosing the gesture you want to record, it will push the gesture in the appropriate array, you can record multiple time in a row without having to choose again.
All records will be draw on the graph with an opacity, the last record will be draw on top.

The goal was to submit that to some kind of backend in order to aggregate the data.

(I guess you saw that, but English is not my first language :D sorry if i make mistake and/or if i'm not clear enough :D )

[brainfart-be]

Little update: https://bangle.dimitri-gigot.com/gesture.html
In the end i decided to write it in Vue, easier for me to handle the UI
I used Spectre css to match the app store.
I don't handle the export of the data yet, for now i just try to validate the tool itself.
I tried to add some text to help understand how it work.
What do you think ?

[gfwilliams]

That looks great! Seems to work really well :)