Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gesture collector website #85

Open
gfwilliams opened this issue Jan 30, 2020 · 11 comments
Open

Gesture collector website #85

gfwilliams opened this issue Jan 30, 2020 · 11 comments

Comments

@gfwilliams
Copy link
Member

@gfwilliams gfwilliams commented Jan 30, 2020

http://forum.espruino.com/comments/15082797/

We wanted to do this for Nodeconf and never got around to it.

A website where you connect via Web Bluetooth and then perform a bunch of gestures. They all get saved online and then you can use that information to train a tensorflow model - either for you or based on everyone's data.

@brainfart-be
Copy link

@brainfart-be brainfart-be commented Mar 29, 2020

I'm very interested in this gesture collector website and i would like to help to build it.
Unfortunately i don't know enough about bluetooth LE for now.
If someone can help me to get started about how i can collect data from the watch using bluetooth (with some useful ressource for exemple) that would be great!

@gfwilliams
Copy link
Member Author

@gfwilliams gfwilliams commented Mar 30, 2020

Wow, thanks!

To handle the Web Bluetooth it's easiest just to use something like the Puck.js library - there's a bit of info here: http://www.espruino.com/Web+Bluetooth

But basically you can just upload the code from here with Puck.write: https://github.com/gfwilliams/workshop-nodeconfeu2019/blob/master/step4.md#getting-more-data

Then listen for new lines of data coming back. It's basically this example here: http://www.espruino.com/Web+Bluetooth#two-way-communications

With setInterval(function(){Bluetooth.println(Puck.light());},100); replaced with the Bangle.on('gesture',gotGesture); code from the workshop (using Bluetooth.println rather than print)...

@gfwilliams
Copy link
Member Author

@gfwilliams gfwilliams commented Mar 30, 2020

... but let me know if you need any more help - I could knock a simple test app up pretty quick if needed

@brainfart-be
Copy link

@brainfart-be brainfart-be commented Mar 30, 2020

Thank you for the ressources, i will take a look later today

@brainfart-be
Copy link

@brainfart-be brainfart-be commented Mar 30, 2020

I think i got it, thanks for those.
I will try to hack a simple basic ui using vue (or vanilla if you prefer i don't really care)

What's are your though about the storage of those data ?

I will not handle that for now, just display collected data in the browser.

@gfwilliams
Copy link
Member Author

@gfwilliams gfwilliams commented Mar 31, 2020

From my point of view I'm trying to use vanilla for stuff.

I don't have great thoughts about data storage. I guess given we're doing a bunch of stuff on GitHub it might be possible to just store the data in a file there? The Google Colab for training the AI is hosted there so I guess it'd be really fancy if it could update the colab files itself?

@brainfart-be
Copy link

@brainfart-be brainfart-be commented Mar 31, 2020

This is how i imagine the tool. https://bangle.dimitri-gigot.com/gesture.html
Obviously it's all ugly and poorly coded, i'm just trying/learning stuff.

What do you think about it ? Is this how you imagine it ?

@gfwilliams
Copy link
Member Author

@gfwilliams gfwilliams commented Apr 1, 2020

Thanks! I just tried it and I'm not 100% sure if it's doing this right now as I only managed to record one gesture, but the idea is for each gesture you'll need to get lots of examples of it in order to train the network - so you need to be able to record each one.

Ideally there'd be a way to store those somewhere so you didn't have to do everything each time.

If possible we could have some kind of common repository for gestures - if everyone could contribute their own movements then we might have more of a chance of training a model that worked reliably for different people.

@brainfart-be
Copy link

@brainfart-be brainfart-be commented Apr 1, 2020

After choosing the gesture you want to record, it will push the gesture in the appropriate array, you can record multiple time in a row without having to choose again.
All records will be draw on the graph with an opacity, the last record will be draw on top.

The goal was to submit that to some kind of backend in order to aggregate the data.

(I guess you saw that, but English is not my first language :D sorry if i make mistake and/or if i'm not clear enough :D )

@brainfart-be
Copy link

@brainfart-be brainfart-be commented Apr 3, 2020

Little update: https://bangle.dimitri-gigot.com/gesture.html
In the end i decided to write it in Vue, easier for me to handle the UI
I used Spectre css to match the app store.
I don't handle the export of the data yet, for now i just try to validate the tool itself.
I tried to add some text to help understand how it work.
What do you think ?

@gfwilliams
Copy link
Member Author

@gfwilliams gfwilliams commented Apr 6, 2020

That looks great! Seems to work really well :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.