A No-code, one stop platform, to record, annotate, visualize, merge gesture files, augment, pre-process, train models and evaluate dynamic gestures using an RGB camera.
Version 1.1.1:
+Bug fixes
---
Version 1.1:
With added Tensorboard support!
Includes all features in Version 1.0
Check the bottom of this file for features included in previous versions.
- About
- App in action
- The home tab
- The Recording tab
- The Data Processing and Visualization tab
- The Model Training tab
- How to Install AirInteract
- Dependencies
- License
- Version History
To record, pre-process, train and test the dynamic gestures, introducing a no-code platform: AirInteract. AirInteract is based on Streamlit, a python-based web application deployment service. AirInteract is written completely in python. It has 3 main sections: Data Recording, Processing and Model Building.
- The data recording tool can save recorded gesture key points, where users can pause and resume recording. Users can also annotate their data simultaneously. Users have the flexibility to change the recording duration by changing the number of datapoints.
- The data processing tool can merge the collected datasets into one and process it, such as removing the null values, redundant columns and normalizing it. Users can also visualize the recorded gestures. Users can then download the processed data.
- The model training tool takes input of the processed dataframe, from the data processing tool. Users can split into train and test datasets by simply entering percentage of test dataset split. Users can then choose whether to use PCA. On selecting PCA, users can choose the number of components and percentage variability is then displayed. Then users can choose the type of model they wish to build. The application uses TensorFlow as the backend to build the network. First step would be to specify nodes in the input layer and entering total number of layers. Post that the user will have to just enter the number of neurons followed by the activation function. The application then displays the model summary. Users then must select the loss function and optimizer from dropdown menus and choose the number of epochs to train. Upon selecting the train checkbox, the model is trained. The training graph and the final test accuracy is displayed.
My.Movie.mov
The home screen of Airinteract has the navigation section,
where users can navigate to the page of their choice.
The Recording screen of Airinteract,
where users can record their gestures.
The Recording screen of Airinteract,
where users can select the type of the gestures they want.
The Recording screen of Airinteract,
where users can annotate and add custom annotation labels
and save checkpoints as csv files.
My.Movie.1.mov
The Data Processing and Visualization screen of Airinteract,
where users can merge gesture csv checkpoint files and pre-process them.
They can also scroll through samples and visualize them.
They can also save the merged file.
My.Movie.3.mov
Added Tensorboard support!
The model building and training screen of Airinteract,
where users can view the training predictor files.
They can also use techniques to reduce dimensionality.
They can select the type of network they wish to train.
They can set and tune hyperparameters
And also evaluate the model
- Clone the repo
git clone https://github.com/259mit/AirInteract
- Install requirements
pip install -r requirements.txt
- Run app.
python3 -m streamlit run main.py
For Tensorboard:
http://localhost:6006
Python 3.7
Streamlit
MediaPipe HandPose Engine, Google.
MIT License, 2021
Version 1.0:
Supports Dense neural Network,
Recording 3 gesture categories with custom annotations.
PCA + In built pre-processing pipeline!