An Android App for interacting with a Virtual Pet - Dog in AR
This application will allow any Android User to interact and play with with a Virtual Pet (Talking Dog) in a magical way using Gestures (Finger), Natural Langauge (voice commands) in Augmented Reality. The app leverages Android AR Core and Google MLKit and Media Pipe open soure project to create this exprience on the edge.
Project Plan :
Technologies & Sample Code :
-
Media Pipe (TensorFlow Lite) & Android ARCore Depth API by Google to build Android Model for Hand Tracking that will be used to interact with the Pet : https://github.com/suyashjoshi/mediapipe
-
ARCore (Android SDK & Sceneform) : AR is the main User Experience and I will be leveraing AR Core Android SDK and SceneForm fraemwork to create AR Experience for this app https://developers.google.com/ar/develop/java/quickstart
- 3D Model of the Dog from Google Poly : https://poly.google.com/view/0IuRrFSlk5D
-
Audio & Speech Recogniton : Audio is a big driver in order to create immersive experiences, the visual creates will not only produce sound but users will also be able to speak to them using natural langauge. I will be leveraging this project to train the model and use Speech Recognition https://github.com/tensorflow/examples/tree/master/lite/examples/speech_commands/android
I have started to play with #1 MediaPipe in order to build an Android Model for my hand tracking and have started to communicate with the project engineers on Github. I would love to get help on the same from Google ML Researchers.
I have been a full time software developer for about 10 years now and bring background in Android, Web and Deep Learning technologies. Previously I've also been invited to speak on State of Mobile AR at MIT : and on Deep Learning for Java Developers : https://www.youtube.com/watch?v=SORF-lNCfLU , https://www.youtube.com/watch?v=y0vHB4YfvWs
I'm very excited to take part in this cahllenge and build this app and make it open source as later use it as teaching project for other developers.