"What Dog?" is a mobile app based on the TensorFlow example classification app with a new model trained to detect dog breeds.
Yes, there's already one from the guys at Microsoft and probably numerous others, but the Microsoft one simply submits an image to a web service.
Unlike theirs, I wanted my app to do offline on-device inference with my trained model.
Plus this serves as an open source example for any other Udacity AIND students who want to test their models outside on RL dogs.
I did this project as part of the Udacity - Artificial Intelligence Nanodegree on CNNs and transfer learning in Keras. You can read more about the project in my dog_app.ipynb Jupyter Notebook.
The model was based on Google's InceptionV3 model and retrained using Transfer Learning. I followed some suggestions for Transfer Learning and Fine Tuning from @dlsandbox on his Github.
After an afternoon of playing around with an AWS p2.xlarge I got the model to an accuracy of 87.2% on the test set.
Once the model was trained and working, I wanted to test it on real dogs outside, so I figured I would need to run it on my phone.
@timanglade Wrote about his experience making the HBO show Silicon Valley's Not Hotdog App and deploying a Keras model to mobile. I also found this great guide by @stratospark on doing a similar Keras to TensorFlow app for Food Detection.
Inspired by the Not Hotdog App and food-101 guides, I decided to use TensorFlow SDK to make "What Dog?".
Unfortunately there was no simple instructions on how to get my model from Keras into a TensorFlow pb (protobuff) file ready for on-device inference.
After speaking with both @timanglade and @stratospark on Twitter, reading lots of TensorFlow GitHub issues and modifying the useful keras_to_tensorflow tool, I managed to get it working.
This should just work if your model is based off the Google InceptionV3 model, or from my Jupyter Notebook, otherwise you might need to make some adjustments.
- Place your hdf5 keras model in the data folder
- Run my modified keras_to_tensorflow.ipynb in the model_converter folder
- Run test_pb_model.ipynb and verify the model is still working
- If you only want iOS skip to step 7
- For Android I needed to run the tensorflow tool transform_graph on the pb file
- To get the transform_graph tool I had to compile tensorflow using bazel in a Ubuntu VM
- Read the transform_graph instructions
- Pay attention to the inputs and outputs flags listed in the test_pb_model.ipynb, mine were --inputs='input_3' and --outputs='softmax0'
- Copy your new pb file (the transform_graph model works on iOS and Android) into the data folder and replace the old one, or change the project source code to reference your new pb file.
- Do the same for the labels text file
Inside the ios folder run:
$ pod install
- Open whatdog.xcworkspace
- Change Signing Team if necessary
- Select the "whatdog - Debug" scheme
- Build & Run
- Import the android folder into Android Studio
- Build & Run
There plenty of bugs, this is barely Beta quality. The model is large and real time inference takes a lot of CPU power. The app struggles to run on my Sony Xperia Z3 Compact. It runs much better with something like an iPhone 6 or Samsung Galaxy 6.
If the app isn't working properly just quit it and open it again. Sometimes the inference gets stuck or confused by certain images.
Pull Requests are welcome. 😊
Thanks to everyone who wrote guides, code, or answered my questions on Twitter. Big thanks to Udacity for the AIND and Google for making TensorFlow!
The icon graphics used for "What Dog?" are from flaticon.com.
Icons made by Freepik from www.flaticon.com is licensed by CC 3.0 BY