An application that can recognise different types of safety signs; it offers an exploration mode where the user can discover safety signs and a quiz mode where the users are being quizzed about what they see.
In order to build and run the app, you only need to install the latest version of Xcode. Once you install it make sure you open the
.xcworkspace file, otherwise the dependencies will not be recognised.
The app is built on top of the object recognition example provided by Apple which offers a basic setup for object recognition with an already trained model.
Training a model
The models in this project were trained using turicreate, a library designed for easy creation of CoreML models which can be used for training models to perform image classification as well as object detection. A very good resource for getting started with training an object detection model with turicreate is this tutorial since it gives a detailed step by step guide.
Even though turicreate is used for training in this project, you could use any other framework as long as you convert to a CoreML model at the end. This is relatively easy to do with the coremltools library as long as you use one of their specified frameworks (Caffe v1, Keras 1.2.2+, scikit-learn 0.18, XGBoost 0.6, LIBSVM 3.22).
In order to have your trained model compatible with this project, you will have to comply to the following restrictions:
- All the classes names will have to follow the ISO standard names (ISO_7010_####). The
bobbyen.jsonfile contains display information about each class as well as a
keywhich represents the name of the class.
- The input for the model has to be a square image and the output has to be two
MultiArraysof confidence and coordinates. More detailed information about the types can be found by opening Xcode and clicking one of the example models.
Adding the model
- Open the Xcode project
- Drag and drop the
GetModel()function, change one of the models that is loaded.
- You could replace
- The switch statement corresponds to the Theme menu within the app (Options->Change Theme)
It is very important to have the class names match the name in the .json file! Otherwise, the app will not function properly. If you just want to quick test your model and make sure it works, I would definitely recommend using the object recognition example from Apple; even though it misses the functionality this app has, it is still enough for testing different trained models.
The app consists of four menus: explore, quiz, learn and options.
Highlights the position of the signs and gives information about the name of the object as well as its confidence. If the user presses one of the boxes, they will be sent to the learn menu, where they will receive more information about the identified object.
Highlights the position of the signs but does not give any information about it (left). If the user presses one of the boxes, they will be prompted with a multiple choice question generated based on the object they chose (right).
Gives the user the option to browse the 109 signs and get more detailed information about each one of them. All the information contained here is taken straight from the ISO website.
- Change contact information: changes the name and email of the user (mostly used for sending the xAPI statements)
- Change language: changes the language (related to all the bobby json files)
- Change theme: changes the currently loaded