- Safety: Always make sure you operate in a safe environment. Keep in mind, that your phone could be damaged in a collision! Special care is neccessary when using automated control (e.g. person following or driving policy). Make sure you always have a game controller connected and are familiar with the key mapping so you can stop the vehicle at any time. Use at your own risk!
- App under development: The application is under development and may crash or exhibit unexpected behaviour depending on your phone model and version of the operating system. Make sure to test all functionalities with no wheels connected. Use at your own risk!
The app starts with a menu screen that shows all available screens. The settings screen can be opened with a click on the icon at the top right corner. By clicking on the other icons the user can access various screens whose functionalities are explained in the following.
The DefaultActivity includes the most important features of the OpenBot app in a single screen. It displays the connection status to the vehicle and reports measurements from vehicle sensors. The robot can be controlled by standard BT game controllers or another smartphone running the OpenBot controller app. We have also implemented a data logger to collect datasets with the robot. Currently, we record readings from following sensors: camera, gyroscope, accelerometer, magnetometer, ambient light sensor, and barometer. Using the Android API, we are able to obtain the following sensor readings: RGB images, angular speed, linear acceleration, gravity, magnetic field strength, light intensity, atmospheric pressure, latitude, longitude, altitude, bearing, and speed. In addition to the phone sensors, we record body sensor readings (wheel odometry, obstacle distance and battery voltage), which are transmitted via the serial link. We also record and timestamp control signals received from a connected controller, if present. Lastly, we integrate several neural networks for person following and autonomous navigation.
The drop-down menu is used to set the baud rate. The default is 115200 and you should not need to change this unless you mess with the Arduino firmware. The app will attempt to connect automatically, but in case you encounter issues you can use this switch to disconnect/connect.
The field Battery displays the battery voltage as measured by the Arduino via the voltage divider. The field Speed (l,r) reports the left and right speed of the (front) wheels in rpm. It is measured by the Arduino via the optical wheel speed sensors. The field Sonar shows the free space in front of the car in centimeters. It is measured by the Arduino via the ultrasonic sensor. Note, you will only receive values a few seconds after the USB connections has been established.
The first button is for selecting the control mode. There are two different control modes:
- Gamepad: The app receives controls from a connected BT controller.
- Phone: The robot can be controlled via another smartphone with the controller app installed or though a Python script running on a computer connected to the same network.
The second button is for selecting the drive mode. There are three different drive modes when using a game controller (e.g. PS4):
- Game: Use the right and left shoulder triggers (R2, L2) for forward and reverse throttle and either joystick for steering. This mode imitates the control mode of car racing video games.
- Joystick: Use either one of the joysticks to control the robot.
- Dual: Use the left and right joystick to control the left and right side of the car. This is raw differential steering.
The third button is for selecting the speed mode. There are three different speed modes:
- Slow: The voltage applied to the motors is limited to 50% of the input voltage (~6V).
- Normal: The voltage applied to the motors is limited to 75% of the input voltage (~9V).
- Fast: There is no limit. The full input voltage will be applied to the motors at full throttle (~12V). This is the default setting for running the neural networks.
Running at higher speeds will reduce the lifetime of the motors but is more fun. The controls that are sent to the robot are displayed on the right side. When using the game controller, the speed mode can be increased by pressing down the right joystick (R3) and decrased by pressing down the left joystick (L3).
There are four different logging modes:
- only_sensors: All sensor data but no images are saved.
- crop_img: All sensor data and a cropped images that have the input size of the network are saved. This is the default setting and is what should be used for data collection.
- preview_img: All sensor data and a full-size images are saved. This will require a lot of memory and can be slow. However, it is nice for compiling FPV videos.
- all_imgs: All sensor data and both cropped and full-size images are saved. This will require a lot of memory and can be slow.
The switch on the right is used to toggle logging on and off. On the game controller this switch can be toggled with the X button.
The first item shows the preview resolution. The second item shows the crop resolution. This is the image that is used as input to the neural networks. You will notice that this resolution changes depending on which model you select below. If you train your own autopilot, make sure to select the AUTOPILOT_F model. The crop resolution should show 256x96. The switch on the right is used to toggle between the rear and the front camera.
There are two models that come with the app:
- MobileNetV1-300: This model is used for person following. It uses a SSD object detector with MobileNet V1 backbone. The model is quantized for better performance on embedded devices. It comes with the app.
- CIL-Mobile: This model is used for autonomous navigation. It will predict controls directly from the camera input. Chances are that it will not work in your environment. You should follow our instructions to train your own Driving Policy and replace it.
Additonal models can be downloaded from the Model Management screen.
The switch on the right is used to turn the network on and off. When the network is running, it produces the controls for the robot and the game controller is disabled. However, you may still use the buttons on the game controller, for example to toggle this switch with the R1 trigger button to regain control of the robot.
Use the drop-down menu to select the device on which the neural network should be executed. You have the following choices:
- CPU: Using the CPU works on most phones and is the default choice. You can adjust the number of threads to optimize performance.
- GPU: Most smartphones have a GPU. Networks with large inputs such as images often run faster on a GPU.
- NNAPI: This will use the TensorFlow Lite NNAPI delegate. Modern smartphones often come with dedicated AI accelerators. The Neural Network API (NNAPI) provides acceleration for TensorFlow Lite models on Android devices with Graphics Processing Unit (GPU), Digital Signal Processor (DSP) and Neural Processing Unit (NPU). Note that on some older phones this can be very slow!
If a model is active, the inference speed in [ms] will be displayed next to the device which is running the model.
Free Roam offers simple robot control with real time updates and information about battery, speed and distance from surfaces.
-
Battery: The battery icon shows realtime battery levels of the connected robot.
-
Drive Mode: There are 3 drive modes displayed on the view:
-
D -> Drive, when the robot is driving forward
-
N -> Neutral, when the robot is stationary
-
R -> Reverse, when the robot is moving backwards
-
-
Speed: The speedometer shows the realtime speed of the robot.
-
Sonar: The sonar view distance of robot from an oncoming object in cm.
-
Control: Controller, Drive Mode and Speed are used to control robot settings as described above in the Control section.
Simple UI for collection of data sets.
-
Preview Resolution: Used to switch between resolutions of camera preview. There are 3 settings:
- FULL_HD (1920x1080p)
- HD (1280x720p)
- SD (640x360)
-
Model Resolution: Used to switch between resolutions of images saved for training different models.
-
Save/Discard the Collected Data: the data collection process can be controlled from the screen or remotely, for instance from a bluetooth controller. When using a bluetooth controller, you may:
- press the A button to start the data collection process
- press the A button again to stop data collection and save the collected data in a .zip file
- alternatively press the R1 button to stop data collection without saving the collected data (for instance because of an unexpected collision with the environment)
- remember to use the controller mapping fragment to ensure you are using the correct buttons.
Simple UI to check the button and joystick mapping of a connected BT controller.
Simple UI for running autopilot models.
Simple UI for tracking objects of 80 different classes. A short description of the different AI models for object tracking and performance benchmarks can be found in Model Management.
Note that this fragment requires ARCore and camera permission. If your device does not support ARCore and you continue anyways, the app will crash. In this screen you can specify a goal via a 2D vector with respect to the current position and orientation of the robot. The 2D vector contains the distance to the front and left of the robot in meters. Both values can also be negative and correspond to back and right of the robot in that case. After specifying the goal and pressing Start the robot will exectue an AI policy that attempts to reach the goal while avoiding obstacles.
All models are quantized for better performance on embedded devices. Please refer to section below for a short description of the available models and benchmarking results. The mean Average Precision (mAP) is computed on the validation set of the COCO Detection 2017 dataset. The runtime is averaged across 100 frames and reported in frames per second (fps).
| Model Name | Chipset | RAM | OS |
|---|---|---|---|
| Samsung S22 Ultra | Exynos 2200 | 12GB | 12 |
| Samsung S20FE 5G | Snapdragon 865 | 6GB | 12 |
| Huawei P30 Pro | Kirin 980 | 8GB | 10 |
| Google Pixel 6XL | Google Tensor | 12GB | 12 |
| Xiaomi Mi9 | Snapdragon 855 | 6GB | 9 |
SSD object detector with MobileNet V1 backbone and input resolution of 300x300.
| phone/device (fps) | CPU | GPU | NNAPI |
|---|---|---|---|
| Samsung S22 Ultra | 33 | 13 | 30 |
| Samsung S20FE 5G | 34 | 57 | 87 |
| Huawei P30 Pro | 36 | 25 | 10 |
| Google Pixel 6XL | 35 | 42 | 53 |
| Xiaomi Mi9 | 22 | 41 | 33 |
SSD object detector with MobileNet V3 backbone and input resolution of 320x320.
| phone/device (fps) | CPU | GPU | NNAPI |
|---|---|---|---|
| Samsung S22 Ultra | 30 | 17 | 30 |
| Samsung S20FE 5G | 34 | 42 | 28 |
| Huawei P30 Pro | 32 | 27 | 23 |
| Google Pixel 6XL | 33 | 43 | 27 |
| Xiaomi Mi9 | 20 | 45 | 10 |
State-of-the-art object detector YoloV4 with input resolution of 224x224.
| phone/device (fps) | CPU | GPU | NNAPI |
|---|---|---|---|
| Samsung S22 Ultra | 3.7 | 5.6 | 3.5 |
| Samsung S20FE 5G | 3.1 | 7.1 | 4.2 |
| Huawei P30 Pro | 2.4 | 6.2 | 0.7 |
| Google Pixel 6XL | 2.7 | 11 | 0.9 |
| Xiaomi Mi9 | 2.1 | 6.4 | 1.7 |
Tiny version of YoloV4 with input resolution of 224x224.
| phone/device (fps) | CPU | GPU | NNAPI |
|---|---|---|---|
| Samsung S22 Ultra | 31 | 12 | 31 |
| Samsung S20FE 5G | 30 | 21 | 14 |
| Huawei P30 Pro | 27 | 17 | 22 |
| Google Pixel 6XL | 29 | 24 | 19 |
| Xiaomi Mi9 | 16 | 14 | 9.3 |
Tiny version of YoloV4 with input resolution of 416x416.
| phone/device (fps) | CPU | GPU | NNAPI |
|---|---|---|---|
| Samsung S22 Ultra | 13 | 9.8 | 13 |
| Samsung S20FE 5G | 12 | 9.4 | 7.7 |
| Huawei P30 Pro | 8.4 | 7.6 | 6.9 |
| Google Pixel 6XL | 10 | 9.6 | 7.2 |
| Xiaomi Mi9 | 9.0 | 7.3 | 5.0 |
Please refer to the ContributionGuide to learn how to add your own fragments to the OpenBot app.
The TensorFlow Lite Object Detection Android Demo was used as starting point to integrate TFLite models and obtain the camera feed. The DefaultActivity runs the main thread and inherits from the CameraActivity to manage the camera and UI. The SensorService reads all other phone sensors and logs them. The ServerService and NsdService establish a connection to a local Python server with a React frontend. If you collect data it can be uploaded automatically for visualization, training ML models and downloading trained models to the robot. The env folder contains utility classes such as the Vehicle interface, GameController interface, PhoneController interface and an AudioPlayer for the audible feedback. The tflite folder contains the model definitions for the Autopilot and Detector networks.
Train your own Driving Policy