The AI-deck enables low power on-board artificial intelligence capabilities for the Crazyflie using a GAP8 chip with RISC-V multi-core architecture, 512 Mbit HyperFlash and 64 Mbit HyperRAM. In addition the deck has a Himax HM01B0 grayscale camera with a NiNA W102 Wi-Fi module to stream your images to a desktop. These features fit the prerequites of a convolutional neural network, but the AI-deck is not limited to the application of CNN's.
The example shows a simple convolutional neural network based on the MNIST example of the GAP8 SDK. MNIST is a CNN that classifies handwritten digits ranging from 0 to 9. Added to this is a simple application for the Crazyflie where it takes the output of the MNIST network and uses that to switch states. If the neural network identifies a 1, the GAP8 code will send a UART byte to the Crazyflie which in turn gives a command to unlock. If the neural network identifies a 9 it will unlock. If the Crazyflie is unlocked then it will turn right or left when the neural network identifies a 4 or 5, respectively.
This example can be easily modified to a different classification task by using a similar and simple dataset.
In addition, to get quickly started on your own neural network you can make use of the principles of Transfer Learning.
For users starting with embedded applications and the Crazyflie
For this example a basic understanding on how to:
- Program in Python and C
- Use Linux systems
- Use Makefiles
- Olimex ARM-USB-TINY-H JTAG
- ARM JTAG 20 to 10 pin adapter
- CrazyFlie 2.1
Setting up the GAP8 SDK
To get started, first set up the GAP8 SDK using the instruction on this repository:
Workflow using GAPFlow
To design a neural network and deploy it on the AI-deck, you should know the workflow of the GAP8 SDK for AI applications that is provided by GreenWaves Technologies. A neural network can be designed, trained, and evaluated using Tensorflow and Keras in Python. To let this code be able to run on the AI deck, an automated process is executed by the GAPFlow of the GAP8 SDK.[insert image about GAPFlow]
Courtesy of GreenWaves Technologies: https://greenwaves-technologies.com/
What you should provide in this workflow is:
- dataset with labels
- neural network model
- GAP application code
- optional: own autotiler operator
Tensorflow and Keras use
Keras is a framework within Tensorflow. In addtion to providing a neural network framework, it also provides examples of common and simple neural networks and provides datasets along with it. Though when you want to make an application you might want to use other datasets that are relevant for your application. For this you have to supply your own dataset with labels, augmentation and transformations when required.
The NNTool makes use of post-training quantization and adjust the quantized weights using a selection of images used for training the neural network.
The Autotiler supports basic operators needed for a convolutional neural network. If you need a specialized operator for your neural network, then you can make your own autotiler operator by using this page.
Application code for the AI-deck
Environment in the GAP8 SDK
This example mainly uses the following set up.
- FreeRTOS as RTOS
- PMSIS-API as low-level driver
- PMSIS-BSP as high-level driver
- OpenOCD as hardware debugger
Application code for the Crazyflie
For users starting with the Crazyflie the following should be known on how to:
- Use LOG and PARAM in Crazyflie
- Make an application in the Crazyflie Applayer
An example on how to do this can be found here https://github.com/ataffanel/crazyflie-push-demo.