Native Tensorflow Lite support

[hierophect]

Tensorflow Light for Microcontrollers is a subset of Tensorflow Light designed specifically for use with microcontrollers, taking only 16 KB for the core runtime and requiring no system libraries or dynamically allocated memory. Adding these tools would allow for things like camera-based person detection, gesture sorting, voice recognition, and other tasks that use trained models.

Tensorflow Light is already implemented for Micropython on the OpenMV and the Sipeed MAix BiT. It is available using platforms like Arduino and Mbed on a variety of other development boards.

This thread is for general discussion of this implementation and tracking potential issues with its implementation.

[mocleiri]

I've built a micropython integration for tensorflow lite micro: https://github.com/mocleiri/tensorflow-micropython-examples

I've implemented all of the upstream examples except for magic wand. Some are just running micropython code with hard coded sample data and others like micro_speech can run in real-time with audio sampled over i2s.

I'm progressively expanding what has been done and I'm interested in integrating with circuit python to expand the number of people that can use it to experiment with python on microcontrollers.

The main issue is firmware size. The C++ api linker uses dead-code detection and only includes the associated code for the ops you are using which drastically reduces the size of the final firmware.

In a scripting language this is not possible because when we make the firmware its impossible to know which ops are needed,

At the moment the overhead for libtensorflow-microlite.a is 1.5 MB impact on the firmware size. This is fine for ESP32 because most have 4 MB flash but for many boards that is way to much.

I think it should be possible to add in compile time flags to control what subset of ops are included by default to shrink down the firmware size.

What is a common board with 2MB or more of flash that I can start with trying to build for?

[hierophect]

@mocleiri my first instinct would be the STM32H7 (probably H743), which we have basic support for, is used by OpenMV, and has a precedent for being used for this kind of application. However, a number of Circuitpython boards use external flash, including the ESP32S2, the i.MX RT, and the RP2040, all of which could theoretically support the firmware size you're mentioning. It'd probably be nice to pair it with a platform that supports a camera or audio features.

[mocleiri]

Thanks for your feedback.

I'm tracking circuitpython integration on: mocleiri/tensorflow-micropython-examples#27

I'm looking at reducing the ops and firmware size in:
mocleiri/tensorflow-micropython-examples#28

I'll start with the Cortex M7 boards like openmv and teensy 4.1 and then into the others you listed.

I'll ping back on this issue once there are firmwares to test.

[tannewt]

Thanks for looking into this @mocleiri.

[mocleiri]

Thanks for the mention on the adafruit blog (https://blog.adafruit.com/2021/08/23/tensorflow-lite-for-microcontrollers-in-micropython-tensorflow-micropython-esp32/).

I wonder if I can take the amazing micropython and tensorflow lite image from the article for my project?

I found out that the size of tensorflow lite micro is actually about 200-300 kb in terms of firmware size.

I turned off all 84 ops and it saved 200 kb on esp32. The other 100 kb is taken up with the base api and then the audio_frontend which wraps tensorflow c++ code used to generate the spectrograms in the micro_speech example.

I think its likely that it will fit on more of the circuitpython microcontrollers.

We're starting with boards that should not have a problem using i2s for real-time audio acquisition but for other boards where this is not possible it should be possible to make the audio_frontend class optional.

I was linking the c++ standard library into the tensorflow which was causing me to think the size was larger. The libtensorflow-microlite.a file was 1.5 MB but it turns out most of that was standard library which was stripped off during the dead-code removal.

I'm working on building libtensorflow-microlite.a using the arm cross compiler and will then attempt to link that into the circuitpython build. I'm building locally but will switch over to github actions ci on my fork when I get a little bit closer.

[ladyada]

sounds great! please keep us informed and let us know if we can help in any way :)

[mocleiri]

I found out that the circuitpython stm32 port doesn't support C++ which is a problem but I think it can be worked around.

Would it be possible to intake this micropython commit to add c++ support to stm32?

micropython@97960dc

The other option is to build the c++ parts externally and link them into the firmware. I'm already doing this for the tensorflow bits but it could be expanded to also include the micropython c++ binding layer.

It took me a while due to linker issues (due to differences of esp32 being based on freertos versus the stm32 port being bare metal) but I was able to build the microlite firmware today for the Nucleo-H743ZI2/ in micropython as a user_c_module.

I was able to flash and connect to repl and it looks like the microlite module is present. I will test and verify its working correctly by running the hello_world and person detection examples.

I built the libtensorflow-microlite.a in my project via a ci job, (action) and then copied the file locally and used it to build the firmware.

Now that it works I will add a ci job in my project for the micropython build for this board and then come back to the circuitpython build. I will create a pull request with the circuitpython changes and we can iterate on how exactly to structure the c++ parts so that circuitpython can be built and be used.

One issue is that on the tensorflows side it builds a different version of the library for cortex m0, m4 and m7 including with the different floating point numbers and cmsis_nn options.

For esp32 it was easy 1:1 tflite to micropython but for stm32 I think some kind of scheme where the board can specify the kind of libtensorflow-microlite.a it should link to will be needed. I would improve the naming of the artifacts published and then the circuitpython ci scripts can be updated to use a github action like:https://github.com/dawidd6/action-download-artifact to download the correct kind of library from my repository.

[tannewt]

I'd be happy to see C++ support integrated into our build files. I've not used C++ for embedded though so I'm not sure what exactly would need to change.

Why are you planning on linking to the prebuilt .a files? I'd prefer to build it directly into the binary if we can. That way we can tweak the compilation options and maybe use LTO.

[mocleiri]

Using the static library was a shortcut which worked on esp32 so I didn't have to know how it all worked on the tensorflow side.

In getting to the answer on how to link it I built most of such an approach to build everything from the micropython side.

I will continue and incorporate into my eventual pull request.

[mocleiri]

Sorry for the delay.

I've been having a lot of trouble getting the stm32 port to work right on my h743zi2 nucleo board..

Every time I try to load the hello-world model the board reset's.

I've just been through a deep dive on how a bare metal port works and how to debug on on the hardware in visual studio code but to no avail.

In Java you can set breakpoints on Exceptions and the IDE will pause when the error occurs and then you can see via the back trace where it originated from.

I don't know how to do this in C++/C so I can see the back trace before control flows back through the reset handler so I'm pretty stuck on what exactly is causing the reset.

I do have it working for esp32 and for stm32 where it will use the tensorflow lite micro project generator to put the tflm code within my microlite micropython module

I still need to incorporate the audio_frontend into the microlite module which is needed for micro_speech. I also only have one build on gnu make (stm32) and one on cmake (esp32) and I'm doing board specific things in those makefiles which needs to be changed to allow for building different boards through the same core makefiles.

You should be able to run this by downloading the firmware from:
https://github.com/mocleiri/tensorflow-micropython-examples/actions/runs/1397545963

I've been flashing the firmware.elf using stm32cubeprogrammer. I have it with alot of debug symbols present because I didn't want variables getting optimized out when trying to see what was going on in visual studio code.

After flashing copy hello_world.py and model.tflite
to /flash.
Then in run import hello_world to trigger the below behavior:

MicroPython v1.17-123-g7f1434442 on 2021-10-29; NUCLEO_H743ZI2 MICROLITE with STM32H743

Type "help()" for more information.
>>> import microlite
>>> help (microlite)
object <module 'microlite'> is of type module
  __name__ -- microlite
  __version__ -- e87305ee53c124188d0390b1ef8ec0555760d4d6
  interpreter -- <class 'interpreter'>
  tensor -- <class 'tensor'>
>>> import hello_world
interpreter_make_new: model size = 2488, tensor area = 2064
MicroPython v1.17-123-g7f1434442 on 2021-10-29; NUCLEO_H743ZI2 MICROLITE with STM32H743
Type "help()" for more information.
>>> 

What is supposed to happen is what happens on ESP32 which prints out an approximate sine wave: https://github.com/mocleiri/tensorflow-micropython-examples/blob/main/examples/hello-world/README.md

If anyone is able to debug and tell me why its not working I would continue through with the circuit python changes needed to support it. At the moment I want to switch to either the rp2040 or teensy4 boards I have to see if either work because I would like to build out the CircuitPython support from a working build.

[tannewt]

I've been having a lot of trouble getting the stm32 port to work right on my h743zi2 nucleo board..

Every time I try to load the hello-world model the board reset's.

I've just been through a deep dive on how a bare metal port works and how to debug on on the hardware in visual studio code but to no avail.

That sounds really frustrating! I can help you get a backtrace when you are ready to circle back to it. I wrote a guide about my process here: https://learn.adafruit.com/debugging-the-samd21-with-gdb It's written for SAMD21 but easily applies to any Cortex M chip.

If it's CircuitPython causing the reset, then you can set a break point on reset_into_safe_mode to catch the backtrace before the micro is reset.

I'd encourage you to join #circuitpython-dev on the Adafruit discord. We can help you get setup for debugging there.

[mocleiri]

Thanks for the useful article and breakpoint advice.

I found out my issue with stm32 is most likely related to it being bare metal and me not doing the right thing in terms of wiring up the standard library entry points (malloc/free, c++ new/delete) to use the micropython heap instead.

I was able to get the Adafruit feather RP2040 to build just fine: https://github.com/mocleiri/tensorflow-micropython-examples/actions/workflows/build_rp2.yml

I'm part way through getting tflm to build as part of my microlite module. The core library builds but I need to do more work still to re-integrate the audio frontend which wraps the tflm micro_speech c spectrogram generating code.

I'm going to try a mimxrt port since I have a teensy 4.1 (although it also looks to be bare metal like stm32) and then after that circle back to this issue and taking what ever I have working through to integrate with Circuit Python