Using TinyEngine with TensorFlow Lite custom models & different/custom datasets.

[antragoudaras]

Is it possible to capture/improve the performance of (in terms of accuracy and peak memory usage) a custom tflite already trained model (that I converted from an originally simple keras model) using TinyEngine, when compared to the plain TensorFlow Lite implementation of the same model? Also do I need to add any extra functionality to the existing code-base in order to evaluate my model against my own dataset(dataset form: training & validation sets as numpy arrays, classification problem with 4 classes)?

Any suggestion/guidance would be deeply appreciated on how to conduct the performance analysis described above by using the TinyEngine inference library, given that my model only supports compatible TinyEngine operators(aka neural net layers).
-Antonios.
p.s. Novice fan/user of TinyEngine.

[meenchen]

Hi @antragoudaras thanks for your interest in our work. You can deploy your model with TFLite micro and Tinyengine to compare the performance improvement. You can refer to TFLite micro to see how to deploy your TFLite model. For Tinyenigne, you can refer to our examples. If you also want to evaluate the accuracy on-device, you may need to implement some evaluation scripts (both on MCU and PC) so that your PC can communicate with the MCU through a serial port and evaluate each input in the dataset.

[meenchen]

Close due to inactivity. Feel free to reopen.

[ellial]

Hi @meenchen . I am trying to validate the on-device accuracy for the visual wake words dataset. Is there any chance you could reopen this issue and share the scripts you used when measuring the on-device accuracy for STM32?

[meenchen]

Hi @ellial,

Our quantization implementation basically follows tensorflow lite micro, so we evaluate the accuracy on servers with TF Lite runtime (example: https://github.com/mit-han-lab/mcunet/blob/be404ea0dbb7402783e1c825425ac257ed35c5fc/eval_tflite.py). We are also working on supporting different ISA like x86 or ARM. It will be much easier to evaluate the on-device accuracy.