Advancing Our Amazing DL Engine: MNN 1.0.0
As many of you MNN fans may know, MNN was initially released in May 2019 by a team of enthusiastic Alibaba engineers who thought existing open source DL frameworks at the time weren’t good enough to handle the particular and demanding requirements of a billion-user mega application such as Taobao and that it was better to roll their own.
And so we did it. We built an inference engine from the ground up and we supported a vast variety of applications inside Alibaba with it. It was many times faster (and smaller!) than other inference engines such as TFLite and NCNN. We open sourced it and you guys loved it. The 3.9k stars on the Github repo are 3.9k votes of confidence from our biggest fans: you. What’s more, we are thrilled to publish our paper on MNN design principles in MLSys 2020 (Read the paper here: https://proceedings.mlsys.org/static/paper_files/mlsys/2020/7-Paper.pdf).
A year has passed since the initial release and we’ve made a great number of new features, improvements and bug fixes. And today we’re delighted to announce the 1.0.0 official stable release of MNN. In this release, we add training capability to MNN, which means MNN is no longer an “inference” engine, but a fully-fledged DL engine capable of training and inference, with numerous hardware backends including Arm (32bit, 64bit, v8.2), x86, Metal, OpenCL, Vulkan and OpenGL.
The following are some release highlights.
Highlights
There are three major new features in this MNN stable release: training and quantization, performance improvements and MNN Python API (currently in BETA).
Training and Quantization With the Express API
We added a new set of C++ APIs (located in the express/ directory) to dynamically construct a graph, run training and inference, quantize models, etc. These APIs stand side-by-side with the existing APIs to use MNN and will be the officially recommended API surface once we iron out more details.
This new set of C++ APIs allows you to:
- train a model from scratch using MNN,
- finetune a model trained by other DL frameworks,
- and perform quantization-aware training (a.k.a QAT).
The documentation has been updated with instructions. (Quantization-aware Training)
The following table is a comparison of model accuracy and size between MNN QAT and Tensorflow QAT using MobileNet V2.
Note 1
Both training and eval use ImageNet. Training batchsize is 32, with 100 iterations, i.e. 3200 images are used in training. Eval phase uses 50000 images.
Note 2
The original tensorflow model has an accuracy of 71.8%. Due to minor differences in image preprocessing, the original model in MNN format has a higher accuracy than the TF original.
On top of the improved accuracy with MNN QAT, MNN has great training speed on-device and on a PC. The following graph shows the time (in secs) to train a Lenet model with MNIST from scratch.
As you can see, MNN training performance on a mobile phone (MI6 or Mate 20) is comparable to the performance of PyTorch or Caffe on a 2015 Macbook Pro. What's more, MNN training speed on a Mac is more than 2 times as fast as PyTorch.
Performance Improvements
We have been hard at work pushing the limits of Arm CPU and x86 CPU, and bringing every bit of performance improvement into MNN.
Arm V8.2 Performance Improvements
We utilized asimdhp (Advanced SIMD half precision) extension and asimddp (Advanced SIMD dot product) in ArmV8.2 to increase inference speed by roughy 100%.
Inference time (ms) before and after half precision optimization:
MNN doubles the inference performance without any noticeable drop in accuracy. Details below:
Inference time (ms) before and after SIMD dot product optimization:
MNN doubles the speed with SIMD dot product optimization, even on devices whose int8 performance is worse than fp32. Details below:
Arm64 Performance Improvements
By rethinking matrix multiplication tiling, cacheline alignment and cache prefetch, we've made significant performance improvements on mid to low tier chipsets (Qualcomm 652 and 425), which are the types of devices where a sizable perf improvement could mean the difference between smooth 30 fps and a janky unusable experience.
Inference speed (ms) for MobileNetV1 before and after optimization.
X86 Performance Improvements
Even though Arm platform is MNN's "basecamp", we've been investing in the x86 platform so as to make MNN training on laptops faster.
By improving the weight matrix layout and turning on FMA instruction, we've improved the x86 performance by about 40% ~ 60% single-threaded. (The performance improvement is 100% compared to MNN at initial release).
The following table shows the inference performance before and after the optimization.
New Python Express API (BETA)
The aforementioned "Express API" in C++ allows you to train a model with MNN, finetune a model and perform quantization-aware training with MNN. We added a set of corresponding Python Express APIs for people with prior experience of Tensorflow or PyTorch. The new Python Express API documentation is here.
The new Python Express API is currently in beta as we strive to make it more pythonic and PyTorch-esque. In the mean time, please give us feedback by posting an issue on Github and leave a message in the Dingtalk group.