Add export and detection for TensorFlow saved_model, graph_def and TFLite

[zldrobit]

This PR add models/tf.py to export TensorFlow saved_model, graph_def and TFLite models.
This script support both yolov5 v2 (LeakyReLU activations) and v3 (Hardswish activations) models.

Export TensorFlow and TFLite models using:

PYTHONPATH=. python models/tf.py --weights weights/yolov5s.pt --cfg models/yolov5s.yaml --img 640

and use one of the following command to detect objects:

python detect.py --weights weights/yolov5s_saved_model --img 640
python detect.py --weights weights/yolov5s.pb --img 640
python detect.py --weights weights/yolov5s.tflite --img 640

This PR is tested sucessfully with PyTorchc 1.6, TensorFlow 1.15.3/2.3.0.
However, TFLite export is only supported under TensorFlow 2.3.0.

🛠️ PR Summary

_{Made with ❤️ by Ultralytics Actions}

🌟 Summary

The PR introduces support for the Android platform to the YOLOv5 model, focusing on setting up the necessary Android project structure.

📊 Key Changes

• Added Android app project structure and configuration files.
• Included resources such as layouts and vector icons for the Android application interface.
• Provided Gradle build scripts and properties for the Android project build automation.
• Implemented Java/Kotlin classes for activities, custom views, TensorFlow Lite integration, and utilities specific to Android.

🎯 Purpose & Impact

• The integration of YOLOv5 into an Android app allows for on-device object detection, which can benefit various applications such as real-time image analysis, augmented reality, and surveillance.
• Users can leverage the power of YOLOv5 directly from their mobile devices without the need for server-side processing, leading to faster and privacy-compliant applications.
• This update potentially opens the project to a wider community of mobile developers interested in deploying machine learning models on Android.

[github-actions]

Hello @zldrobit, thank you for submitting a PR! To allow your work to be integrated as seamlessly as possible, we advise you to:

• Verify your PR is up-to-date with origin/master. If your PR is behind origin/master update by running the following, replacing 'feature' with the name of your local branch:

git remote add upstream https://github.com/ultralytics/yolov5.git
git fetch upstream
git checkout feature  # <----- replace 'feature' with local branch name
git rebase upstream/master
git push -u origin -f

• Verify all Continuous Integration (CI) checks are passing.
• Reduce changes to the absolute minimum required for your bug fix or feature addition. "It is not daily increase but daily decrease, hack away the unessential. The closer to the source, the less wastage there is." -Bruce Lee

[bartvollebregt]

Very nice!

Do you have a way to perform the detection with Java / Android already? This is where the TFLite formats will probably be used the most.

[lei522]

@zldrobit ,thank you for your pr, i have exported TFLite models successfully. but how can i use the letterbox function in tflite project ?

[zldrobit]

@bartvollebregt Thanks :D I am cleaning my Android project code, and I am going to push that code soon.
@lei522 No thanks :D Please wait for me to upload the Android demo code. TFLite model runs directly in the demo.

[BernardinD]

@zldrobit Do you also plan to look into an edgetpu implementation? I am having issue with int8 quantization of the keras model

[Guilhen]

Some operations used in the model such as resize_nearest_neighbor are not compatible with the edge TPU

[zldrobit]

@BernardinD I have never used an Edge TPU and I found it has 4T FLOPs computation power today.
I will try to buy one, but it takes a long time for shipping.

@Guilhen yes, resize_nearest_neighbor has 3D input/output limitations.

[zldrobit]

@bartvollebregt @lei522 I updated the TF conversion script for GPU delegate compatibility and uploaded Android TFLite demo.
Please use

PYTHONPATH=. python3 models/tf.py --weights weights/yolov5s.pt --cfg models/yolov5s.yaml --no-tfl-detect

to generate yolov5s.tflite, and copy it to the Android asset folder.
Then, build and run the Android TFLite demo.

I have tested yolov5s batch-size 1 run time on Android devices:
Snapdragon 820: 2.1s on CPU (4threads) and 1.5s on GPU
Snapdragon 845: 1.2s on CPU (4threads) and 0.7s on GPU

Snapdragon 820: 1.9s on CPU (4threads) and 1.3s on GPU
Snapdragon 845: 1.1s on CPU (4threads) and 0.6s on GPU

[lei522]

@zldrobit i can run the tflite model, but when i run the model with hybrid quantization model or float16 quantization, i get the ValueError:Input array not provided for operation 'reshape'.

[zldrobit]

@lei522 I added fp16 and int8 TFLite model export.
Plz try again with

PYTHONPATH=. python3  models/tf.py --weight weights/yolov5s.pt --cfg models/yolov5s.yaml --no-tfl-detect --tfl-int8 --source /dataset/coco/coco2017/train2017 --ncalib 100

This will generate yolov5s-fp16.tflite and yolov5s-int8.tflite.
You can use them in the new Android project.

@BernardinD @Guilhen The updated code supports full integer quantization (with --tfl-int8), which can be compiled to Edge TPU format. I have tested int8 quantization model on Android devices.

The inference time of fp16 model (batch-size 1) is:
Snapdragon 820: 1.9s on CPU (4threads) and 1.3s on GPU
Snapdragon 845: 1.1s on CPU (4threads) and 0.6s on GPU
The inference time agrees with fp32's, due to the default fp16 precision of TFLite.

The inference time of int8 model (batch-size 1) is:
Snapdragon 820: 1.7s on CPU (4threads)
Snapdragon 845: 1.0s on CPU (4threads)
CPU inference time reduction is around 10% from fp32 precision to int8 precision.

[BernardinD]

Thanks @zldrobit, I was able to run the edgetpu compilation with your current conversion. Although when I try to run the outputted graph I'm given the error ValueError: Found too many dimensions in the input array of operation 'reshape'.

Any suggestions? The compilation also only gets 1 operation to get mapped

[zldrobit]

@BernardinD plz check your representative image forlder.
Have you provided enough images (defualt --ncalib is 100)?
If so, you may encounter a bug to be fixed in edge TPU, see google-coral/edgetpu#74.
People disscussed in that issue, and @Namburger helped to convert tflite model to edge TPU format manually to avoid the reshape problem.
BTW, could you share your edgetpu_compiler output log? I am curious about why other operations are not mapped.

[BernardinD]

@BernardinD plz check your representative image forlder.
Have you provided enough images (defualt --ncalib is 100)?
If so, you may encounter a bug to be fixed in edge TPU, see google-coral/edgetpu#74.
People disscussed in that issue, and @Namburger helped to convert tflite model to edge TPU format manually to avoid the reshape problem.
BTW, could you share your edgetpu_compiler output log? I am curious about why other operations are not mapped.

Sure no problem. Here's the log file:
best_yolov5s_results-int8_edgetpu.log

And I used the default 100 ncalib out of a folder of 1000 pictures. That should be fine, correct?

[minthawzin1995]

I managed to convert to int8 model with no issue. However, may i know if the same inference code (detect.py) could still be used to inference the int8 tflite model. I tried changing the input type to uint8 and the results seem to be wrong.

Thank you and really appreciate it for the conversion scripts.

[idenc]

The android demo does not seem to be runnable. A bunch of the drawables are missing, and the env package is missing

[glenn-jocher]

@zldrobit hi, thanks for this PR! Export to *.tflite is an amazing contribution, I know a lot of people have been asking for this functionality.

I think originally this PR featured only the export code (first 3 or 4 commits), and the accompanying android app was later added in commit 5.

For us to merge this into the repo we'd ideally want to strip this down to the minimum export functionality required as you had originally, as supporting and maintaining an android app is beyond the scope of the repo (even though it's an amazing contribution in and of itself).

EDIT: maybe the best way to move forward is to submit a new PR featuring only the export, or to reset the head to an earlier commit on the branch, whatever is easiest for you, let me know, thanks!

[zldrobit]

@BernardinD sorry for this late reply, I was on vocation.
That should be good enough to use 100 out of 1000 pictures.

[zldrobit]

@minthawzin1995 int8 quantization does not support some ops in Detect module, some code needs to be added in detect.py .

[zldrobit]

@idenc I'll check the android code again, thx for reporting.

[zldrobit]

@glenn-jocher I am planning to submit a new PR.
I think I should leave detect.py untouched and just add models/tf.py and other code for *.tflite export.
Do we need the detection code for *.tflite in the new PR?

[glenn-jocher]

@zldrobit yes, please submit a new PR with tf.py and associated export code, but without detect.py integration and without the /android folder. This will help us minimize keep code maintenance going forward.

[zldrobit]

Export and Detection (/w TFLite int8 inference) functionality and Android project has been integrated in a new branch https://github.com/zldrobit/yolov5/tree/tf-android

@minthawzin1995 Plz try the new branch and int8 inference with:

python3 detect.py --weights weights/yolov5s-int8.tflite --tfl-detect --tfl-int8

inference time is around 18s on CPU.

@idenc Sorry for my unfamiliar with Android Studio. I have added necessary files in the Android project.
Plz try the new branch.

[minthawzin1995]

Export and Detection (/w TFLite int8 inference) functionality and Android project has been integrated in a new branch https://github.com/zldrobit/yolov5/tree/tf-android

@minthawzin1995 Plz try the new branch and int8 inference with:

python3 detect.py --weights weights/yolov5s-int8.tflite --tfl-detect --tfl-int8

inference time is around 18s on CPU.

@idenc Sorry for my unfamiliar with Android Studio. I have added necessary files in the Android project.
Plz try the new branch.

@zldrobit Thank you for the fast response, I have tested the new update and the detection works perfectly now for the int-8 model.

[jsn5]

Hi @zldrobit, I'm getting different results on the android app compared to detect.py.

I converted yolov5s model to fp16 using the following command:

PYTHONPATH=. python models/tf.py --weights models/yolov5s.pt --cfg models/yolov5s.yaml --img 320

I setup the android app with the fp-16 model but on the app the results are much worse compared to when running inference using detect.py like this:

python detect.py --weights models/yolov5s-fp16.tflite --img-size 320

I've verified that the masks and output shapes are correct. I set SAVE_PREVIEW_BITMAP as true and got the images going to inference on the phone and tried them with detect.py and it works well. I've tried with the default coco model as well as a custom model. For the custom model, the results are much worse from the android app. Here are the results on the coco model:

Using android app:

Using detect.py:

Am I missing something?

[zldrobit]

@jsn5 I notice that there are difference between above and below pictures.
The below pictures have higher brightness and contrast.
Try changing light conditions while comparing, you'll get a more comprehensive result.

[jsn5]

@jsn5 I notice that there are difference between above and below pictures.
The below pictures have higher brightness and contrast.
Try changing light conditions while comparing, you'll get a more comprehensive result.

The pics look different because for the phone results I had to take a screenshot of the app. Regardless of lighting and contrast, the results seem very different overall. For a custom dataset that I trained yolov5s on, the detection on phone would miss many of the objects while taking the same preview image and running using detect.py is able to detect most of the objects in the scene.

I'll try to do some more tests and share it with you.

[glenn-jocher]

@jsn5 @zldrobit the differences may also be due to image transforms in Android. In iOS, there are a few options for how the camera image is transformed into the model input shape. iDetection iOS app currently uses scaleFill, which distorts the aspect ratio slightly, so we have a TODO item to move to scaleFit (same as YOLOv5 PyTorch dataloader), but this is not as simple as it sounds because the bounding box reconstruction is dependent on the exact input transforms, so modifications at the input must be accompanied by modification of the box reconstruction code as well.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[zldrobit]

Since this PR is closed, any further questions about TensorFlow/TFLite export may be addressed in #1127.

[glenn-jocher]

@zldrobit I just saw that the bot closed the PR, I'll reopen it again. Is the Android app here maintained and working correctly with models exported from #1127?

[zldrobit]

@glenn-jocher This PR is out-of-date, and its Android project is not compatible with the models exported from #1127.
#1127 has downstream branches tf-android and tf-android-tfl-detect.
They have Android project in them and are described in the first comment of #1127.
The only advantage of this PR is to use TensorFlow 1.X.
Otherwise, we could close this PR and move on to #1127.

[glenn-jocher]

@zldrobit ok makes sense! Will close.

[NandhiniN85]

@BernardinD I use -a with edgetpu_compiler and I got 133 ops run on Edge TPU.

By the way, I am using the newest version of edgetpu_compiler.
I am about to get an Edge TPU, but now I am unable to do a speed test.

@gdebrecz I had the same issue earlier. Refer to the comment above

@BernardinD We were able to compile the quantized model without -a and got TPU ops as 1. But when we used -a, got "Internal compiler error. Aborting!". Any suggestion to resolve it?. Thanks!

[NandhiniN85]

@BernardinD I use -a with edgetpu_compiler and I got 133 ops run on Edge TPU.

By the way, I am using the newest version of edgetpu_compiler.
I am about to get an Edge TPU, but now I am unable to do a speed test.

@gdebrecz I had the same issue earlier. Refer to the comment above

@BernardinD We were able to compile the quantized model without -a and got TPU ops as 1. But when we used -a, got "Internal compiler error. Aborting!". Any suggestion to resolve it?. Thanks!

here is the screenshot of the logfile,

[zldrobit]

@NandhiniN85 Could you share more information to inspect the problem, such as the model size, the number of classes, and the network structure (any change to yolov5*.yaml)?

[NandhiniN85]

@NandhiniN85 Could you share more information to inspect the problem, such as the model size, the number of classes, and the network structure (any change to yolov5*.yaml)?

@zldrobit I have used yolov5s and the number of classes is 1. The only modification made to the config file is the nc=1. The input resolution is 640 x 640. Every image frame has around 100 objects at least, the size of the objects are very small (for the same reason used 640 x 640). Altogether the model has been trained on with 250k objects.

[zldrobit]

@NandhiniN85 640x640 may be too large for YOLOv5 EdgeTPU model.
Could you try exporting a smaller EdgeTPU model, e.g. with 320x320 resolution?

[NandhiniN85]

@NandhiniN85 640x640 may be too large for YOLOv5 EdgeTPU model.
Could you try exporting a smaller EdgeTPU model, e.g. with 320x320 resolution?

@zldrobit Thanks for the details! I will try with a smaller resolution and check out the accuracy. Thanks!

[ziyuwzf]

After i converted .pt to .tflite successfully,i runed the command as below:
D:/PycharmProjects/yolov5-tf-android/detect.py --weights models/yolov5s-fp16.tflite --img 320 --save-crop
Namespace(agnostic_nms=False, augment=False, classes=None, conf_thres=0.25, device='', exist_ok=False, hide_conf=False, hide_labels=False, img_size=[320, 320], iou_thres=0.45, line_thickness=3, max_det=1000, name='exp', nosave=False, project='runs/detect', save_conf=False, save_crop=True, save_txt=False, source='data/images', tfl_int8=False, update=False, view_img=False, weights=['models/yolov5s-fp16.tflite'])
YOLOv5 2021-6-17 torch 1.8.0+cpu CPU

image 1/2 D:\PycharmProjects\yolov5-tf-android\data\images\bus.jpg: 320x320 4 persons, 1 bus, Done. (18.252s)
image 2/2 D:\PycharmProjects\yolov5-tf-android\data\images\zidane.jpg: 320x320 3 persons, 2 ties, Done. (18.273s)
Results saved to runs\detect\exp5
Done. (36.643s)

Process finished with exit code 0
why .tflite runs so slower than .pt?i missed something?

[zldrobit]

@ziyuwzf If you run detection on a PC, TFLite could not use GPU. So it runs slower than PyTorch. You could also run your detection on Google Colab to confirm the performance issue. I ran detection with a 20-thread Intel i9-9820X CPU, and the results are

On what device you ran the detection?

[rashmi-learning]

While exporting the weights getting error:

File "models/tf.py", line 453, in
converter = tf.lite.TFLiteConverter.from_keras_model(keras_model)
NameError: name 'keras_model' is not defined

[glenn-jocher]

@rashmi-learning exports are consolidated in export.py now. For TFLite:

python export.py --weights yolov5s.pt --include tflite