How to use tensorflow model in Unity inference engine?

[Amanpradhan]

I am planning to make an object detector using mobilenet ssd model in unity. How can I use that model in unity inference engine?

[ervteng]

Hi @Amanpradhan, the place to start would be the tensorflow_to_barracuda.py script included in this repo. Note that Unity Inference Engine (Barracuda) is still experimental outside of ml-agents models, and may not work as expected. You can see inside that Python script which Tensorflow operations are supported by Barracuda.

Some documentation here: https://github.com/Unity-Technologies/ml-agents/blob/e3b86a27a2547cdd177bf9e848b4337aa71b887a/UnitySDK/Assets/ML-Agents/Plugins/Barracuda.Core/Barracuda.md

Also see: #1826

[dahburj]

I converted a PoseNet python based model and integrated it using @infocom-tpo's PoseNet Unity repo as a guide for the translation into Tensorflowc#, which I translated into Barracuda, rather horribly, and I would love some help! It's working alot better than using Tensorflow sharp, I'm getting an ok framerate in the editor (10-25 FPS Macbook book pro mid-2015), Android seems to have way too many problems, not just with Barracuda. Does Barracuda automatically handle the multi-threading, or should we be assigning Jobs/ handling the threads ourselves? When it actually runs on Android, I get about 3 FPS before it crashes. Every error I fix somehow creates others. I tried 2018.3.8, 2018.3.9, 2019 beta, 2019 beta was the best for android. I actually got somewhere debugging, figured out for some reason my class wasn't being called into the scene at start. About to test on iOS after a few more jabs at this android problem, any more Barracuda documentation or guidance would be greatly appreciated.

[ervteng]

Great to see a lot of interest in using Barracuda with other models outside of ml-agents! I've added @mantasp who might be able to help.

[mantasp]

@dahburj that's currently a bit outside of our tested/optimized area, but we would be interested to have closer look at your project. Which devices did you use for testing performance and what kind of input resolution did you use?

[dahburj]

@mantasp yeah sure here's a repo of what I'm working on. I explain what's going on in the ReadMe.

I tried 224x224, 256x256, 199x199, 179x179, 512x512, I changed a lot since I commented here, the main performance boost came from setting/reading the pixels from a GPU queue. The frame rate in the editor is insane, sometimes saw 90-140 FPS playing with the frame rate inference rate &/or gpu buffer rate. I'm wondering if I can convert the tensors to native arrays or containers and use them in burst jobs (or any jobs) using ECS. I began trying but this is my first time playing with ECS so it's really rough. As far as devices, Samsung Galaxy S6 Edge International Model, iPad Pro 5th Generation, and Macbook Pro Mid-2015.

[Amanpradhan]

I am trying to convert my protobuf file to use in unity inference engine. I ran:python tensorflow_to_barracuda.py obj.pb nn
but I am getting the error:
[tensors[0].data[0], 1, 1, tensors[0].data[1]]
IndexError: index 1 is out of bounds for axis 0 with size 1

[thomasvanta]

I'm trying to get the original model from Posenet to work on Barracuda as well.
I have now a .pb model downloaded with posenet-python but when trying to use tensorflow_to_barracuda I get this:
IGNORED: SpaceToBatchND unknown layer
Traceback (most recent call last):
File "tensorflow_to_barracuda.py", line 23, in
tf2bc.convert(args.source_file, args.target_file, args.trim_unused_by_output, args)
File "C:\Users\vanta\Desktop\ml-agents-0.8.1\ml-agents\mlagents\trainers\tensorflow_to_barracuda.py", line 940, in convert
process_model(i_model, args)
File "C:\Users\vanta\Desktop\ml-agents-0.8.1\ml-agents\mlagents\trainers\tensorflow_to_barracuda.py", line 894, in process_model
process_layer(node, o_context, args)
File "C:\Users\vanta\Desktop\ml-agents-0.8.1\ml-agents\mlagents\trainers\tensorflow_to_barracuda.py", line 746, in process_layer
shapes = klass.out_shapes([x.shape for x in o_l.tensors])
File "C:\Users\vanta\Desktop\ml-agents-0.8.1\ml-agents\mlagents\trainers\tensorflow_to_barracuda.py", line 93, in
[s[0][0], s[0][1], s[0][3], s[0][2]], # K TF:[H, W, in_channels, channel_multiplier] => [H, W, 1, in_channels]
IndexError: list index out of range

Can it be that Barracuda doesnt have image preprocessor implemented yet or something alike?
@dahburj how did you get your model to work with barracuda?

[dahburj]

@thomasvanta I was getting an "UpSample: unknown layer" error, or something along those lines, when I tried to download Tensorflow's Tensorflow-Lite PoseNet, if you click on the repo I linked it explains the steps in detail. There seems to be a lot of issues converting these models to .bytes with other models I've tried (layers being skipped over, labelled-unknown, etc.), but I got it running around >60 FPS now standing. Oddly enough, a HUGE performance increase came when I placed "Yield Return New Wait For Seconds(float)" with the parameter float ranging between (around).025f and 1f, in between all the Barracuda worker/tensor fetch and dispose calls in the Coroutine that is being called. I also see random memory leaks on iOS, but they only exist when the app is first installed on, if I quit it and come back we're back to smooth pose tracking. I think it can be made ALOT faster, I've been kind of shooting in the dark and the performance has exceeded my expectations, but more Barracuda documentation or information is needed, I have high hopes for what's possible but constantly feel like I'm stumbling around a dark room without a flash-light. I mean is there a list of supported Ops? Can we write custom ops or maybe even use it in conjunction with tensorflow-lite running separately on a native thread(s)? Does Barracuda have an image preprocessor?

[sandeepkiks]

I too was looking into "how to load tensorflow model in Unity", tried @dahburj repo but did not worked for some reason. What I did is just cloned the repo and played the scene, poses[] array was empty :( I might be missing something (@dahburj if possible can you please help me?)
Anyhow, started with @infocom-tpo's PoseNet Unity repo which runs at about 35 FPS (~39ms) on my i7 laptop. Had to make some optimisations else it was running around 12 FPS (~1400ms). Had to get rid of Array.Reverse() which saved a lot of CPU time. Also removed GLRenderer as I just needed keypoints and no visual points. It also runs on Android at around 10 FPS without crash. Was not able to run on WebGL though (which I need the most)
Not super happy with the overall results but it is ok for now.
Hope this might help someone.

[vinay5395b]

Hi @sandeepkiks! I was wondering if I could somehow get your code which is running fine with the Unity Inference engine integration. I am facing issues trying to convert the PoseNet with tensorflow code to support the inference engine.
Thanks!

[sandeepkiks]

Hello @vinay5395b I am not using Unity Inference engine, instead using PoseNet Unity repo.