Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to use tensorflow model in Unity inference engine? #1825

Closed
Amanpradhan opened this issue Mar 15, 2019 · 14 comments
Closed

How to use tensorflow model in Unity inference engine? #1825

Amanpradhan opened this issue Mar 15, 2019 · 14 comments
Assignees

Comments

@Amanpradhan
Copy link

@Amanpradhan Amanpradhan commented Mar 15, 2019

I am planning to make an object detector using mobilenet ssd model in unity. How can I use that model in unity inference engine?

@ervteng

This comment has been minimized.

Copy link
Collaborator

@ervteng ervteng commented Mar 15, 2019

Hi @Amanpradhan, the place to start would be the tensorflow_to_barracuda.py script included in this repo. Note that Unity Inference Engine (Barracuda) is still experimental outside of ml-agents models, and may not work as expected. You can see inside that Python script which Tensorflow operations are supported by Barracuda.

Some documentation here: https://github.com/Unity-Technologies/ml-agents/blob/e3b86a27a2547cdd177bf9e848b4337aa71b887a/UnitySDK/Assets/ML-Agents/Plugins/Barracuda.Core/Barracuda.md

Also see: #1826

@dahburj

This comment has been minimized.

Copy link

@dahburj dahburj commented Mar 20, 2019

I converted a PoseNet python based model and integrated it using @infocom-tpo's PoseNet Unity repo as a guide for the translation into Tensorflowc#, which I translated into Barracuda, rather horribly, and I would love some help! It's working alot better than using Tensorflow sharp, I'm getting an ok framerate in the editor (10-25 FPS Macbook book pro mid-2015), Android seems to have way too many problems, not just with Barracuda. Does Barracuda automatically handle the multi-threading, or should we be assigning Jobs/ handling the threads ourselves? When it actually runs on Android, I get about 3 FPS before it crashes. Every error I fix somehow creates others. I tried 2018.3.8, 2018.3.9, 2019 beta, 2019 beta was the best for android. I actually got somewhere debugging, figured out for some reason my class wasn't being called into the scene at start. About to test on iOS after a few more jabs at this android problem, any more Barracuda documentation or guidance would be greatly appreciated.

@ervteng

This comment has been minimized.

Copy link
Collaborator

@ervteng ervteng commented Mar 20, 2019

Great to see a lot of interest in using Barracuda with other models outside of ml-agents! I've added @mantasp who might be able to help.

@mantasp

This comment has been minimized.

Copy link
Member

@mantasp mantasp commented Mar 27, 2019

@dahburj that's currently a bit outside of our tested/optimized area, but we would be interested to have closer look at your project. Which devices did you use for testing performance and what kind of input resolution did you use?

@dahburj

This comment has been minimized.

Copy link

@dahburj dahburj commented Mar 29, 2019

@mantasp yeah sure here's a repo of what I'm working on. I explain what's going on in the ReadMe.

I tried 224x224, 256x256, 199x199, 179x179, 512x512, I changed a lot since I commented here, the main performance boost came from setting/reading the pixels from a GPU queue. The frame rate in the editor is insane, sometimes saw 90-140 FPS playing with the frame rate inference rate &/or gpu buffer rate. I'm wondering if I can convert the tensors to native arrays or containers and use them in burst jobs (or any jobs) using ECS. I began trying but this is my first time playing with ECS so it's really rough. As far as devices, Samsung Galaxy S6 Edge International Model, iPad Pro 5th Generation, and Macbook Pro Mid-2015.

@Amanpradhan

This comment has been minimized.

Copy link
Author

@Amanpradhan Amanpradhan commented Apr 12, 2019

I am trying to convert my protobuf file to use in unity inference engine. I ran:python tensorflow_to_barracuda.py obj.pb nn
but I am getting the error:
[tensors[0].data[0], 1, 1, tensors[0].data[1]]
IndexError: index 1 is out of bounds for axis 0 with size 1

@thomasvanta

This comment has been minimized.

Copy link

@thomasvanta thomasvanta commented Apr 18, 2019

I'm trying to get the original model from Posenet to work on Barracuda as well.
I have now a .pb model downloaded with posenet-python but when trying to use tensorflow_to_barracuda I get this:
IGNORED: SpaceToBatchND unknown layer
Traceback (most recent call last):
File "tensorflow_to_barracuda.py", line 23, in
tf2bc.convert(args.source_file, args.target_file, args.trim_unused_by_output, args)
File "C:\Users\vanta\Desktop\ml-agents-0.8.1\ml-agents\mlagents\trainers\tensorflow_to_barracuda.py", line 940, in convert
process_model(i_model, args)
File "C:\Users\vanta\Desktop\ml-agents-0.8.1\ml-agents\mlagents\trainers\tensorflow_to_barracuda.py", line 894, in process_model
process_layer(node, o_context, args)
File "C:\Users\vanta\Desktop\ml-agents-0.8.1\ml-agents\mlagents\trainers\tensorflow_to_barracuda.py", line 746, in process_layer
shapes = klass.out_shapes([x.shape for x in o_l.tensors])
File "C:\Users\vanta\Desktop\ml-agents-0.8.1\ml-agents\mlagents\trainers\tensorflow_to_barracuda.py", line 93, in
[s[0][0], s[0][1], s[0][3], s[0][2]], # K TF:[H, W, in_channels, channel_multiplier] => [H, W, 1, in_channels]
IndexError: list index out of range

Can it be that Barracuda doesnt have image preprocessor implemented yet or something alike?
@dahburj how did you get your model to work with barracuda?

@dahburj

This comment has been minimized.

Copy link

@dahburj dahburj commented Apr 24, 2019

@thomasvanta I was getting an "UpSample: unknown layer" error, or something along those lines, when I tried to download Tensorflow's Tensorflow-Lite PoseNet, if you click on the repo I linked it explains the steps in detail. There seems to be a lot of issues converting these models to .bytes with other models I've tried (layers being skipped over, labelled-unknown, etc.), but I got it running around >60 FPS now standing. Oddly enough, a HUGE performance increase came when I placed "Yield Return New Wait For Seconds(float)" with the parameter float ranging between (around).025f and 1f, in between all the Barracuda worker/tensor fetch and dispose calls in the Coroutine that is being called. I also see random memory leaks on iOS, but they only exist when the app is first installed on, if I quit it and come back we're back to smooth pose tracking. I think it can be made ALOT faster, I've been kind of shooting in the dark and the performance has exceeded my expectations, but more Barracuda documentation or information is needed, I have high hopes for what's possible but constantly feel like I'm stumbling around a dark room without a flash-light. I mean is there a list of supported Ops? Can we write custom ops or maybe even use it in conjunction with tensorflow-lite running separately on a native thread(s)? Does Barracuda have an image preprocessor?

@sandeepkiks

This comment has been minimized.

Copy link

@sandeepkiks sandeepkiks commented May 16, 2019

I too was looking into "how to load tensorflow model in Unity", tried @dahburj repo but did not worked for some reason. What I did is just cloned the repo and played the scene, poses[] array was empty :( I might be missing something (@dahburj if possible can you please help me?)
Anyhow, started with @infocom-tpo's PoseNet Unity repo which runs at about 35 FPS (~39ms) on my i7 laptop. Had to make some optimisations else it was running around 12 FPS (~1400ms). Had to get rid of Array.Reverse() which saved a lot of CPU time. Also removed GLRenderer as I just needed keypoints and no visual points. It also runs on Android at around 10 FPS without crash. Was not able to run on WebGL though (which I need the most)
Not super happy with the overall results but it is ok for now.
Hope this might help someone.

@vinay5395b

This comment has been minimized.

Copy link

@vinay5395b vinay5395b commented Jun 4, 2019

Hi @sandeepkiks! I was wondering if I could somehow get your code which is running fine with the Unity Inference engine integration. I am facing issues trying to convert the PoseNet with tensorflow code to support the inference engine.
Thanks!

@sandeepkiks

This comment has been minimized.

Copy link

@sandeepkiks sandeepkiks commented Jun 5, 2019

Hello @vinay5395b I am not using Unity Inference engine, instead using PoseNet Unity repo.

@vinay5395b

This comment has been minimized.

Copy link

@vinay5395b vinay5395b commented Jun 5, 2019

@sandeepkiks I am using the same repo. Are you using the old packages for ML in Unity which support tensorflow? I just want to get it running on my system as I want the results for further processing.
Thanks!

@xiaomaogy

This comment has been minimized.

Copy link
Collaborator

@xiaomaogy xiaomaogy commented Jul 9, 2019

Thank you for the discussion. We are closing this issue due to inactivity. Feel free to reopen it if you’d like to continue the discussion though.

@xiaomaogy xiaomaogy closed this Jul 9, 2019
@AkhilRaja

This comment has been minimized.

Copy link

@AkhilRaja AkhilRaja commented Jul 23, 2019

Hi guys. I am also trying to convert a posenet.pb to a .nn file using the tensorflow_to_barracuda.py. I get the following error

Converting openpose.pb to Models/openpose.bytes
IGNORED: Shape unknown layer
IGNORED: Pack unknown layer
Traceback (most recent call last):
File "tensorflow_to_barracuda.py", line 26, in
tf2bc.convert(args.source_file, args.target_file, args.trim_unused_by_output, args)
File "/Users/apappu/Downloads/ml-agents-master/ml-agents/mlagents/trainers/tensorflow_to_barracuda.py", line 1537, in convert
i_model, args
File "/Users/apappu/Downloads/ml-agents-master/ml-agents/mlagents/trainers/tensorflow_to_barracuda.py", line 1366, in process_model
nodes, var_tensors, const_tensors, o_context
File "/Users/apappu/Downloads/ml-agents-master/ml-agents/mlagents/trainers/tensorflow_to_barracuda.py", line 472, in
inputs[0],

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
9 participants
You can’t perform that action at this time.