Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to use tensorflow model in Unity inference engine? #1825

Closed
Amanpradhan opened this issue Mar 15, 2019 · 15 comments
Closed

How to use tensorflow model in Unity inference engine? #1825

Amanpradhan opened this issue Mar 15, 2019 · 15 comments
Assignees
Labels
discussion Issue contains general discussion. help-wanted Issue contains request for help or information.

Comments

@Amanpradhan
Copy link

Amanpradhan commented Mar 15, 2019

I am planning to make an object detector using mobilenet ssd model in unity. How can I use that model in unity inference engine?

@ervteng
Copy link
Collaborator

ervteng commented Mar 15, 2019

Hi @Amanpradhan, the place to start would be the tensorflow_to_barracuda.py script included in this repo. Note that Unity Inference Engine (Barracuda) is still experimental outside of ml-agents models, and may not work as expected. You can see inside that Python script which Tensorflow operations are supported by Barracuda.

Some documentation here: https://github.com/Unity-Technologies/ml-agents/blob/e3b86a27a2547cdd177bf9e848b4337aa71b887a/UnitySDK/Assets/ML-Agents/Plugins/Barracuda.Core/Barracuda.md

Also see: #1826

@ervteng ervteng added help-wanted Issue contains request for help or information. discussion Issue contains general discussion. labels Mar 15, 2019
@dahburj
Copy link

dahburj commented Mar 20, 2019

I converted a PoseNet python based model and integrated it using @infocom-tpo's PoseNet Unity repo as a guide for the translation into Tensorflowc#, which I translated into Barracuda, rather horribly, and I would love some help! It's working alot better than using Tensorflow sharp, I'm getting an ok framerate in the editor (10-25 FPS Macbook book pro mid-2015), Android seems to have way too many problems, not just with Barracuda. Does Barracuda automatically handle the multi-threading, or should we be assigning Jobs/ handling the threads ourselves? When it actually runs on Android, I get about 3 FPS before it crashes. Every error I fix somehow creates others. I tried 2018.3.8, 2018.3.9, 2019 beta, 2019 beta was the best for android. I actually got somewhere debugging, figured out for some reason my class wasn't being called into the scene at start. About to test on iOS after a few more jabs at this android problem, any more Barracuda documentation or guidance would be greatly appreciated.

@ervteng
Copy link
Collaborator

ervteng commented Mar 20, 2019

Great to see a lot of interest in using Barracuda with other models outside of ml-agents! I've added @mantasp who might be able to help.

@mantasp
Copy link
Member

mantasp commented Mar 27, 2019

@dahburj that's currently a bit outside of our tested/optimized area, but we would be interested to have closer look at your project. Which devices did you use for testing performance and what kind of input resolution did you use?

@dahburj
Copy link

dahburj commented Mar 29, 2019

@mantasp yeah sure here's a repo of what I'm working on. I explain what's going on in the ReadMe.

I tried 224x224, 256x256, 199x199, 179x179, 512x512, I changed a lot since I commented here, the main performance boost came from setting/reading the pixels from a GPU queue. The frame rate in the editor is insane, sometimes saw 90-140 FPS playing with the frame rate inference rate &/or gpu buffer rate. I'm wondering if I can convert the tensors to native arrays or containers and use them in burst jobs (or any jobs) using ECS. I began trying but this is my first time playing with ECS so it's really rough. As far as devices, Samsung Galaxy S6 Edge International Model, iPad Pro 5th Generation, and Macbook Pro Mid-2015.

@Amanpradhan
Copy link
Author

Amanpradhan commented Apr 12, 2019

I am trying to convert my protobuf file to use in unity inference engine. I ran:python tensorflow_to_barracuda.py obj.pb nn
but I am getting the error:
[tensors[0].data[0], 1, 1, tensors[0].data[1]]
IndexError: index 1 is out of bounds for axis 0 with size 1

@thomasvanta
Copy link

thomasvanta commented Apr 18, 2019

I'm trying to get the original model from Posenet to work on Barracuda as well.
I have now a .pb model downloaded with posenet-python but when trying to use tensorflow_to_barracuda I get this:
IGNORED: SpaceToBatchND unknown layer
Traceback (most recent call last):
File "tensorflow_to_barracuda.py", line 23, in
tf2bc.convert(args.source_file, args.target_file, args.trim_unused_by_output, args)
File "C:\Users\vanta\Desktop\ml-agents-0.8.1\ml-agents\mlagents\trainers\tensorflow_to_barracuda.py", line 940, in convert
process_model(i_model, args)
File "C:\Users\vanta\Desktop\ml-agents-0.8.1\ml-agents\mlagents\trainers\tensorflow_to_barracuda.py", line 894, in process_model
process_layer(node, o_context, args)
File "C:\Users\vanta\Desktop\ml-agents-0.8.1\ml-agents\mlagents\trainers\tensorflow_to_barracuda.py", line 746, in process_layer
shapes = klass.out_shapes([x.shape for x in o_l.tensors])
File "C:\Users\vanta\Desktop\ml-agents-0.8.1\ml-agents\mlagents\trainers\tensorflow_to_barracuda.py", line 93, in
[s[0][0], s[0][1], s[0][3], s[0][2]], # K TF:[H, W, in_channels, channel_multiplier] => [H, W, 1, in_channels]
IndexError: list index out of range

Can it be that Barracuda doesnt have image preprocessor implemented yet or something alike?
@dahburj how did you get your model to work with barracuda?

@dahburj
Copy link

dahburj commented Apr 24, 2019

@thomasvanta I was getting an "UpSample: unknown layer" error, or something along those lines, when I tried to download Tensorflow's Tensorflow-Lite PoseNet, if you click on the repo I linked it explains the steps in detail. There seems to be a lot of issues converting these models to .bytes with other models I've tried (layers being skipped over, labelled-unknown, etc.), but I got it running around >60 FPS now standing. Oddly enough, a HUGE performance increase came when I placed "Yield Return New Wait For Seconds(float)" with the parameter float ranging between (around).025f and 1f, in between all the Barracuda worker/tensor fetch and dispose calls in the Coroutine that is being called. I also see random memory leaks on iOS, but they only exist when the app is first installed on, if I quit it and come back we're back to smooth pose tracking. I think it can be made ALOT faster, I've been kind of shooting in the dark and the performance has exceeded my expectations, but more Barracuda documentation or information is needed, I have high hopes for what's possible but constantly feel like I'm stumbling around a dark room without a flash-light. I mean is there a list of supported Ops? Can we write custom ops or maybe even use it in conjunction with tensorflow-lite running separately on a native thread(s)? Does Barracuda have an image preprocessor?

@sandeepkiks
Copy link

sandeepkiks commented May 16, 2019

I too was looking into "how to load tensorflow model in Unity", tried @dahburj repo but did not worked for some reason. What I did is just cloned the repo and played the scene, poses[] array was empty :( I might be missing something (@dahburj if possible can you please help me?)
Anyhow, started with @infocom-tpo's PoseNet Unity repo which runs at about 35 FPS (~39ms) on my i7 laptop. Had to make some optimisations else it was running around 12 FPS (~1400ms). Had to get rid of Array.Reverse() which saved a lot of CPU time. Also removed GLRenderer as I just needed keypoints and no visual points. It also runs on Android at around 10 FPS without crash. Was not able to run on WebGL though (which I need the most)
Not super happy with the overall results but it is ok for now.
Hope this might help someone.

@vinay5395b
Copy link

vinay5395b commented Jun 4, 2019

Hi @sandeepkiks! I was wondering if I could somehow get your code which is running fine with the Unity Inference engine integration. I am facing issues trying to convert the PoseNet with tensorflow code to support the inference engine.
Thanks!

@sandeepkiks
Copy link

sandeepkiks commented Jun 5, 2019

Hello @vinay5395b I am not using Unity Inference engine, instead using PoseNet Unity repo.

@vinay5395b
Copy link

vinay5395b commented Jun 5, 2019

@sandeepkiks I am using the same repo. Are you using the old packages for ML in Unity which support tensorflow? I just want to get it running on my system as I want the results for further processing.
Thanks!

@xiaomaogy
Copy link
Contributor

xiaomaogy commented Jul 9, 2019

Thank you for the discussion. We are closing this issue due to inactivity. Feel free to reopen it if you’d like to continue the discussion though.

@AkhilRaja
Copy link

AkhilRaja commented Jul 23, 2019

Hi guys. I am also trying to convert a posenet.pb to a .nn file using the tensorflow_to_barracuda.py. I get the following error

Converting openpose.pb to Models/openpose.bytes
IGNORED: Shape unknown layer
IGNORED: Pack unknown layer
Traceback (most recent call last):
File "tensorflow_to_barracuda.py", line 26, in
tf2bc.convert(args.source_file, args.target_file, args.trim_unused_by_output, args)
File "/Users/apappu/Downloads/ml-agents-master/ml-agents/mlagents/trainers/tensorflow_to_barracuda.py", line 1537, in convert
i_model, args
File "/Users/apappu/Downloads/ml-agents-master/ml-agents/mlagents/trainers/tensorflow_to_barracuda.py", line 1366, in process_model
nodes, var_tensors, const_tensors, o_context
File "/Users/apappu/Downloads/ml-agents-master/ml-agents/mlagents/trainers/tensorflow_to_barracuda.py", line 472, in
inputs[0],

@github-actions
Copy link

github-actions bot commented Jan 31, 2021

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jan 31, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
discussion Issue contains general discussion. help-wanted Issue contains request for help or information.
Projects
None yet
Development

No branches or pull requests

9 participants