Any way to load or convert a .pth?

[bassmit]

Hi, sorry to ask this here, but I can find very little on google. We have a trained agent that we would like to run in unity. Currently this is trained with pytorch and is in the form of a .pth file. Is there any way I can import this in to unity, perhaps by converting it to a .nn file?

Thanks, Bas

[sam598]

You might be able to convert your pytorch model to ONNX and then to Barracuda:
https://github.com/Unity-Technologies/ml-agents/blob/master/UnitySDK/Assets/ML-Agents/Plugins/Barracuda.Core/Barracuda.md

It looks like the Barracuda converter is still a work in progress and only supports some layers/nodes.

EDIT

It looks like the script "onnx_to_barracuda.py" mentioned in that readme has not been written yet. Sorry for the confusion.

[bassmit]

Thanks, I'll check it out

…

On Tue, 5 Mar 2019, 18:35 sam598, ***@***.***> wrote: You might be able to convert your pytorch model to ONNX and then to Barracuda: https://github.com/Unity-Technologies/ml-agents/blob/master/UnitySDK/Assets/ML-Agents/Plugins/Barracuda.Core/Barracuda.md It looks like the Barracuda converter is still a work in progress and only supports some layers/nodes. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1789 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEW2W5gbknH22sF0cDeUGI-2vedHO5heks5vTqrKgaJpZM4beaiG> .

[bassmit]

EDIT

It looks like the script "onnx_to_barracuda.py" mentioned in that readme has not been written yet. Sorry for the confusion.

Ok no worries, I found this script tensorflow_to_barracuda.py, I'll try to use mmdnn to convert .pth to tensorflow then run that script. Fwiw I think the md linked is out of date, the .bytes format has been dropped afaict

[mantasp]

Hi @bassmit!
At the moment we do not support Pytorch models, but in the future we might support it via ONNX. Though we are interested to hear more details on what kind of models would you like to run in Unity?

[bassmit]

Hi Mantas, I tried getting to a tensorflow model through ONNX, but that was way too optimistic :) We have since abandoned pytorch, and Im currently looking through the ml-agent python scripts to figure out how to convert our agents to .nn's. I'll share some more details over email

[mantasp]

Ok, lets move it to the email! :)

[kumarict]

Hi! Are there any updates on this? I have generated pytorch models as .pt or .pth files and would like to load them into Unity. It would be great to convert them into .nn files or import them directly or at least know when they will be supported. Thank you.

[bassmit]

We convert to pytorch to onnx https://pytorch.org/docs/stable/onnx.html then to nn with onnx_to_barracuda https://github.com/mantasp/barracuda-release/tree/release/0.2.4/Tools

…

On Fri, Aug 16, 2019 at 8:36 PM kumarict ***@***.***> wrote: Hi! Are there any updates on this? I have generated pytorch models as .pt or .pth files and would like to load them into Unity. It would be great to convert them into .nn files or import them directly or at least know when they will be supported. Thank you. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1789?email_source=notifications&email_token=ABC3MW6A63K3NZHFPELHJSDQE3XUVA5CNFSM4G3ZVCDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4PMGYI#issuecomment-522109793>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABC3MWYQET4TV4HAXGNFLODQE3XUVANCNFSM4G3ZVCDA> .

[ghost]

Hello @bassmit !

I followed your steps but when I have to drag my model into the agent in Unity I have this following error:

"NullReferenceException: Object reference not set to an instance of an object"

Versions:
· Windows 10
· ML-Agents 0.13.1
· Unity 2018.4.16f1
· Barracuda 0.3.2
*but to convert from onnx to nn I use the scripts of barracuda 0.2.4 because if I use 0.3.2 or higher I have this error:

Traceback (most recent call last):
File "onnx_to_barracuda.py", line 20, in
onnx2bc.convert(args.source_file, args.target_file, args.trim_unused_by_output, args)
File "C:\Users\Ocin\Desktop\pytorch to unity model\onnx_to_barracuda.py", line 570, in convert
lambda tensor: get_tensor_data(tensor))
File "C:\Users\Ocin\Desktop\pytorch to unity model\barracuda.py", line 221, in setup_constants
data = np.reshape(get_tensor_data_lambda(tensor), shape).astype(np.float32))]
File "C:\Users\Ocin\Desktop\pytorch to unity model\onnx_to_barracuda.py", line 570, in
lambda tensor: get_tensor_data(tensor))
File "C:\Users\Ocin\Desktop\pytorch to unity model\onnx_to_barracuda.py", line 336, in get_tensor_data
floats = struct.unpack('<'+str(int(elems))+'f', tensor.raw_data)
struct.error: unpack requires a buffer of 4 bytes

Did you do any more step?

Thank you !

[chriselion]

I'm closing and locking this issue in favor of #3398