Creating a UWP build of a Unity ML Agents project

[ActiveNick]

I noticed that the only Tensorflow plugins provided for Unity ML Agents projects are for standalone/editor, iOS and Android. That said, I started exploring the possibility of producing a UWP build to support deployment to the Microsoft Store on Windows or Xbox One, and more specifically for Windows Mixed Reality projects (VR or HoloLens).

Looking at the TensorflowSharp package on Nuget, I see that versions 1.5 and 1.7 also include a .NET Standard 2.0 DLL, so I downloaded this package and imported it in my Unity ML Agent project as a WSA plugin only. Unless told otherwise, I am running under the assumption that the TF# 1.7 wrapper DLL can be used on top of TF 1.4 (which is what I am using).

I also copied the following files and configured them as WSA plugins since I assume that TF# also needs them as dependencies, just like the desktop/standalone plugin.
liblibtensorflow.so
libtensorflow.dll
libtensorflow.bundle

I switched my build target to UWP, and set the scripting backend to IL2CPP since that is the only backend that supports .NET Standard 2.0 API compatibility (to the best of my knowledge).

Since the current version of Newtonsoft JSON.NET packaged in Unity ML Agents is for .NET 4.6, I downloaded the Nuget package from https://www.nuget.org/packages/Newtonsoft.Json/11.0.2 and used the .NET Standard 2.0 DLL configured as a plugin in Unity for WSA. I changed the other DLL from all platforms to individually checking each platform except WSA.

I was able to produce a UWP build in Unity and successfully compile the UWP VC++ project in VS2017. The project runs but I get the following error at runtime:
DllNotFoundException: Unable to load DLL 'libtensorflow': the specified module could not be found.
(see attached screenshot)

This is where my limited knowledge of C++ and IL2CPP fails me (which is why I love using C# in Unity). The issue is probably tied to the libtensorflow.dll being a Win32 DLL and calling it from UWP isn't trivial. I'll start exploring there in the TF# source. If anyone has recommended options to fix this and produce a working UWP build, please let me know what I can try.

[vladimiroster]

Hi @ActiveNick . Unfortunately, TF# only supports a certain number of platforms. We are working on adding support for other platforms via various internal and external libraries. Specifically, for UWP, we are working on an experimental use of Windows ML (https://docs.microsoft.com/en-us/windows/uwp/machine-learning/). I don't have a date for you, but this is being actively worked on.
In the mean time, perhaps someone can suggest a way to tweak the TF# DLL to work in UWP.

[b-fazamb]

G'day,
I have actually tried building for UWP using ml-agents 0.7 (beta).
Given the dependency on TennsforFlowSharp is gone (due to the introduction of the new inference engine), I was hoping to manage getting an UWP build going myself.

However, I am running into different build issues: a missing API in UWP (System.Runtime.Serialization.SerializationInfo) plus some more errors probably caused by the specific project (mis)configuration.

Would you like me to open a new issue or to keep contributing to this, instead?
Hopefully, there may be a way to patch/fix this to get a fully functioning UWP target?

[vincentpierre]

Hi @b-fazamb,
I was able to reproduce your bug. Thank you for trying ML-Agents out on UWP. It seems you are using .NET Scripting Backend in your Player Settings. I was able to build fine when using IL2CPP. Could you give it a try ?

[b-fazamb]

Hi @vincentpierre , I had tried IL2CPP first, in fact.

Building with UWP/IL2CPP works fine in Unity, however issues crop up when trying to build the resulting solution in VS to deploy to HoloLens.
I am on VS latest stable 2017 (15.9.7) and the build fails there.

Before I post any diagnostics, can you please check if the build is successful in VS, as well?

[b-fazamb]

Actually @vincentpierre , without dumping the whole log the critical part is below (reporting the two errors).

1>Creating library C:\Users\fzamb\Downloads\ml-agents-release-v0.7\ml-agents-release-v0.7\UnitySDK\UWP-MLA\build\obj\il2cppOutputProject\Win32\Release\linkresult_F994396AB729935A1EC74CA9439671BB\GameAssembly.lib and object C:\Users\fzamb\Downloads\ml-agents-release-v0.7\ml-agents-release-v0.7\UnitySDK\UWP-MLA\build\obj\il2cppOutputProject\Win32\Release\linkresult_F994396AB729935A1EC74CA9439671BB\GameAssembly.exp

1>B8D19A0C8FE51A2E348663505371FAEB.obj : error LNK2019: unresolved external symbol _dlopen@8 referenced in function _Mono_dlopen_m0A57E032EF60E81CAEAA00AEF1F8E4AFBCB7D991

1>B8D19A0C8FE51A2E348663505371FAEB.obj : error LNK2019: unresolved external symbol _dlsym@8 referenced in function _Mono_dlsym_mEADD0CFB8DDC1C7F3DF7E8D6AA93B559376B551F

1>C:\Users\fzamb\Downloads\ml-agents-release-v0.7\ml-agents-release-v0.7\UnitySDK\UWP-MLA\build\obj\il2cppOutputProject\Win32\Release\linkresult_F994396AB729935A1EC74CA9439671BB\GameAssembly.dll : fatal error LNK1120: 2 unresolved externals

[vincentpierre]

I think this might be related to gRPC. It is not possible to build IL2CPP for windows as of today. .NET does not seem like a viable option either given the missing libraries it has.

[b-fazamb]

I am assuming you meant "it is not possible to build IL2CPP for UWP"?

Sadly, that seems to be the conclusion I have come to, as well :(

[vincentpierre]

#1743 is a proposed fix