Unity 2019.1 TriLib WebGL build failed

[HyunmokMoon]

Hello,
in Unity 2019.1 and TriLib (assimp for unity)
when build WebGL, there is a error message and failed to build.

Failed running "...\Emscripten_Win\python\2.7.5.3_64bit\python.exe" -E "...\WebGLSupport\BuildTools\Emscripten\emcc" @"...\Assets..\Temp\emcc_arguments.resp"

stdout:
stderr:error: Linking globals named 'gzgetc': symbol multiply defined!ERROR:root:Failed to run llvm optimizations:
UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr)

Do you guys have any idea?
Thank you

[kripken]

From the error, I'd guess that zlib is being linked in twice - maybe both Unity and TriLib contain it?
)

[HyunmokMoon]

Thank you respond.
I think gzgetc defined multiply, so I removed libz.bc from TriLib/Plugins/WebGL/Emscripten1.38.11
After removed, I can build to release WebGL version.
But Debug build, I have another error

Failed running "C:\Program Files\Unity\Hub\Editor\2019.1.0b3\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten_Win\python\2.7.5.3_64bit\python.exe" -E "C:\Program Files\Unity\Hub\Editor\2019.1.0b3\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\emcc" @"D:\UnityProjects\ub_unity2019\ub_unity\Assets..\Temp\emcc_arguments.resp"

stdout:
stderr:warning: unexpected number of arguments 1 in call to '_ZNKSt3__28ios_base6getlocEv', should be 2warning: unexpected return type %"class.std::__2::locale::__imp"* in call to '_ZNKSt3__28ios_base6getlocEv', should be voidwarning: unexpected number of arguments 1 in call to '_ZNKSt3__28ios_base6getlocEv', should be 2warning: unexpected return type %"class.std::__2::locale::__imp"* in call to '_ZNKSt3__28ios_base6getlocEv', should be voidwarning: unexpected number of arguments 1 in call to '_ZNKSt3__28ios_base6getlocEv', should be 2warning: unexpected return type %"class.std::__2::locale::__imp"* in call to '_ZNKSt3__28ios_base6getlocEv', should be voidwarning: unexpected number of arguments 1 in call to '_ZNKSt3__28ios_base6getlocEv', should be 2warning: unexpected return type %"class.std::__2::locale::__imp"* in call to '_ZNKSt3__28ios_base6getlocEv', should be voidwarning: unexpected number of arguments 1 in call to '_ZNKSt3__28ios_base6getlocEv', should be 2warning: unexpected return type %"class.std::__2::locale::__imp"* in call to '_ZNKSt3__28ios_base6getlocEv', should be voidwarning: unexpected number of arguments 1 in call to '_ZNKSt3__28ios_base6getlocEv', should be 2warning: unexpected return type %"class.std::__2::locale::__imp"* in call to '_ZNKSt3__28ios_base6getlocEv', should be voidwarning: unexpected number of arguments 1 in call to '_ZNKSt3__28ios_base6getlocEv', should be 2warning: unexpected return type %"class.std::__2::locale::__imp"* in call to
...
Files\Unity\Hub\Editor\2019.1.0b3\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\lib\WebRequest.js', 'D:\UnityProjects\ub_unity2019\ub_unity\Assets\Plugins\WebBrowserInterface.jslib', 'C:\Program Files\Unity\Hub\Editor\2019.1.0b3\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\src\library_pthread_stub.js'] to finish with return code 0, but it returned with code 1 instead! Output: // The Module object: Our interface to the outside world. We import// and export values on it. There are various ways Module can be used:// 1. Not defined. We create it here// 2. A function parameter, function(Module) { ..generated code.. }// 3. pre-run appended it, var Module = {}; ..generated
UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr)

[kripken]

unresolved symbol: inflate sounds like zlib is not being linked in.

Perhaps one of the two projects has a partial build of zlib? That would explain both problems.

[rickomax]

Hi!

I'm the developer of TriLib and came accross this today. And indeed, that's the reason of many TriLib WebGL issues where facing.

I couldn't find any way to hide the internal ZLIB (among other libraries) symbols of Assimp (used internally by TriLib) from outter code yet.

We are working to fix these issues, but any tip is appreciated.

Best regards,
Ricardo Reis.

[kevor]

Hi everyone, I managed to compile removing the duplicated emscripten folders under Plugin/WebGL (I left only Emscripten1.38.11). I also removed the Android, iOS, Linux, OSX, UWP folders.

I'm using the class AssetLoader (not async because as far as I know Threads are not yet supported in webgl).
Here the code i'm using (it works perfectly in Editor)

       WWW www = new WWW(url);
        yield return www;
        while (!www.isDone) { }

        AssetLoader assetLoader = new AssetLoader();
        var loadedGameObject = assetLoader.LoadFromMemory(www.bytes, "aaa");
        loadedGameObject.transform.position = new Vector3(0f, 0f, 0f);

When I run the WebGL compiled application the assetLoader.LoadFromMemory function calls InternalLoadFromMemory that calls ImportFileFromMemory that calls AssimpInterop.ai_ImportFileFromMemory that calls _aiImportFileFromMemory.
This last call fails due to an "Entry Point not found" error (_aiImportFileFromMemory)

My first guess is that the following instruction:

EntryPoint = "aiImportFileFromMemory"

at line 112 in AssimpInterop.cs

[DllImport(DllPath, CharSet = CharSet.Ansi, CallingConvention = CallingConvention.StdCall, EntryPoint = "aiImportFileFromMemory")]
private static extern IntPtr _aiImportFileFromMemory(IntPtr ptrBuffer, uint uintLength, uint uintFlags, int fileId, string strHint, [MarshalAs(UnmanagedType.FunctionPtr)] DataCallback dataCallback, [MarshalAs(UnmanagedType.FunctionPtr)] ExistsCallback existsCallback, [MarshalAs(UnmanagedType.FunctionPtr)] ProgressCallback progressCallback);

is not recognized properly.

Modifying the name of the external funtion to aiImportFileFromMemory (without the underscore) and removing the EntryPoint property I get another error in webgl build saying "missing function: aiImportFileFromMemory".

At this point my next guess is that i left the wrong Emscripten folder.

I'm using Unity 2018.3.10f1, which Emscripten folder am I supposed to use?
Also using WebAssembly.

Thanks
Cosimo

[kripken]

cc @juj

[rickomax]

@kevor you don't need to change the methods declarations on the .cs files.

The latest TriLib version has four WebGL libraries variations:
1.37.3 for Unity 2017.4.1f1 onwards
1.37.33 for Unity 2018.2 onwards
1.37.40 for Unity 2018.3 onwards
1.38.11 for Unity 2019.1 onwards

The reason for four different libraries is because the different Emscripten versions used by Unity won't recognize a library built with earlier Emscripten tools. I would suggest you to try another Unity version on this list to check if that's an issue with your specific Unity version..

[rickomax]

@kripken I've generated an output file with "llvm-nm" from the library I'm building (libassimp.so <-dynamic) where we can see the issue better:
output.txt
I'm using a version-script that basically says only methods that begin with "ai", like "aiMaterial_HasOpacity", "aiMaterial_HasSpecular", etc, are global symbols.
If we take a look at output.txt, there are many symbols declared as globals, regardless the version-script, like "unz_copyright, unzOpen, uncompress", which are both ZLIB\MiniZip methods which gives "multiple symbols declaration" errors when linking with these libraries which are already linked by Unity by default. The point is that, I wonder if there is a way to link my own ZLIB\MiniZip versions with the output library, but keeping all these symbols as local. By local, I mean that my dynamic library will use its own ZLIB\MiniZip versions instead of trying to use Unity ones (which may not exist, depending on some Unity configurations).

[kripken]

You can make them local by renaming them, and using the new name in the places calling them. Another option is to mark them as static (so not usable outside the current file) and #include the files into the file that calls them.

[FanEric]

@kevor you don't need to change the methods declarations on the .cs files.

The latest TriLib version has four WebGL libraries variations:
1.37.3 for Unity 2017.4.1f1 onwards
1.37.33 for Unity 2018.2 onwards
1.37.40 for Unity 2018.3 onwards
1.38.11 for Unity 2019.1 onwards

The reason for four different libraries is because the different Emscripten versions used by Unity won't recognize a library built with earlier Emscripten tools. I would suggest you to try another Unity version on this list to check if that's an issue with your specific Unity version..

Does that mean there is a solution on Unity2019.1 on WebGL? I downloaded the latest version 1.8.9a of Trilib on assetstore and found it wouldn't work still

[rickomax]

@kripken I was able to fix the issues by replacing the C files functions and const names with batch scripts, but the Emscripten tools 1.37.3 can't find some stdlib symbols like this one:
std::__2::basic_string<char, std::__2::char_traits, std::__2::allocator >::at(unsigned int) const
That's what is causing the issues with Unity 2019 (Emscripten 1.38.11). If I compile using newer Emscripten tools, the BC library won't be compatible with Emscripten 1.37.3, right?
Have you ever seen issues like that? I've read somewhere else other Emscripten users had this issue with stdlib symbols that could not be found.

[kripken]

It's very likely 1.38.11 and 1.37.3 have incompatible bitcode, yes, those are over a year apart, and we upgraded LLVM in between.

I'm not aware of any open bugs on missing stdlib symbols. But if you can get a testcase I can take a look at that - the bitcode to compile + the command to see the error with that bitcode should be enough.

[rickomax]

@kripken I can give you access to my project GitHub repo. Testing is pretty simple, using LLVM-NM will show the U (Undefined) stdlib symbols on the generated library.

Edit: What OS do you use for testing?

[kripken]

@rickomax sure, any way I can access the bitcode file to test is good.

I'm on linux.

[rickomax]

@kripken I've invited you to the private repository. The usage is described in the README, it should be simple to follow. Thank you!

[kripken]

@rickomax I followed the instructions there but it hits a build error. I opened an issue on the repo so the info is private.

However, I didn't realize this depends on 1.37.3. That's over a year old, and it's very possible you are seeing a bug that has been fixed, so updating may resolve things.

[rickomax]

Sorry @kripken !
I've fixed the files and commited to the repo.
Could you update your local repo?
I've tried to compile the same library on newer Emscripten versions and I get the same result.

[kripken]

Ok, it builds libassimp.so without errors. So far so good, or should I have seen an error?

If that was expected, then next it prints some undefined symbols on the .so file. It's expected that those should be undefined - this is before linking in system libraries. So that looks ok as well.

[rickomax]

@kripken exactly, the library compiles just fine.

When linking this library with the Emscripten project Unity outputs, it will complain about these undefined symbols and won't finish the compilation. So as far as I understood, the Emscripten version used in Unity is missing them, right?

It's odd because when I link the library with Unity versions prior to Emscripten 1.38.11, it will link the library just fine, the error occurs on Emscripten version 1.38.11 onwards. It will complain specifically about these std::__2::basic_string symbols and some others on the std namespace. Maybe Unity Clang version is stripped-out or these symbols aren't available on newer Clang libraries.

[kripken]

Hmm, I'm not sure what's wrong, but it's impossible to tell without looking at the emscripten build inside unity.

Testing with 1.37.3 in your testcase, ./emcc libassimp.so links ok, and it includes libcxx and libcxxabi which should provide those missing symbols. (I also verified this works also with -s LINKABLE=1 to disable dce, since there is no main()). So this looks fine on the emscripten side. Perhaps unity disables some system libraries from being linked in?

If you have a ticket open with unity, feel free to cc me into any conversation there, happy to help there if I can.

[rickomax]

@kripen I'm trying to simulate the Unity environment on my repo.

Meanwhile, I was able to capture the full output log and the arguments passed to EMCC:

EMCC arguments (Emscripten 1.38.11):

 -O3 -g0 -DUNITY_WEBGL=1 -s PRECISE_F32=2 -s USE_WEBGL2=1 -s FULL_ES3=1 -s "EXTRA_EXPORTED_RUNTIME_METHODS=['addRunDependency','removeRunDependency','FS_createPath','FS_createDataFile','ccall','cwrap','stackTrace']" -s DISABLE_EXCEPTION_CATCHING=0 -s TOTAL_MEMORY=32MB -s WASM=1 -s "BINARYEN_TRAP_MODE='clamp'" -s ALLOW_MEMORY_GROWTH=1 -s MODULARIZE=1 -s EXPORT_NAME=UnityModule --memory-init-file 1 --emit-symbol-map --output_eol linux  -o "/media/rickomax/extra/Test/New Unity Project/Temp/StagingArea/Data/linkresult_wasm/build.js" --pre-js "/media/rickomax/extra/Unity2019/Editor/Editor/Data/PlaybackEngines/WebGLSupport/BuildTools/prejs/CachedXMLHttpRequest.js" --pre-js "/media/rickomax/extra/Unity2019/Editor/Editor/Data/PlaybackEngines/WebGLSupport/BuildTools/prejs/FileSystem.js" --pre-js "/media/rickomax/extra/Unity2019/Editor/Editor/Data/PlaybackEngines/WebGLSupport/BuildTools/prejs/FullScreen.js" --pre-js "/media/rickomax/extra/Unity2019/Editor/Editor/Data/PlaybackEngines/WebGLSupport/BuildTools/prejs/LoaderSetup.js" --pre-js "/media/rickomax/extra/Unity2019/Editor/Editor/Data/PlaybackEngines/WebGLSupport/BuildTools/prejs/MediaDevices.js" --pre-js "/media/rickomax/extra/Unity2019/Editor/Editor/Data/PlaybackEngines/WebGLSupport/BuildTools/prejs/SendMessage.js" --js-library "/media/rickomax/extra/Unity2019/Editor/Editor/Data/PlaybackEngines/WebGLSupport/BuildTools/lib/Audio.js" --js-library "/media/rickomax/extra/Unity2019/Editor/Editor/Data/PlaybackEngines/WebGLSupport/BuildTools/lib/Cursor.js" --js-library "/media/rickomax/extra/Unity2019/Editor/Editor/Data/PlaybackEngines/WebGLSupport/BuildTools/lib/Eval.js" --js-library "/media/rickomax/extra/Unity2019/Editor/Editor/Data/PlaybackEngines/WebGLSupport/BuildTools/lib/FileSystem.js" --js-library "/media/rickomax/extra/Unity2019/Editor/Editor/Data/PlaybackEngines/WebGLSupport/BuildTools/lib/Logging.js" --js-library "/media/rickomax/extra/Unity2019/Editor/Editor/Data/PlaybackEngines/WebGLSupport/BuildTools/lib/Profiler.js" --js-library "/media/rickomax/extra/Unity2019/Editor/Editor/Data/PlaybackEngines/WebGLSupport/BuildTools/lib/SystemInfo.js" --js-library "/media/rickomax/extra/Unity2019/Editor/Editor/Data/PlaybackEngines/WebGLSupport/BuildTools/lib/UnetWebSocket.js" --js-library "/media/rickomax/extra/Unity2019/Editor/Editor/Data/PlaybackEngines/WebGLSupport/BuildTools/lib/Video.js" --js-library "/media/rickomax/extra/Unity2019/Editor/Editor/Data/PlaybackEngines/WebGLSupport/BuildTools/lib/WebCam.js" --js-library "/media/rickomax/extra/Unity2019/Editor/Editor/Data/PlaybackEngines/WebGLSupport/BuildTools/lib/WebRequest.js" --js-library "/media/rickomax/extra/Test/New Unity Project/Assets/TriLib/TriLib/Plugins/WebGL/JsHelperNative.jslib" "/media/rickomax/extra/Test/New Unity Project/Temp/StagingArea/Data/Native/build.bc" "/media/rickomax/extra/Test/New Unity Project/Assets/TriLib/TriLib/Plugins/WebGL/libstb_image.bc" "/media/rickomax/extra/Test/New Unity Project/Assets/TriLib/TriLib/Plugins/WebGL/libassimp.bc"

Output:

stdout:
stderr:
warning: unresolved symbol: _ZNKSt3__212basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE2atEj
warning: unresolved symbol: _ZNKSt3__212basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE4findEcj
warning: unresolved symbol: _ZNKSt3__212basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE5rfindEcj
warning: unresolved symbol: _ZNKSt3__212basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE7compareEjjPKcj
warning: unresolved symbol: _ZNSt3__212__next_primeEj
warning: unresolved symbol: _ZNSt3__212basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE2atEj
warning: unresolved symbol: _ZNSt3__212basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE5eraseEjj
warning: unresolved symbol: _ZNSt3__212basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE6__initEPKcj
warning: unresolved symbol: _ZNSt3__212basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE6__initEPKcjj
warning: unresolved symbol: _ZNSt3__212basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE6appendEPKcj
warning: unresolved symbol: _ZNSt3__212basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE6insertEjPKc
warning: unresolved symbol: _ZNSt3__212basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE6insertEjPKcj
warning: unresolved symbol: _ZNSt3__212basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE6resizeEjc
warning: unresolved symbol: _ZNSt3__212basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE7replaceEjjPKc
warning: unresolved symbol: _ZNSt3__212basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE7reserveEj
warning: unresolved symbol: _ZNSt3__212basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEC1ERKS5_jjRKS4_
warning: unresolved symbol: _ZNSt3__213basic_istreamIcNS_11char_traitsIcEEE4readEPci
warning: unresolved symbol: _ZNSt3__215basic_streambufIcNS_11char_traitsIcEEE6setbufEPci
warning: unresolved symbol: _ZNSt3__215basic_streambufIcNS_11char_traitsIcEEE6xsgetnEPci
warning: unresolved symbol: _ZNSt3__215basic_streambufIcNS_11char_traitsIcEEE6xsputnEPKci
warning: unresolved symbol: _ZNSt3__219__shared_weak_count12__add_sharedEv
warning: unresolved symbol: _ZNSt3__219__shared_weak_count16__release_sharedEv
warning: unresolved symbol: _Znaj
warning: unresolved symbol: _Znwj
warning: unresolved symbol: _ZnwjRKSt9nothrow_t
ERROR:
root:'/media/rickomax/extra/Unity2019/Editor/Editor/Data/PlaybackEngines/WebGLSupport/BuildTools/Emscripten_FastComp_Linux/binaryen/bin/asm2wasm /media/rickomax/extra/Test/New Unity Project/Temp/StagingArea/Data/linkresult_wasm/build.temp.asm.js --total-memory=33554432 --trap-mode=clamp -O3 --mem-init=/media/rickomax/extra/Test/New Unity Project/Temp/StagingArea/Data/linkresult_wasm/build.js.mem --mem-base=1024 --wasm-only --symbolmap=/media/rickomax/extra/Test/New Unity Project/Temp/StagingArea/Data/linkresult_wasm/build.js.symbols -o /media/rickomax/extra/Test/New Unity Project/Temp/StagingArea/Data/linkresult_wasm/build.wasm' failed

[rickomax]

I guess I found the issue.

Unity uses it's own pre-compiled system libraries.

Versions prior to Unity 2019 have the needed symbols:

Unity 2019 onwards don't have the needed symbols:

The UnityNative.js file isn't referenced directly on the EMCC command above, but I'm almost sure it's being included.

The best I can do is trying to remove these symbols from my code.

[kripken]

Have you filed a ticket with them? This sounds like a bug they would want to know about. Or perhaps there is a reason for the change that isn't obvious to us.

cc @juj

[rickomax]

@kripken, Yes:
https://fogbugz.unity3d.com/default.asp?1163092_ata4ac2o89voc2fn

[De-Panther]

As far as I know, UnityNative.js is included with all the ".jspre" files. Maybe you can add your own ".jspre" file and include those missing declarations there.

[rickomax]

@De-Panther I'm not sure how I could add only some symbols from a library into a .jspre. I will take a look and check if it's possible.

[stale]

This issue has been automatically marked as stale because there has been no activity in the past year. It will be closed automatically if no further activity occurs in the next 7 days. Feel free to re-open at any time if this issue is still relevant.