[wasm-mt] Emscripten SDK runtime pack is missing multi-threaded static libraries #75263

lambdageek · 2022-09-08T12:50:20Z

In dotnet/emsdk#43 we removed *-mt.a from the emscripten sysroot.

Unfortunately this means that /p:WasmBuildNative=true does not work for multi-threaded builds.

We should either put the libraries back in the main SDK, or figure out how to provide an additional nuget with the threaded libs along with required build flags to make emcc look for them in another directory.

$ dotnet build
MSBuild version 17.4.0-preview-22416-02+5d102ae37 for .NET
Determining projects to restore...
Restored /Users/alklig/work/net7-playground/hithread/browser.csproj (in 2 ms).
/Users/alklig/work/net7-nightly/sdk/7.0.100-rc.2.22425.5/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.RuntimeIdentifierInference.targets(219,5): message NETSDK1057: You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy [/Users/alklig/work/net7-playground/hithread/browser.csproj]
browser -> /Users/alklig/work/net7-playground/hithread/bin/Debug/net7.0/browser-wasm/browser.dll
Compiling native assets with emcc with -O0. This may take a while ...
[2/3] corebindings.c -> corebindings.o [took 0.384s]
[1/3] pinvoke.c -> pinvoke.o [took 0.384s]
[3/3] driver.c -> driver.o [took 0.416s]
Linking for initial memory $(EmccInitialHeapSize)=536870912 bytes. Set this msbuild property to change the value.
Linking with emcc with -O0. This may take a while ...
...
Exception: FROZEN_CACHE is set, but cache file is missing: "sysroot/lib/wasm32-emscripten/libGL-mt.a" (in cache root path "/Users/alklig/work/net7-nightly/packs/Microsoft.NET.Runtime.Emscripten.3.1.12.Sdk.osx-x64/8.0.0-alpha.1.22415.5/tools/emscripten/cache")
/Users/alklig/work/net7-nightly/packs/Microsoft.NET.Runtime.WebAssembly.Sdk/7.0.0-rc.1.22422.12/Sdk/WasmApp.Native.targets(422,5): error MSB3073: The command "emcc "@/Users/alklig/work/net7-nightly/packs/Microsoft.NETCore.App.Runtime.Mono.multithread.browser-wasm/7.0.0-rc.1.22422.12/runtimes/browser-wasm/native/src/emcc-default.rsp" "@/Users/alklig/work/net7-nightly/packs/Microsoft.NETCore.App.Runtime.Mono.multithread.browser-wasm/7.0.0-rc.1.22422.12/runtimes/browser-wasm/native/src/emcc-link.rsp" "@/Users/alklig/work/net7-playground/hithread/obj/Debug/net7.0/browser-wasm/wasm/for-build/emcc-link.rsp"" exited with code 1. [/Users/alklig/work/net7-playground/hithread/browser.csproj]

Part of #74654

The text was updated successfully, but these errors were encountered:

ghost · 2022-09-08T12:50:53Z

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

In dotnet/emsdk#43 we removed *-mt.a from the emscripten sysroot.

Unfortunately this means that /p:WasmBuildNative=true does not work for multi-threaded builds.

We should either put the libraries back in the main SDK, or figure out how to provide an additional nuget with the threaded libs along with required build flags to make emcc look for them in another directory.

$ dotnet build
MSBuild version 17.4.0-preview-22416-02+5d102ae37 for .NET
Determining projects to restore...
Restored /Users/alklig/work/net7-playground/hithread/browser.csproj (in 2 ms).
/Users/alklig/work/net7-nightly/sdk/7.0.100-rc.2.22425.5/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.RuntimeIdentifierInference.targets(219,5): message NETSDK1057: You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy [/Users/alklig/work/net7-playground/hithread/browser.csproj]
browser -> /Users/alklig/work/net7-playground/hithread/bin/Debug/net7.0/browser-wasm/browser.dll
Compiling native assets with emcc with -O0. This may take a while ...
[2/3] corebindings.c -> corebindings.o [took 0.384s]
[1/3] pinvoke.c -> pinvoke.o [took 0.384s]
[3/3] driver.c -> driver.o [took 0.416s]
Linking for initial memory $(EmccInitialHeapSize)=536870912 bytes. Set this msbuild property to change the value.
Linking with emcc with -O0. This may take a while ...
...
Exception: FROZEN_CACHE is set, but cache file is missing: "sysroot/lib/wasm32-emscripten/libGL-mt.a" (in cache root path "/Users/alklig/work/net7-nightly/packs/Microsoft.NET.Runtime.Emscripten.3.1.12.Sdk.osx-x64/8.0.0-alpha.1.22415.5/tools/emscripten/cache")
/Users/alklig/work/net7-nightly/packs/Microsoft.NET.Runtime.WebAssembly.Sdk/7.0.0-rc.1.22422.12/Sdk/WasmApp.Native.targets(422,5): error MSB3073: The command "emcc "@/Users/alklig/work/net7-nightly/packs/Microsoft.NETCore.App.Runtime.Mono.multithread.browser-wasm/7.0.0-rc.1.22422.12/runtimes/browser-wasm/native/src/emcc-default.rsp" "@/Users/alklig/work/net7-nightly/packs/Microsoft.NETCore.App.Runtime.Mono.multithread.browser-wasm/7.0.0-rc.1.22422.12/runtimes/browser-wasm/native/src/emcc-link.rsp" "@/Users/alklig/work/net7-playground/hithread/obj/Debug/net7.0/browser-wasm/wasm/for-build/emcc-link.rsp"" exited with code 1. [/Users/alklig/work/net7-playground/hithread/browser.csproj]

Part of #74654

Author:	lambdageek
Assignees:	-
Labels:	`arch-wasm`, `area-Build-mono`
Milestone:	-

lambdageek · 2022-09-08T12:51:31Z

Not sure if we will be able to do this in time for 7, or if it will have to wait until 8

lambdageek · 2022-09-08T12:52:33Z

A workaround is for users to manually copy the required libraries from an EMSDK install

cp ~/work/dotnet-runtime/runtime/src/mono/wasm/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/*-mt.a ~/work/net7-nightly/packs/Microsoft.NET.Runtime.Emscripten.3.1.12.Sdk.osx-x64/8.0.0-alpha.1.22415.5/tools/emscripten/cache/sysroot/lib/wasm32-emscripten

lambdageek · 2022-09-08T17:31:26Z

Report from

% du -c -h ~/work/dotnet-runtime/runtime/src/mono/wasm/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/*-mt.a

Size	Library	Required	Notes
40K	libGL-mt.a
1.6M	libasan_rt-mt.a
6.3M	libc++-mt.a	✅
916K	libc++abi-mt.a	✅
3.4M	libc-mt.a	✅
528K	libcompiler_rt-mt.a	✅
532K	libcompiler_rt-wasm-sjlj-mt.a	❓	not sure what this is
80K	libdlmalloc-mt.a	✅
68K	libemmalloc-memvalidate-mt.a
80K	libemmalloc-memvalidate-verbose-mt.a
52K	libemmalloc-mt.a
76K	libemmalloc-verbose-mt.a
16K	libfetch-mt.a	❓
220K	liblsan_common_rt-mt.a
496K	liblsan_rt-mt.a
36K	libprintf_long_double-mt.a	❓
10M	libsanitizer_common_rt-mt.a
40K	libsockets-mt.a	❓
36K	libsockets_proxy-mt.a	❓
32K	libubsan_minimal_rt-mt.a
604K	libubsan_rt-mt.a
4.0K	libunwind-mt.a	✅
6.2M	libwasmfs-asan-debug-mt.a
6.1M	libwasmfs-asan-mt.a
5.0M	libwasmfs-debug-mt.a
5.0M	libwasmfs-mt.a	❓
184K	libwebgpu_cpp-mt.a	❓
48M	total

Just the ✅ total is around 11 megabytes

Adding the ❓is near 17 with libwasmfs making up the bulk of the extra space

lambdageek · 2022-09-08T20:35:28Z

@lewing what do you think about a separate (OOB) nuget that just has the mt libs and some SDK.props that injects emscipten linker flags?

So to do a multi-threaded native build one would have to do something like this:

<PropertyGroup>
  <WasmEnableThread>true</WasmEnableThread>
  <WasmBuildNative>true</WasmBuildNative>
</PropertyGroup>

<ItemGroup>
  <PackageReference Include="Microsoft.NET.Runtime.Emscripten.3.0.12.Threading.Sdk" Version="7.0.0" IncludeAssets="build" />
</ItemGroup>

lewing · 2022-09-08T20:38:42Z

no need, we can do all that in the workload invisibly, the complicated part is packaging it and setting the appropriate compiler options.

lewing · 2022-09-08T20:43:07Z

The steps are essentially adding the threading package to the emsdk build, and referencing it in the emscripten manifest when WasmEnableThreads is set. It's props should add the required flags and paths to the emcc options.

lambdageek · 2022-09-08T20:44:23Z

no need, we can do all that in the workload invisibly, the complicated part is packaging it and setting the appropriate compiler options.

@lewing I was thinking OOB in case we don't make it by RTM 😁

lambdageek · 2022-09-08T20:46:22Z

I'm looking at the emsdk build right now - moving the -mt.a files out of the way (instead of deleting) and making a new nuget seems straightforward. I'll try to have something to show by tomorrow or Monday

lewing · 2022-09-08T21:02:18Z

no need, we can do all that in the workload invisibly, the complicated part is packaging it and setting the appropriate compiler options.

@lewing I was thinking OOB in case we don't

either way it starts with making the pack ;)

lambdageek · 2022-09-09T17:26:21Z

bit of a change of plans. Emscripten doesn't make it easy to look for some (but not all) system libs in another directory. So we're going to spin out the whole emscripten cache into a separate nupkg instead and put both normal and -mt libraries in there

#208) Spin out the emscripten system libraries cache into a separate nupkg. - Split up the Emscripten SDK by moving the system libraries cache into a separate nupkg. - make a new Emscripten Cache SDK that includes just the cache directory - the Cache SDK overrides the WasmCachePath (if it's not set by the user) to point into the Cache SDK we leave the mt multithreaded libraries in the cache. (But we remove the ww WebWorker (non-pthread) Emscripten API libraries) Addresses dotnet/runtime#75263

lambdageek · 2022-09-09T20:27:34Z

Fixed in emsdk. Now we wait for flow...

Spin out the emscripten system libraries cache into a separate nupkg. - Split up the Emscripten SDK by moving the system libraries cache into a separate nupkg. - make a new Emscripten Cache SDK that includes just the cache directory - the Cache SDK overrides the WasmCachePath (if it's not set by the user) to point into the Cache SDK we leave the mt multithreaded libraries in the cache. (But we remove the ww WebWorker (non-pthread) Emscripten API libraries) Addresses dotnet/runtime#75263

dotnet-issue-labeler bot added the area-Build-mono label Sep 8, 2022

ghost added the untriaged New issue has not been triaged by the area owner label Sep 8, 2022

lambdageek added arch-wasm WebAssembly architecture and removed untriaged New issue has not been triaged by the area owner labels Sep 8, 2022

lambdageek added this to Incoming in wasm-mt via automation Sep 8, 2022

lambdageek added this to the 7.0.0 milestone Sep 8, 2022

lambdageek moved this from Incoming to To do (main) in wasm-mt Sep 8, 2022

lambdageek mentioned this issue Sep 8, 2022

[wasm-mt] Issues with enabling threads in .NET 7 RC 1 #74654

Closed

3 tasks

lambdageek self-assigned this Sep 8, 2022

lambdageek mentioned this issue Sep 9, 2022

[release/7.0][wasm-mt] Create a Microsoft.NET.Runtime.Emscripten.Cache dotnet/emsdk#208

Merged

lambdageek mentioned this issue Sep 9, 2022

[main] [wasm-mt] Create a Microsoft.NET.Runtime.Emscripten.Cache dotnet/emsdk#211

Merged

lewing closed this as completed Sep 14, 2022

wasm-mt automation moved this from To do (main) to Done Sep 14, 2022

ghost locked as resolved and limited conversation to collaborators Oct 14, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[wasm-mt] Emscripten SDK runtime pack is missing multi-threaded static libraries #75263

[wasm-mt] Emscripten SDK runtime pack is missing multi-threaded static libraries #75263

lambdageek commented Sep 8, 2022 •

edited

Loading

ghost commented Sep 8, 2022

lambdageek commented Sep 8, 2022

lambdageek commented Sep 8, 2022

lambdageek commented Sep 8, 2022 •

edited

Loading

lambdageek commented Sep 8, 2022 •

edited

Loading

lewing commented Sep 8, 2022 •

edited

Loading

lewing commented Sep 8, 2022 •

edited

Loading

lambdageek commented Sep 8, 2022

lambdageek commented Sep 8, 2022

lewing commented Sep 8, 2022

lambdageek commented Sep 9, 2022

lambdageek commented Sep 9, 2022

[wasm-mt] Emscripten SDK runtime pack is missing multi-threaded static libraries #75263

[wasm-mt] Emscripten SDK runtime pack is missing multi-threaded static libraries #75263

Comments

lambdageek commented Sep 8, 2022 • edited Loading

ghost commented Sep 8, 2022

lambdageek commented Sep 8, 2022

lambdageek commented Sep 8, 2022

lambdageek commented Sep 8, 2022 • edited Loading

lambdageek commented Sep 8, 2022 • edited Loading

lewing commented Sep 8, 2022 • edited Loading

lewing commented Sep 8, 2022 • edited Loading

lambdageek commented Sep 8, 2022

lambdageek commented Sep 8, 2022

lewing commented Sep 8, 2022

lambdageek commented Sep 9, 2022

lambdageek commented Sep 9, 2022

lambdageek commented Sep 8, 2022 •

edited

Loading

lambdageek commented Sep 8, 2022 •

edited

Loading

lambdageek commented Sep 8, 2022 •

edited

Loading

lewing commented Sep 8, 2022 •

edited

Loading

lewing commented Sep 8, 2022 •

edited

Loading