[wasm] Compile .bc->.o in parallel, before passing to the linker #54053

radical · 2021-06-11T09:12:18Z

Currently, we use mono-aot-cross to compile the managed assemblies
into bitcode files. Then we pass the .bc files to emcc for linking.
We can save some amount of work, if precompile .bc files to .o
files, and pass them for linking.

This would also allow us to rebuild the .o files only when needed,
thus improving incremental builds.
Also, emscripten has recently removed some amount of parallelization, so
this allows us to do some pre-linking work in parallel, and speed up the
builds a little bit.

This is implemented via a new EmccCompile task, that is meant only
for compiling, and not as a general purpose emcc invocation task.

Also, this bumps msbuild used for tasks compilation to 16.9.0.

- this cleans up `Utils.RunProcess` a bit, but doesn't change the behavior for the existing non-wasm users (Apple, and Android build tasks) - New overloads are added to support getting the output, and exit code, and throwing only when really needed - And running shell commands, appropriately on windows, and unix.

Currently, we use `mono-aot-cross` to compile the managed assemblies into bitcode files. Then we pass the `.bc` files to `emcc` for linking. We can save some amount of work, if precompile `.bc` files to `.o` files, and pass them for linking. This would also allow us to rebuild the `.o` files only when needed, thus improving incremental builds. Also, emscripten has recently removed some amount of parallelization, so this allows us to do some pre-linking work in parallel, and speed up the builds a little bit. This is implemented via a new `EmccCompile` task, that is meant *only* for compiling, and not as a general purpose `emcc` invocation task.

`System.IO.MemoryMappedFiles.Tests` - Issue: dotnet#54194

ghost · 2021-06-15T00:23:50Z

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

Author:	radical
Assignees:	-
Labels:	`arch-wasm`, `area-Build-mono`
Milestone:	-

- `IBuildEngine9.RequestCores` - This is available only in 16.10+, so bump msbuild used for building the tests

radical · 2021-06-16T17:07:48Z

@akoeplinger Does the msbuild bump to 16.10.0 look fine?

radical · 2021-06-16T21:38:04Z

/azp run runtime,runtime-staging

azure-pipelines · 2021-06-16T21:38:38Z

Azure Pipelines successfully started running 2 pipeline(s).

…third

lewing · 2021-06-17T00:39:44Z

/azp run runtime

- This is meant to be disabled for CI (it has `IgnoreForCI=true` in the csproj)

.. because we keep running into the following on browser, and other platforms: ``` /__w/1/s/src/libraries/Microsoft.NETCore.Platforms/tests/Microsoft.NETCore.Platforms.Tests.csproj : error NU1201: Project System.Security.AccessControl is not compatible with net472 (.NETFramework,Version=v4.7.2). Project System.Security.AccessControl supports: net6.0 (.NETCoreApp,Version=v6.0) [/__w/1/s/Build.proj] \#\#[error]src/libraries/Microsoft.NETCore.Platforms/tests/Microsoft.NETCore.Platforms.Tests.csproj(0,0): error NU1201: (NETCORE_ENGINEERING_TELEMETRY=Restore) Project System.Security.AccessControl is not compatible with net472 (.NETFramework,Version=v4.7.2). Project System.Security.AccessControl supports: net6.0 (.NETCoreApp,Version=v6.0) ```

This is already not being sent to CI with `IgnoreForCI=true`. It fails on CI with: ``` /_/src/Shared/NativeMethodsShared.cs(855,29): error IL1005: Microsoft.Build.Shared.NativeMethodsShared.SystemInformation.get: Error processing method 'Microsoft.Build.Shared.NativeMethodsShared.SystemInformationData.SystemInformationData()' in assembly 'Microsoft.Build.Utilities.Core.dll' [/__w/1/s/src/libraries/Microsoft.NETCore.Platforms/tests/Microsoft.NETCore.Platforms.Tests.csproj] \##[error]/_/src/Shared/NativeMethodsShared.cs(855,29): error IL1005: (NETCORE_ENGINEERING_TELEMETRY=Build) Microsoft.Build.Shared.NativeMethodsShared.SystemInformation.get: Error processing method 'Microsoft.Build.Shared.NativeMethodsShared.SystemInformationData.SystemInformationData()' in assembly 'Microsoft.Build.Utilities.Core.dll' Mono.Linker.LinkerFatalErrorException: /_/src/Shared/NativeMethodsShared.cs(855,29): error IL1005: Microsoft.Build.Shared.NativeMethodsShared.SystemInformation.get: Error processing method 'Microsoft.Build.Shared.NativeMethodsShared.SystemInformationData.SystemInformationData()' in assembly 'Microsoft.Build.Utilities.Core.dll' ---> System.NotImplementedException: switch at Mono.Linker.Steps.UnreachableBlocksOptimizer.BodyReducer.IsConstantBranch(OpCode opCode, Int32 operand) in illink.dll:token 0x6000770+0x41 at Mono.Linker.Steps.UnreachableBlocksOptimizer.BodyReducer.RemoveConditions() in illink.dll:token 0x6000768+0x22a at Mono.Linker.Steps.UnreachableBlocksOptimizer.BodyReducer.RewriteBody() in illink.dll:token 0x6000766+0xa at Mono.Linker.Steps.UnreachableBlocksOptimizer.ProcessStack() in illink.dll:token 0x60004e9+0x102 at Mono.Linker.Steps.UnreachableBlocksOptimizer.ProcessMethod(MethodDefinition method) in illink.dll:token 0x60004e5+0x44 at Mono.Linker.Steps.MarkStep.ProcessMethod(MethodDefinition method, DependencyInfo& reason, Scope& scope) in illink.dll:token 0x600040e+0x9b at Mono.Linker.Steps.MarkStep.ProcessQueue() in illink.dll:token 0x6000399+0x24 --- End of inner exception stack trace --- at Mono.Linker.Steps.MarkStep.ProcessQueue() in illink.dll:token 0x6000399+0xaa at Mono.Linker.Steps.MarkStep.ProcessPrimaryQueue() in illink.dll:token 0x6000396+0xa at Mono.Linker.Steps.MarkStep.Process() in illink.dll:token 0x6000393+0x0 at Mono.Linker.Steps.MarkStep.Process(LinkContext context) in illink.dll:token 0x600038a+0x41 at Mono.Linker.Pipeline.ProcessStep(LinkContext context, IStep step) in illink.dll:token 0x6000264+0x0 at Mono.Linker.Pipeline.Process(LinkContext context) in illink.dll:token 0x6000263+0xf at Mono.Linker.Driver.Run(ILogger customLogger) in illink.dll:token 0x60000fd+0x20 Optimizing assemblies for size, which may change the behavior of the app. Be sure to test after publishing. See: https://aka.ms/dotnet-illink /__w/1/s/.dotnet/sdk/6.0.100-preview.4.21255.9/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.ILLink.targets(80,5): error NETSDK1144: Optimizing assemblies for size failed. Optimization can be disabled by setting the PublishTrimmed property to false. [/__w/1/s/src/libraries/Microsoft.NETCore.Platforms/tests/Microsoft.NETCore.Platforms.Tests.csproj] \##[error].dotnet/sdk/6.0.100-preview.4.21255.9/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.ILLink.targets(80,5): error NETSDK1144: (NETCORE_ENGINEERING_TELEMETRY=Build) Optimizing assemblies for size failed. Optimization can be disabled by setting the PublishTrimmed property to false. ```

radical · 2021-06-22T21:07:34Z

cc @ericstj @ViktorHofer for the change disabling Microsoft.NETCore.Platforms.Tests for wasm.

Issue: dotnet#54194

radical · 2021-06-23T05:28:39Z

1 test failure for wasm:

[04:55:37] info: browser: Console websocket connected.
[04:55:37] info: Arguments: --run,WasmTestRunner.dll,System.IO.Compression.Tests.dll,-notrait,category=IgnoreForCI,-notrait,category=OuterLoop,-notrait,category=failing
[04:55:37] fail: [out of order message from the browser]: http://127.0.0.1:34229/runtime.js 234:26 Uncaught TypeError: Cannot set property 'loaded_cb' of null
[05:10:37] fail: Application has finished with exit code TIMED_OUT but 0 was expected

@Daniel-Genkin is already looking at fixing this.
I think we can merge this.

lewing · 2021-06-23T05:44:36Z

@Daniel-Genkin is already looking at fixing this.
I think we can merge this.

fix should be in #54587

ViktorHofer · 2021-06-23T06:25:46Z

@ViktorHofer Any idea why a msbuild bump to 16.10.0 in this PR might be failing wasm/android/iOS/etc builds? That's the only non-wasm specific change here.

That very much sounds like NuGet/Home#10368. Usually you can fix this by adding the transitive dependencies, in this case in the Platforms.Tests.csproj file.

radical · 2021-06-23T16:53:46Z

The failures are all unrelated to this PR.

ViktorHofer · 2021-06-23T17:47:16Z

The failures are all unrelated to this PR.

Usually we link to issues that track the failures according to https://github.com/dotnet/runtime/blob/main/docs/issues-pr-management.md#common-policies. Can you please do so for the issues that you were seeing in the last build?

radical · 2021-06-23T18:09:03Z

Failures:

Libraries Test Run checked coreclr windows x86 Release - #54628
Libraries Test Run release mono OSX x64 Debug - #54624
Libraries Test Run release coreclr windows x64 Debug - #54627
Libraries Test Run release coreclr Linux x64 Debug - #54623

And for CoreCLR Pri0 Runtime Tests Run windows arm64 checked - profiler.gc, and JIT.Directed failed but I'm not able to access the logs for them.

…nitial_config * origin/main: (362 commits) [wasm][debugger] Reuse debugger-agent on wasm debugger (dotnet#52300) Put Crossgen2 in sync with dotnet#54235 (dotnet#54438) Move System.Object serialization to ObjectConverter (dotnet#54436) Move setting fHasVirtualStaticMethods out of sanity check section (dotnet#54574) [wasm] Compile .bc->.o in parallel, before passing to the linker (dotnet#54053) Change PathInternal.IsCaseSensitive to a constant (dotnet#54340) Make mono_polling_required a public symbol (dotnet#54592) Respect EventSource::IsSupported setting in more codepaths (dotnet#51977) Root ComActivator for hosting (dotnet#54524) Add ILLink annotations to S.D.Common related to DbConnectionStringBuilder (dotnet#54280) Fix finalizer issue with regions (dotnet#54550) Add support for multi-arch install locations (dotnet#53763) Update library testing docs page to reduce confusion (dotnet#54324) [FileStream] handle UNC and device paths (dotnet#54483) Update NetAnalyzers version (dotnet#54511) Added runtime dependency to fix the intermittent test failures (dotnet#54587) Disable failing System.Reflection.Tests.ModuleTests.GetMethods (dotnet#54564) [wasm] Move AOT builds from `runtime-staging` to `runtime` (dotnet#54577) Keep obj node for ArrayIndex. (dotnet#54584) Disable another failing MemoryCache test (dotnet#54578) ...

dotnet-issue-labeler bot added the area-Build-mono label Jun 11, 2021

radical added 2 commits June 14, 2021 20:09

radical mentioned this pull request Jun 15, 2021

[wasm][aot] Runtime crash in System.IO.MemoryMappedFiles.Tests #54194

Closed

Disable failing tests

1909fcc

`System.IO.MemoryMappedFiles.Tests` - Issue: dotnet#54194

radical force-pushed the wasm-bc-to-o-parallel branch from 376b844 to 1909fcc Compare June 15, 2021 00:19

radical marked this pull request as ready for review June 15, 2021 00:20

radical requested a review from marek-safar as a code owner June 15, 2021 00:20

radical requested review from lewing, vargaz, steveisok, mdh1418 and akoeplinger June 15, 2021 00:23

radical added the arch-wasm WebAssembly architecture label Jun 15, 2021

radical added 5 commits June 15, 2021 14:57

Merge remote-tracking branch 'origin/main' into wasm-bc-third

24338c6

[wasm] EmccCompile task: Use msbuild's new resource management API

e7b638c

- `IBuildEngine9.RequestCores` - This is available only in 16.10+, so bump msbuild used for building the tests

cleanup

9f23826

Correctly handle non-aot native case

f87f705

Merge remote-tracking branch 'origin/main' into wasm-bc-third

093e133

radical requested a review from rainersigwald June 16, 2021 17:08

radical added 2 commits June 16, 2021 16:43

Merge remote-tracking branch 'origin/main' into wasm-bc-third

d7f867e

Merge branch 'main' into wasm-bc-to-o-parallel

f562d2e

lewing approved these changes Jun 16, 2021

View reviewed changes

radical added 2 commits June 16, 2021 19:56

Merge remote-tracking branch 'origin/main' into wasm-bc-third

f20240b

Merge remote-tracking branch 'rf/wasm-bc-to-o-parallel' into wasm-bc-…

6e90a22

…third

radical added 4 commits June 21, 2021 15:55

Merge remote-tracking branch 'origin/main' into wasm-bc-to-o-parallel

d816231

Disable Microsoft.NETCore.Platforms.Tests on wasm, for local builds too

1be69d6

- This is meant to be disabled for CI (it has `IgnoreForCI=true` in the csproj)

Merge remote-tracking branch 'origin/main' into wasm-bc-to-o-parallel

76d370a

radical removed the NO-MERGE The PR is not ready for merge yet (see discussion for detailed reasons) label Jun 22, 2021

radical added 3 commits June 22, 2021 12:36

Merge remote-tracking branch 'origin/main' into wasm-bc-to-o-parallel

9374c61

undo test exclusion change related to the 16.10.0 bump

b7f52e4

radical added 2 commits June 22, 2021 17:30

[wasm] Disable System.IO.MemoryMappedFiles.Tests

38e4fb8

Issue: dotnet#54194

Merge remote-tracking branch 'origin/main' into wasm-bc-to-o-parallel

ad2255c

Merge remote-tracking branch 'origin/main' into wasm-bc-to-o-parallel

e677e8a

radical merged commit 0416c34 into dotnet:main Jun 23, 2021

radical deleted the wasm-bc-to-o-parallel branch June 23, 2021 16:54

runfoapp bot mentioned this pull request Jun 24, 2021

[wasm] System.Net.Http functional test failing:GetAsync_ManyDifferentResponseHeaders_ParsedCorrectly #54655

Closed

karelz mentioned this pull request Jun 24, 2021

Connect_DualMode_DnsConnect_RetrievedEndPoints_Success - Assert ::1 #54677

Closed

radical mentioned this pull request Jun 28, 2021

[wasm] Compile bitcode files, and use the .o files for the linking step #50285

Closed

karelz mentioned this pull request Jun 30, 2021

System.Net tests failing with System.TimeoutException #54778

Closed

ghost locked as resolved and limited conversation to collaborators Jul 23, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[wasm] Compile .bc->.o in parallel, before passing to the linker #54053

[wasm] Compile .bc->.o in parallel, before passing to the linker #54053

radical commented Jun 11, 2021 •

edited

ghost commented Jun 15, 2021

radical commented Jun 16, 2021

radical commented Jun 16, 2021

azure-pipelines bot commented Jun 16, 2021

lewing commented Jun 17, 2021

radical commented Jun 22, 2021

radical commented Jun 23, 2021

lewing commented Jun 23, 2021

ViktorHofer commented Jun 23, 2021

radical commented Jun 23, 2021

ViktorHofer commented Jun 23, 2021

radical commented Jun 23, 2021

[wasm] Compile .bc->.o in parallel, before passing to the linker #54053

[wasm] Compile .bc->.o in parallel, before passing to the linker #54053

Conversation

radical commented Jun 11, 2021 • edited

ghost commented Jun 15, 2021

radical commented Jun 16, 2021

radical commented Jun 16, 2021

azure-pipelines bot commented Jun 16, 2021

lewing commented Jun 17, 2021

radical commented Jun 22, 2021

radical commented Jun 23, 2021

lewing commented Jun 23, 2021

ViktorHofer commented Jun 23, 2021

radical commented Jun 23, 2021

ViktorHofer commented Jun 23, 2021

radical commented Jun 23, 2021

radical commented Jun 11, 2021 •

edited