[9.0.1xx-rc1] `MSB0001: Internal MSBuild Error` while building command-line-api

[ellahathaway]

While building in the CentOSStream9_Mono_Offline_CurrentSourceBuiltSdk_x64 leg (Mono Stage 2), the build encountered the following error:

Time Elapsed 00:00:02.47
    MSBUILD : error MSB1025: An internal failure occurred while running MSBuild.
    Microsoft.Build.Framework.InternalErrorException: MSB0001: Internal MSBuild Error: Cannot acquire required number of nodes.
    =============
    System.AggregateException: One or more errors occurred. (Method has zero rva)
     ---> Microsoft.Build.BackEnd.NodeFailedToLaunchException: Method has zero rva
     ---> System.BadImageFormatException: Method has zero rva
    File name: 'System.Diagnostics.Process'
       at System.Runtime.Serialization.SerializationGuard.ThrowIfDeserializationInProgress(String switchSuffix, Int32& cachedValue)
       at System.Diagnostics.Process.Start()
       at System.Diagnostics.Process.Start(ProcessStartInfo startInfo)
       at Microsoft.Build.BackEnd.NodeLauncher.StartInternal(String msbuildLocation, String commandLineArgs)
       --- End of inner exception stack trace ---
       at Microsoft.Build.BackEnd.NodeLauncher.StartInternal(String msbuildLocation, String commandLineArgs)
       at Microsoft.Build.BackEnd.NodeLauncher.<>c__DisplayClass3_0.<Start>b__0()
       at Microsoft.Build.BackEnd.NodeLauncher.DisableMSBuildServer(Func`1 func)
       at Microsoft.Build.BackEnd.NodeLauncher.Start(String msbuildLocation, String commandLineArgs, Int32 nodeId)
       at Microsoft.Build.BackEnd.NodeProviderOutOfProcBase.<>c__DisplayClass14_0.<GetNodes>g__StartNewNode|2(Int32 nodeId)
       at Microsoft.Build.BackEnd.NodeProviderOutOfProcBase.<>c__DisplayClass14_0.<GetNodes>b__0(Int32 nodeId)
       --- End of inner exception stack trace ---

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[ellahathaway]

This is the same error from dotnet/source-build#4519. Based on the mitigation of that issue, I'm going to rerun the leg and will follow up based on that.

[ellahathaway]

Rerun succeeded - again, I think that this might just be a timing issue with mono:

The warning "cannot acquire required number of nodes" indicates the system could not allocate the necessary resources at the time of the request. This can often be a transient issue due to temporary high load or delays in resource allocation. Similarly, when the rva of a method is zero, it means that the method's il code is not correctly mapped in the executable file. The problem seems to be related to the timing of resource allocation or mapping during the initial run, which was successfully resolved on the subsequent attempt.

ccing the compiler team for verification on my assumptions @dotnet/roslyn

[ellahathaway]

@dotnet/roslyn - friendly ping on the question above :) TIA

[333fred]

Maybe I'm missing something, but this seems to be an MSBuild failure, not anything to do with Roslyn.

[333fred]

Maybe @rainersigwald would have a better idea.

[MichaelSimons]

[Triage] @uweigand - Is this anything you by chance have seen before? @omajid suggested pinging you. TIA.

[rainersigwald]

This is MSBuild reporting a failure to create a new worker process with a command line like dotnet path/to/MSBuild.dll. Looking at it I'd suspect a runtime problem, maybe System.Process itself?

[ellahathaway]

Looking at it I'd suspect a runtime problem, maybe System.Process itself?

@ericstj - do you know what might be causing this issue?

[uweigand]

[Triage] @uweigand - Is this anything you by chance have seen before? @omajid suggested pinging you. TIA.

Sorry, but no - I've never seen this before.

[ericstj]

System.BadImageFormatException: Method has zero rva

That's a new one for me. It looks like it's coming from the mono runtime.

runtime/src/mono/mono/metadata/loader.c

Line 2065 in 66e45ef

mono_error_set_bad_image (error, img, "Method has zero rva");

Surprised to see the mono runtime being used at all for building the product - I thought we'd only be running on coreclr.

@steveisok @agocke