[mono] msbuild aborts due to 4.7 dependencies

[uweigand]

Building the .NET 6 final source-build on s390x (using the Mono runtime) results in an msbuild assembly that will always crash on startup, via an exception along these lines:

MSBUILD : error MSB1025: An internal failure occurred while running MSBuild. 
System.TypeLoadException: Could not load type of field 'Microsoft.Build.Evaluation.ToolsetConfigurationReader:_configurationSection' (0) due to: Could not load file or assembly 'System.Configuration.ConfigurationManager, Version=4.0.3.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' or one of its dependencies. 
  at Microsoft.Build.Evaluation.ProjectCollection.InitializeToolsetCollection(ToolsetConfigurationReader configReader) 
  at Microsoft.Build.Evaluation.ProjectCollection..ctor(IDictionary`2 globalProperties, IEnumerable`1 loggers, IEnumerable`1 remoteLoggers, ToolsetDefinitionLocations toolsetDefinitionLocations, Int32 maxNodeCount, Boolean onlyLogCriticalEvents, Boolean loadProjectsReadOnly) 
  at Microsoft.Build.CommandLine.MSBuildApp.BuildProject(String projectFile, String[] targets, String toolsVersion, Dictionary`2 globalProperties, Dictionary`2 restoreProperties, ILogger[] loggers, LoggerVerbosity verbosity, DistributedLoggerRecord[] distributedLoggerRecords, Int32 cpuCount, Boolean enableNodeReuse, TextWriter preprocessWriter, TextWriter targetsWriter, Boolean detailedSummary, ISet`1 warningsAsErrors, ISet`1 warningsAsMessages, Boolean enableRestore, ProfilerLogger profilerLogger, Boolean enableProfiler, Boolean interactive, Boolean isolateProjects, GraphBuildOptions graphBuildOptions, Boolean lowPriority, String[] inputResultsCaches, String outputResultsCache) 
  at Microsoft.Build.CommandLine.MSBuildApp.Execute(String[] commandLine)

The System.Configuration.ConfigurationManager assembly that is missing here is hard-coded to use a 4.7.0 version in the msbuild eng/Packages.props file:

    <PackageReference Update="System.Configuration.ConfigurationManager" Version="4.7.0" />

When doing a non-source-build build, the version of this assembly gets restored from a nuget package and is actually included in the resulting SDK tarball.

However, when using source-build, that nuget package is not used, and the assembly is not present in the SDK tarball, but there is still a dependency on it from msbuild, as can be seen e.g. in the sdk/6.0.100/MSBuild.deps.json file that is in the tarball:

      "Microsoft.Build/17.0.0": {
        "dependencies": {
          "Microsoft.Build.Framework": "17.0.0",
          "Microsoft.NET.StringTools": "1.0.0",
          "Microsoft.Win32.Registry": "4.3.0",
          "System.Collections.Immutable": "5.0.0",
          "System.Configuration.ConfigurationManager": "4.7.0",
          "System.Reflection.Metadata": "1.8.0",
          "System.Security.Principal.Windows": "4.7.0",
          "System.Text.Encoding.CodePages": "6.0.0",
          "System.Text.Json": "6.0.0",
          "System.Threading.Tasks.Dataflow": "4.9.0"
        },

Types defined by the missing assembly are used for members of ToolsetConfigurationReader, which is why the Mono JIT aborts when attempting to build the constructor (or other functions) of that class. This happens in particular when building the routine InitializeToolsetCollection from the above error message, which is always called on msbuild startup.

Now, I think that this happens to work even in the absence of that assembly on CoreCLR, because the ToolsetConfigurationReader is not actually used when running msbuild on Linux, instances of it are just passed around. That is because any actual use of the configuration reader is guarded (in ToolsetReader.ReadAllToolsets) by this check:

            if ((locations & ToolsetDefinitionLocations.ConfigurationFile) == ToolsetDefinitionLocations.ConfigurationFile)

and the default value of locations is provided by Microsoft.Build.Evaluation.ToolsetDefinitionLocation:Default, which is defined as:

        Default = None
#if FEATURE_SYSTEM_CONFIGURATION
                | ConfigurationFile
#endif
#if FEATURE_REGISTRY_TOOLSETS
                | Registry
#endif
#if !FEATURE_SYSTEM_CONFIGURATION && !FEATURE_REGISTRY_TOOLSETS
                | Local
#endif

Note that when building on Linux, the FEATURE_SYSTEM_CONFIGURATION define is false, and therefore ConfigurationFile is not included in the Default value.

And indeed, adding a FEATURE_SYSTEM_CONFIGURATION guard to all places where ToolsetConfigurationReader is referenced makes the abort go away, with no apparent adverse effect on the operation of msbuild otherwise. However, there are still places in the msbuild unit test that apparently deliberately use the ConfigurationFile setting even on Linux - not sure what to do about those.

---

There is a second missing dependency on a 4.7 assembly, System.Security.Permissions, which causes similar aborts during JIT. In this case, the function Microsoft.Build.Shared.ExceptionHandling.IsXmlException cannot be compiled on Mono as it refers to a type XmlSyntaxException defined by System.Security.Permissions.

This case triggers much more rarely, only when msbuild runs into some other exception that it would otherwise be able to recover from. I've seen this when building the runtime repo on s390x - msbuild gets a DirectoryNotFoundException during a restore operation since some crossgen2 directory is not present (due to building for Mono which doesn't support crossgen2), and it would normally just recover and ignore that exception, but in the process the IsXmlException routine is called, but it cannot be JITed.

Putting the use of XmlSyntaxException under a #if FEATURE_SECURITY_PERMISSIONS guard makes that abort go away for me as well. With those two work-arounds applies, I can bootstrap source-build (i.e run the source-build using a source-built SDK) on s390x successfully.

---

It seems strange that a Linux build of the SDK even has any references to those 4.7 assemblies (which I understand are Windows only?) in the first place. What is the appropriate fix for these issues?

CC @crummel @leecow @omajid @tmds

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[omajid]

I filed #2596 to track getting CI with a mono-based stack to identify these mono-only issues earlier.

[MichaelSimons]

[Triage] This issue is tracking the overall s390x/mono issue. @crummel, Can you link the related issues in the product repos.

[MichaelSimons]

[Triage] @crummel can you check if the underlying issue has been resolved?

[crummel]

This is still a relevant issue although it has been worked around. It's a symptom of the whole issue with mono trying to load the MSBuild full-framework assemblies eagerly instead of delaying loading them until they are needed like CoreCLR does, which causes issues because they are missing in source-built packages.

[akoeplinger]

Do we have an issue in dotnet/runtime which tracks this for mono?

[uweigand]

Do we have an issue in dotnet/runtime which tracks this for mono?

My understanding was that the Mono behavior isn't actually wrong, just different from CoreCLR - the CIL standard allows either. That's why I opened this as a bug here to fix the code that contains those references, instead of against Mono.

[MichaelSimons]

[Triage] @crummel - please validate this is still an issue for .net 7.0.

[uweigand]

@MichaelSimons in .net 7.0 this particular issue no longer shows up, as msbuild no longer use 4.7.0 versions of these libraries, but was upgraded to 6.0 here:
dotnet/msbuild#7629
Later on, the version number was also made configurable so it can be overridden in source-build:
dotnet/msbuild#7809

[MichaelSimons]

@uweigand - is a fix needed for 6.0 or can this issue be closed?

[uweigand]

@uweigand - is a fix needed for 6.0 or can this issue be closed?

I don't think we need a 6.0 fix at this time. The Red Hat RPMs for 6.0 still have a local workaround applied.