macOS PR validation smoke-test failure: libicu not installed/detected in smoke-test

[dagood]

https://dev.azure.com/dnceng/public/_build/results?buildId=742764&view=logs&j=dd0961f5-eb75-5263-969b-8fed4b0393f9&t=e3c3f94f-154c-5989-8736-91280ba7f044&l=75

  SDK under test is:
  Process terminated. Couldn't find a valid ICU package installed on the system. Set the configuration flag System.Globalization.Invariant to true if you want to run with no globalization support.
     at System.Environment.FailFast(System.String)
     at System.Globalization.GlobalizationMode.GetGlobalizationInvariantMode()
     at System.Globalization.GlobalizationMode..cctor()
     at System.Globalization.CultureData.CreateCultureWithInvariantData()
     at System.Globalization.CultureData.get_Invariant()
     at System.Globalization.CultureInfo..cctor()
     at System.Version.TryParseComponent(System.ReadOnlySpan`1<Char>, System.String, Boolean, Int32 ByRef)
     at System.Version.ParseVersion(System.ReadOnlySpan`1<Char>, Boolean)
     at System.Version.TryParse(System.String, System.Version ByRef)
     at Microsoft.DotNet.PlatformAbstractions.Native.PlatformApis.GetDarwinVersion()
     at Microsoft.DotNet.PlatformAbstractions.Native.PlatformApis.GetOSVersion()
     at Microsoft.DotNet.PlatformAbstractions.RuntimeEnvironment..cctor()
     at Microsoft.DotNet.PlatformAbstractions.RuntimeEnvironment.GetRuntimeIdentifier()
     at Microsoft.DotNet.Cli.MulticoreJitProfilePathCalculator.CalculateProfileRootPath()
     at Microsoft.DotNet.Cli.MulticoreJitActivator.StartCliProfileOptimization()
     at Microsoft.DotNet.Cli.MulticoreJitActivator.TryActivateMulticoreJit()
     at Microsoft.DotNet.Cli.Program.Main(System.String[])
  ./smoke-test.sh: line 399: 86759 Abort trap: 6           "$dotnetDir/dotnet" --info
/Users/runner/work/1/s/build.proj(230,5): error MSB3073: The command "./smoke-test.sh --minimal --projectOutput --configuration Release --archiveRestoredPackages --excludeWebHttpsTests" exited with code 134.

This seems to be flakiness: the PR that hit this didn't change anything about this job, and the "Initialize job" step appears identical to an earlier PR validation run that failed. Time will tell.

[dagood]

This may not be flakiness... happened 3 times in a row on https://dev.azure.com/dnceng/public/_build/results?buildId=742764&view=results. Pool configuration change maybe? FYI @dseefeld @crummel

[dseefeld]

@dotnet/dnceng Is this a known issue or configuration issue for OSX images?

[MattGal]

Taking a peek

[MattGal]

Couple things:

• Checking out https://github.com/actions/virtual-environments, it doesn't look like they've ever consciously been installing libICU on the hosted macOS machines.
• There is a movement currently to move from 3rd to 1st-party hosted mac machines; the agents running on them would be the same but dependencies like this may have been missed.
• The hosted macOS builds do run on VMs, so if you can feasibly bootstrap the version you want, party on.

I need to talk to folks in AzDO about their timeline of moving to the "new" macs; this was supposed to be completed last week but is still ongoing. This is likely your problem as I see the same PR building on different days hitting the problem and not.

[MattGal]

I chatted with @safern and @tarekgh who may want to weigh in here. There has been code change in this area that you may be missing.

Please see dotnet/runtime#39900, specifically this commit.

[safern]

Just to add to what @MattGal mentioned, the extended version of the fix in order to support building runtime in macOS Big Sur is: dotnet/runtime#39900

[dagood]

@safern will that be ported to 3.1 where we hit this error? Is there a tracking issue?

[safern]

@safern will that be ported to 3.1 where we hit this error? Is there a tracking issue?

I believe @ViktorHofer was porting that to downstream branches.

[dagood]

A few more bits to connect the dots between this and dotnet/runtime#39900: it looks like originally this problem came up in https://github.com/dotnet/core-eng/issues/10304, which shows up exactly the same way it does for us. The initial fix was dotnet/runtime#39833:

To sum up the findings, this is because of the recent cmake upgrade to 3.18 that got pushed to brew with Homebrew/homebrew-core@7927380.

A build that ran 8hours ago (presumably with cmake 3.17.2) still got the expected /usr/lib/libicucore.dylib, a build from 4hours ago got the Xcode path.

Filed an issue with CMake: https://gitlab.kitware.com/cmake/cmake/-/issues/21007

Then dotnet/runtime#39900 looks like an improved fix in line with new macOS changes.

[safern]

Then dotnet/runtime#39900 looks like an improved fix in line with new macOS changes.

@ViktorHofer ported it to coreclr3.1 which is the one that builds S.Globalization.Native in 3.1. dotnet/coreclr@d3a7344

[safern]

Note that only the simple fix was ported. If we need to build on Big Sur for 3.1 we can also port the complete fix.

[ViktorHofer]

The fix was backported into the 2.1, 2.2 and 3.1 branches. Here's the change for 3.1: dotnet/coreclr@d3a7344.