"CreateAppHost" task failed unexpectedly (MacOS) - Could not set file permission 493

[lilith]

Describe the bug

This bug occurs intermittently with dotnet build (v6), and only on MacOS. Example log from Github Actions

Hint: if you interpret 755 as octal, you get 493. chmod 755 is far more likely than 493.

Microsoft.NET.Sdk.targets(554,5): error MSB40[18] The "CreateAppHost" task failed unexpectedly.
System.ComponentModel.Win32Exception (2): Could not set file permission 493 for [..]/obj/Release/net6.0/apphost.

at Microsoft.NET.HostModel.AppHost.HostWriter.CreateAppHost(String appHostSourceFilePath, String appHostDestinationFilePath, String appBinaryFilePath, Boolean windowsGraphicalUserInterface, String assemblyToCopyResorcesFrom, Boolean enableMacOSCodeSign)

To Reproduce

• Trigger a MacOS build a few times using this repository
• Note that it does not occur every time; this is an intermittent issue but has happened about a dozen times.

Exception

MSBuild version [17](https://github.com/imazen/imageflow-dotnet-server/runs/7986261285?check_suite_focus=true#step:11:18).3.0+92e077650 for .NET
  Determining projects to restore...
  All projects are up-to-date for restore.
  Imazen.Common -> /Users/runner/work/imageflow-dotnet-server/imageflow-dotnet-server/bin/Release/netstandard2.0/Imazen.Common.dll
/Users/runner/.dotnet/sdk/6.0.400/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(554,5): error MSB40[18](https://github.com/imazen/imageflow-dotnet-server/runs/7986261285?check_suite_focus=true#step:11:19): The "CreateAppHost" task failed unexpectedly. [/Users/runner/work/imageflow-dotnet-server/imageflow-dotnet-server/examples/Imageflow.Server.ExampleDockerDiskCache/Imageflow.Server.ExampleDockerDiskCache.csproj]
/Users/runner/.dotnet/sdk/6.0.400/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(554,5): error MSB4018: System.ComponentModel.Win[32](https://github.com/imazen/imageflow-dotnet-server/runs/7986261285?check_suite_focus=true#step:11:33)Exception (2): Could not set file permission 493 for /Users/runner/work/imageflow-dotnet-server/imageflow-dotnet-server/obj/Release/net6.0/apphost. [/Users/runner/work/imageflow-dotnet-server/imageflow-dotnet-server/examples/Imageflow.Server.ExampleDockerDiskCache/Imageflow.Server.ExampleDockerDiskCache.csproj]
/Users/runner/.dotnet/sdk/6.0.[40](https://github.com/imazen/imageflow-dotnet-server/runs/7986261285?check_suite_focus=true#step:11:41)0/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(5[54](https://github.com/imazen/imageflow-dotnet-server/runs/7986261285?check_suite_focus=true#step:11:55),5): error MSB4018:    at Microsoft.NET.HostModel.AppHost.HostWriter.CreateAppHost(String appHostSourceFilePath, String appHostDestinationFilePath, String appBinaryFilePath, Boolean windowsGraphicalUserInterface, String assemblyToCopyResorcesFrom, Boolean enableMacOSCodeSign) [/Users/runner/work/imageflow-dotnet-server/imageflow-dotnet-server/examples/Imageflow.Server.ExampleDockerDiskCache/Imageflow.Server.ExampleDockerDiskCache.csproj]
/Users/runner/.dotnet/sdk/6.0.400/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(554,5): error MSB4018:    at Microsoft.NET.Build.Tasks.CreateAppHost.ExecuteCore() [/Users/runner/work/imageflow-dotnet-server/imageflow-dotnet-server/examples/Imageflow.Server.ExampleDockerDiskCache/Imageflow.Server.ExampleDockerDiskCache.csproj]
/Users/runner/.dotnet/sdk/6.0.400/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(554,5): error MSB4018:    at Microsoft.NET.Build.Tasks.TaskBase.Execute() [/Users/runner/work/imageflow-dotnet-server/imageflow-dotnet-server/examples/Imageflow.Server.ExampleDockerDiskCache/Imageflow.Server.ExampleDockerDiskCache.csproj]
/Users/runner/.dotnet/sdk/6.0.400/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(554,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute() [/Users/runner/work/imageflow-dotnet-server/imageflow-dotnet-server/examples/Imageflow.Server.ExampleDockerDiskCache/Imageflow.Server.ExampleDockerDiskCache.csproj]
/Users/runner/.dotnet/sdk/6.0.400/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(554,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask) [/Users/runner/work/imageflow-dotnet-server/imageflow-dotnet-server/examples/Imageflow.Server.ExampleDockerDiskCache/Imageflow.Server.ExampleDockerDiskCache.csproj]
/Users/runner/.dotnet/sdk/6.0.400/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(554,5): error MSB4018: The "CreateAppHost" task failed unexpectedly. [/Users/runner/work/imageflow-dotnet-server/imageflow-dotnet-server/examples/Imageflow.Server.ExampleDocker/Imageflow.Server.ExampleDocker.csproj]
/Users/runner/.dotnet/sdk/6.0.400/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(554,5): error MSB4018: Microsoft.NET.HostModel.AppHost.AppHostMachOFormatException: Exception of type 'Microsoft.NET.HostModel.AppHost.AppHostMachOFormatException' was thrown. [/Users/runner/work/imageflow-dotnet-server/imageflow-dotnet-server/examples/Imageflow.Server.ExampleDocker/Imageflow.Server.ExampleDocker.csproj]
/Users/runner/.dotnet/sdk/6.0.400/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(554,5): error MSB4018:    at Microsoft.NET.HostModel.AppHost.MachOUtils.RemoveSignature(FileStream stream) [/Users/runner/work/imageflow-dotnet-server/imageflow-dotnet-server/examples/Imageflow.Server.ExampleDocker/Imageflow.Server.ExampleDocker.csproj]
/Users/runner/.dotnet/sdk/6.0.400/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(554,5): error MSB4018:    at Microsoft.NET.HostModel.AppHost.HostWriter.<>c__DisplayClass2_0.<CreateAppHost>b__2() [/Users/runner/work/imageflow-dotnet-server/imageflow-dotnet-server/examples/Imageflow.Server.ExampleDocker/Imageflow.Server.ExampleDocker.csproj]
/Users/runner/.dotnet/sdk/6.0.400/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(554,5): error MSB4018:    at Microsoft.NET.HostModel.RetryUtil.RetryOnIOError(Action func) [/Users/runner/work/imageflow-dotnet-server/imageflow-dotnet-server/examples/Imageflow.Server.ExampleDocker/Imageflow.Server.ExampleDocker.csproj]
/Users/runner/.dotnet/sdk/6.0.400/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(554,5): error MSB4018:    at Microsoft.NET.HostModel.AppHost.HostWriter.CreateAppHost(String appHostSourceFilePath, String appHostDestinationFilePath, String appBinaryFilePath, Boolean windowsGraphicalUserInterface, String assemblyToCopyResorcesFrom, Boolean enableMacOSCodeSign) [/Users/runner/work/imageflow-dotnet-server/imageflow-dotnet-server/examples/Imageflow.Server.ExampleDocker/Imageflow.Server.ExampleDocker.csproj]
/Users/runner/.dotnet/sdk/6.0.400/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(554,5): error MSB4018:    at Microsoft.NET.Build.Tasks.CreateAppHost.ExecuteCore() [/Users/runner/work/imageflow-dotnet-server/imageflow-dotnet-server/examples/Imageflow.Server.ExampleDocker/Imageflow.Server.ExampleDocker.csproj]
/Users/runner/.dotnet/sdk/6.0.400/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(554,5): error MSB4018:    at Microsoft.NET.Build.Tasks.TaskBase.Execute() [/Users/runner/work/imageflow-dotnet-server/imageflow-dotnet-server/examples/Imageflow.Server.ExampleDocker/Imageflow.Server.ExampleDocker.csproj]
/Users/runner/.dotnet/sdk/6.0.400/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(554,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute() [/Users/runner/work/imageflow-dotnet-server/imageflow-dotnet-server/examples/Imageflow.Server.ExampleDocker/Imageflow.Server.ExampleDocker.csproj]
/Users/runner/.dotnet/sdk/6.0.400/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(554,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask) [/Users/runner/work/imageflow-dotnet-server/imageflow-dotnet-server/examples/Imageflow.Server.ExampleDocker/Imageflow.Server.ExampleDocker.csproj]
  Imageflow.Server.Storage.AzureBlob -> /Users/runner/work/imageflow-dotnet-server/imageflow-dotnet-server/bin/Release/netstandard2.0/Imageflow.Server.Storage.AzureBlob.dll
  Imageflow.Server.Storage.RemoteReader -> /Users/runner/work/imageflow-dotnet-server/imageflow-dotnet-server/bin/Release/netstandard2.0/Imageflow.Server.Storage.RemoteReader.dll
  Imageflow.Server.Storage.S3 -> /Users/runner/work/imageflow-dotnet-server/imageflow-dotnet-server/bin/Release/netstandard2.0/Imageflow.Server.Storage.S3.dll
  Imazen.DiskCache -> /Users/runner/work/imageflow-dotnet-server/imageflow-dotnet-server/bin/Release/netstandard2.0/Imazen.DiskCache.dll
  Imazen.HybridCache -> /Users/runner/work/imageflow-dotnet-server/imageflow-dotnet-server/bin/Release/netstandard2.0/Imazen.HybridCache.dll
  Imageflow.Server.HybridCache -> /Users/runner/work/imageflow-dotnet-server/imageflow-dotnet-server/bin/Release/netcoreapp3.1/Imageflow.Server.HybridCache.dll
  Imageflow.Server -> /Users/runner/work/imageflow-dotnet-server/imageflow-dotnet-server/bin/Release/netcoreapp3.1/Imageflow.Server.dll
  Imazen.Common.Tests -> /Users/runner/work/imageflow-dotnet-server/imageflow-dotnet-server/bin/Release/net6.0/Imazen.Common.Tests.dll
  Imazen.HybridCache.Benchmark -> /Users/runner/work/imageflow-dotnet-server/imageflow-dotnet-server/bin/Release/netcoreapp3.1/Imazen.HybridCache.Benchmark.dll
  Imazen.HybridCache.Benchmark -> /Users/runner/work/imageflow-dotnet-server/imageflow-dotnet-server/bin/Release/net6.0/Imazen.HybridCache.Benchmark.dll
  Imageflow.Server.ExampleMinimal -> /Users/runner/work/imageflow-dotnet-server/imageflow-dotnet-server/bin/Release/netcoreapp3.1/Imageflow.Server.ExampleMinimal.dll
  Imageflow.Server.Example -> /Users/runner/work/imageflow-dotnet-server/imageflow-dotnet-server/bin/Release/net6.0/Imageflow.Server.Example.dll
  Imazen.HybridCache.Tests -> /Users/runner/work/imageflow-dotnet-server/imageflow-dotnet-server/bin/Release/net6.0/Imazen.HybridCache.Tests.dll
  Imazen.HybridCache.Tests -> /Users/runner/work/imageflow-dotnet-server/imageflow-dotnet-server/bin/Release/netcoreapp3.1/Imazen.HybridCache.Tests.dll
  Imazen.Common.Tests -> /Users/runner/work/imageflow-dotnet-server/imageflow-dotnet-server/bin/Release/netcoreapp3.1/Imazen.Common.Tests.dll
  Imageflow.Server.Example -> /Users/runner/work/imageflow-dotnet-server/imageflow-dotnet-server/bin/Release/netcoreapp3.1/Imageflow.Server.Example.dll
  Imageflow.Server.Example -> /Users/runner/work/imageflow-dotnet-server/imageflow-dotnet-server/bin/Release/netcoreapp3.1/Imageflow.Server.Example.Views.dll
  Imageflow.Server.DiskCache -> /Users/runner/work/imageflow-dotnet-server/imageflow-dotnet-server/bin/Release/netcoreapp3.1/Imageflow.Server.DiskCache.dll
  Imageflow.Server.Tests -> /Users/runner/work/imageflow-dotnet-server/imageflow-dotnet-server/bin/Release/net6.0/Imageflow.Server.Tests.dll
  Imageflow.Server.Tests -> /Users/runner/work/imageflow-dotnet-server/imageflow-dotnet-server/bin/Release/netcoreapp3.1/Imageflow.Server.Tests.dll

[lilith]

I have tried <_EnableMacOSCodeSign>false</_EnableMacOSCodeSign> per #22201 with no improvement.

[marcpopMSFT]

@vitek-karas @elinor-fung any ideas? Haven't seen a apphost file permissions issue like this before.

[vitek-karas]

@VSadov for other ideas

The above log shows actually two separate failures:
The first one is Win32Exception (2): Could not set file permission 493 for /Users/runner/work/imageflow-dotnet-server/imageflow-dotnet-server/obj/Release/net6.0/apphost. The (2) means the failure is that the file doesn't exist.

The number is 755 octal, just not formatted properly for the error message.

I don't know what could cause the file to not be there. The code right before this failure creates the file using new FileStream which should always create the file (even if it was empty).

The second one is Microsoft.NET.HostModel.AppHost.AppHostMachOFormatException: Exception of type 'Microsoft.NET.HostModel.AppHost.AppHostMachOFormatException' was thrown. - unfortunately there's no detail here even though the code actually remembers a value in the exception which would tell us more about what exactly failed.

This needs improvements in the task to print out more information.

[elinor-fung]

I don't know what could cause the file to not be there. The code right before this failure creates the file using new FileStream which should always create the file (even if it was empty).

It looks like the IntermediateOutputPath where we put apphost is set to the same path for both Imageflow.Server.ExampleDockerDiskCache (that hit the Win32Exception) and Imageflow.Server.ExampleDocker (that hit the AppHostMachOFormatException). We delete apphost if we fail to update it, so maybe the build for one project fails to remove the signature and deletes apphost some time between when the build for the other porject creates the file and tries to change its permissions.

I wonder if the AppHostMachOFormatException might also be from not handling multiple builds operating on the same apphost - but as Vitek said, the task needs his change to include more information.

[lilith]

I think this might very well be related. I'm going to try dotnet build --maxcpucount:1 #22201 (comment)

[lilith]

I'll let you know if a failure recurs, but I think you're right about it being a concurrency issue.

[marcpopMSFT]

Since Vitek's improvements were checked in and the issue was likely concurrency, closing for now. Reactivate if this comes back.