Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Mono AOT Compiler packs are missing data/UnixFilePermissions.xml? #82201

Closed
jonathanpeppers opened this issue Feb 15, 2023 · 24 comments · Fixed by #82211
Closed

Mono AOT Compiler packs are missing data/UnixFilePermissions.xml? #82201

jonathanpeppers opened this issue Feb 15, 2023 · 24 comments · Fixed by #82211
Assignees
@directhex
Labels
area-Build-mono
Milestone
8.0.0

Comments

@jonathanpeppers
Copy link
Member

jonathanpeppers commented Feb 15, 2023 •
edited

Description

@davidortinau was testing .NET 8 preview 1 builds and hit the error:

/usr/local/share/dotnet/packs/Microsoft.MacCatalyst.Sdk/16.2.232-net8-p1/targets/Xamarin.Shared.Sdk.targets(1037,3): error MSB4018: The "AOTCompile" task failed unexpectedly.
System.AggregateException: One or more errors occurred. (One or more errors occurred. (An error occurred trying to start process '/usr/local/share/dotnet/packs/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.maccatalyst-x64/8.0.0-preview.1.23106.5/Sdk/../tools/mono-aot-cross' with working directory '/Users/dmo/work/ControlGallery/src/ControlGallery'. Permission denied))
 ---> System.AggregateException: One or more errors occurred. (An error occurred trying to start process '/usr/local/share/dotnet/packs/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.maccatalyst-x64/8.0.0-preview.1.23106.5/Sdk/../tools/mono-aot-cross' with working directory '/Users/dmo/work/ControlGallery/src/ControlGallery'. Permission denied)
 ---> System.ComponentModel.Win32Exception (13): An error occurred trying to start process '/usr/local/share/dotnet/packs/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.maccatalyst-x64/8.0.0-preview.1.23106.5/Sdk/../tools/mono-aot-cross' with working directory '/Users/dmo/work/ControlGallery/src/ControlGallery'. Permission denied
   at System.Diagnostics.Process.ForkAndExecProcess(ProcessStartInfo startInfo, String resolvedFilename, String[] argv, String[] envp, String cwd, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, Int32& stdinFd, Int32& stdoutFd, Int32& stderrFd, Boolean usesTerminal, Boolean throwOnNoExec)
   at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo)
   at Xamarin.Utils.Execution.<>c__DisplayClass24_1.<RunAsync>b__0() in /Users/builder/azdo/_work/1/s/xamarin-macios/tools/common/Execution.cs:line 103
--- End of stack trace from previous location ---
   at Xamarin.MacDev.Tasks.XamarinTask.ExecuteAsync(String fileName, IList`1 arguments, String sdkDevPath, Dictionary`2 environment, Boolean mergeOutput, Boolean showErrorIfFailure, String workingDirectory) in /Users/builder/azdo/_work/1/s/xamarin-macios/msbuild/Xamarin.MacDev.Tasks/Tasks/XamarinTask.cs:line 120
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at Xamarin.MacDev.Tasks.AOTCompileTaskBase.<>c__DisplayClass36_0.<Execute>b__7(Task`1 v) in /Users/builder/azdo/_work/1/s/xamarin-macios/msbuild/Xamarin.MacDev.Tasks/Tasks/AOTCompileTaskBase.cs:line 108
   at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.WaitAllCore(Task[] tasks, Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.WaitAll(Task[] tasks)
   at Xamarin.MacDev.Tasks.AOTCompileTaskBase.Execute() in /Users/builder/azdo/_work/1/s/xamarin-macios/msbuild/Xamarin.MacDev.Tasks/Tasks/AOTCompileTaskBase.cs:line 115
   at Xamarin.MacDev.Tasks.AOTCompile.Execute() in /Users/builder/azdo/_work/1/s/xamarin-macios/msbuild/Xamarin.MacDev.Tasks/Tasks/AOTCompile.cs:line 14
   at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute()
   at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask) [/Users/dmo/work/ControlGallery/src/ControlGallery/ControlGallery.csproj]

We found that running this fixed it:

chmod +x /usr/local/share/dotnet/packs/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.maccatalyst-x64/8.0.0-preview.1.23106.5/tools/mono-aot-cross

The Android/iOS workloads define a data/UnixFilePermission.xml file, so that dotnet workload commands can supply the execute bit. Details at: dotnet/sdk#16894

We think this was implemented sometime in dotnet/runtime: #54501

But I don't see the file here:

https://dev.azure.com/dnceng/public/_artifacts/feed/dotnet8/NuGet/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.maccatalyst-x64/overview/8.0.0-preview.1.23106.5

Reproduction Steps

Expected behavior

You should not need to chmod +x mono-aot-cross

Actual behavior

Some Mac machines you have to chmod +x mono-aot-cross.

Regression?

Yes

Known Workarounds

chmod +x /usr/local/share/dotnet/packs/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.maccatalyst-x64/8.0.0-preview.1.23106.5/tools/mono-aot-cross

Configuration

.NET SDK 8.0.100-preview.1.23109.10

Other information

No response
@dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

@ghost ghost added the untriaged New issue has not been triaged by the area owner label Feb 15, 2023
@jonathanpeppers
Copy link
Member Author

/cc @steveisok

@steveisok
Copy link
Member

@directhex, not sure how this regressed. Please look into it.

@steveisok steveisok added this to the 8.0.0 milestone Feb 15, 2023
@ghost ghost removed the untriaged New issue has not been triaged by the area owner label Feb 15, 2023
@directhex
Copy link
Member

This change was made intentionally in b6e2dbe which was part of #75200

@radical do you know what spurred the "non-Windows no longer needs execute permissions" change?

@radical
Copy link
Member

radical commented Feb 15, 2023

Nuget should have the correct permissions now, that was my understanding.

@radical
Copy link
Member

radical commented Feb 15, 2023

hm looks like that depends on this file though?

@directhex
Copy link
Member

Where does Nuget get the correct permissions from? The underlying .zip format doesn't support permissions without non-standard extended attributes which afaik aren't supported in dotnet's zip implementation

@directhex
Copy link
Member

(making every extracted file executable is one approach which doesn't rely on attributes or a metadata xml)

@radical
Copy link
Member

radical commented Feb 15, 2023

IIRC, dotnet's zip implementation had fixes for that, and thus this wasn't required any more. but maybe I misunderstood some part of it.

@directhex
Copy link
Member

It looks like #70735 went in which guarantees permissions on files are read-write and directories are read-write-execute. I don't see anything in that change about adding execute permissions to files.

@radical
Copy link
Member

radical commented Feb 15, 2023

this - #55531
So it should capture the original permissions. But if the package is built on windows then maybe it won't have anything, and the xml file would be useful for that case?

@radical
Copy link
Member

radical commented Feb 15, 2023

To be clear, I'm fine with reverting the change. I'm just trying to understand what is exactly going on here.

@directhex
Copy link
Member

I don't think permissions are being captured during Nuget package generation. I'm running some local tests on zip file generation/unpacking with command-line non-dotnet tools, and have execute perms preserved across zip/unzip. But unzipping a locally generated nupkg with unzip isn't preserving the execute permissions which are on the llc/opt/mono-aot-cross used to build the package. Which to me indicates those values aren't being stored.

@directhex
Copy link
Member

I can chase the Nuget side of things, but as-is, it looks like Preview 1 just doesn't have working cross-compilers on non-Windows. Which sucks a bit. Is it too late to get a revert in?

@radical
Copy link
Member

radical commented Feb 15, 2023

@steveisok could comment on whether we can get a revert in or not.

I wonder why it hasn't broken wasm workloads though

@directhex
Copy link
Member

OK, there's a -Z flag to unzip to validate it with.

Archive:  /home/directhex/Projects/runtime/artifacts/packages/Release/Shipping/Microsoft.NETCore.App.Runtime.AOT.linux-x64.Cross.android-x64.8.0.0-dev.nupkg
Zip file size: 49624622 bytes, number of entries: 15
-rw-r--r--  2.0 unx      551 b- defN 23-Feb-15 21:56 _rels/.rels
-rw-r--r--  2.0 unx      997 b- defN 23-Feb-15 21:56 Microsoft.NETCore.App.Runtime.AOT.linux-x64.Cross.android-x64.nuspec
-rw-r--r--  2.0 unx      173 b- defN 23-Feb-15 21:56 Sdk/Sdk.props
-rw-r--r--  2.0 unx 47022208 b- defN 23-Feb-13 18:18 tools/llc
-rw-r--r--  2.0 unx 36135376 b- defN 23-Feb-15 21:55 tools/mono-aot-cross
-rw-r--r--  2.0 unx 53036464 b- defN 23-Feb-13 18:18 tools/opt
-rw-r--r--  2.0 unx      839 b- defN 23-Feb-15 21:56 tools/monocrossaot.deps.json
-rw-r--r--  2.0 unx      152 b- defN 23-Feb-15 21:56 tools/monocrossaot.runtimeconfig.json
-rw-r--r--  2.0 unx     1116 b- defN 20-Sep-28 21:50 LICENSE.TXT
-rw-r--r--  2.0 unx      152 b- defN 23-Feb-15 21:56 data/RuntimeList.xml
-rw-r--r--  2.0 unx     7006 b- defN 23-Feb-13 15:43 Icon.png
-rw-r--r--  2.0 unx        0 b- stor 21-Jan-19 20:13 useSharedDesignerContext.txt
-rw-r--r--  2.0 unx    64000 b- defN 23-Jan-25 17:38 THIRD-PARTY-NOTICES.TXT
-rw-r--r--  2.0 unx      928 b- defN 23-Feb-15 21:56 [Content_Types].xml
-rw-r--r--  2.0 unx      760 b- defN 23-Feb-15 21:56 package/services/metadata/core-properties/ca04b5010d49490499790adeb0fef710.psmdcp
15 files, 136270722 bytes uncompressed, 49622690 bytes compressed:  63.6%

note the lack of x on the files in tools/

@steveisok
Copy link
Member

@steveisok could comment on whether we can get a revert in or not.

No, it's too late unfortunately.

@lewing
Copy link
Member

lewing commented Feb 15, 2023 •
edited

lets revert the change asap and add known-issues to the release notes

radical added a commit to radical/runtime that referenced this issue Feb 15, 2023
@radical
Workloads: Fix a regression in file permissions for workload packs
93e8f00 
Reverts
dotnet@b6e2dbe
essentially.

Fixes dotnet#82201 .
@radical radical mentioned this issue Feb 15, 2023
Merged
@ghost ghost added the in-pr There is an active PR which will close this issue when it is merged label Feb 15, 2023
@radical
Copy link
Member

radical commented Feb 15, 2023

@davidortinau @jonathanpeppers How was this workload installed?

@jonathanpeppers
Copy link
Member Author

The gist above has instructions, but it was a regular dotnet workload install command.

@radical
Copy link
Member

radical commented Feb 16, 2023

  • I removed /usr/local/share/dotnet completely
  • Installed dotnet-sdk-8.0.100-preview.1.23109.10-osx-arm64.pkg.
  • And then ran sudo dotnet workload install maui --source https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet8/nuget/v3/index.json --source https://api.nuget.org/v3/index.json --from-rollback-file https://maui.blob.core.windows.net/metadata/rollbacks/9567bfa68859d611cc9aad2555fd9847f8233cda.json --verbosity diag.
$ find /usr/local/share/dotnet -name mono-aot-cross -ls
25088028    25808 -rwxr-xr-x    1 root             wheel            13213312 Sep 26 14:04 /usr/local/share/dotnet/packs/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.ios-arm/6.0.10/tools/mono-aot-cross
25085902    32056 -rwxr--r--    1 root             wheel            16409472 Feb 10 16:37 /usr/local/share/dotnet/packs/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.ios-arm/8.0.0-preview.1.23110.8/tools/mono-aot-cross
25088123    26080 -rwxr-xr-x    1 root             wheel            13350752 Sep 26 13:54 /usr/local/share/dotnet/packs/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.iossimulator-x64/6.0.10/tools/mono-aot-cross
25086007    32360 -rwxr--r--    1 root             wheel            16564568 Feb 10 16:26 /usr/local/share/dotnet/packs/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.iossimulator-x64/8.0.0-preview.1.23110.8/tools/mono-aot-cross
25088086    25880 -rwxr-xr-x    1 root             wheel            13248264 Sep 26 13:56 /usr/local/share/dotnet/packs/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.iossimulator-arm64/6.0.10/tools/mono-aot-cross
25085966    32192 -rwxr--r--    1 root             wheel            16478896 Feb 10 16:28 /usr/local/share/dotnet/packs/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.iossimulator-arm64/8.0.0-preview.1.23110.8/tools/mono-aot-cross
25091042    26080 -rwxr-xr-x    1 root             wheel            13352192 Sep 26 14:07 /usr/local/share/dotnet/packs/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.maccatalyst-x64/6.0.10/tools/mono-aot-cross
25090309    32360 -rwxr--r--    1 root             wheel            16564536 Feb 10 16:39 /usr/local/share/dotnet/packs/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.maccatalyst-x64/8.0.0-preview.1.23110.8/tools/mono-aot-cross
25083648    25264 -rwxr-xr-x    1 root             wheel            12931608 Oct 18 12:29 /usr/local/share/dotnet/packs/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.android-arm64/7.0.0/tools/mono-aot-cross
25082199    25872 -rwxr-xr-x    1 root             wheel            13245840 Sep 26 13:36 /usr/local/share/dotnet/packs/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.android-arm64/6.0.10/tools/mono-aot-cross
25085048    31936 -rwxr--r--    1 root             wheel            16351216 Feb 10 16:07 /usr/local/share/dotnet/packs/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.android-arm64/8.0.0-preview.1.23110.8/tools/mono-aot-cross
25083587    25456 -rwxr-xr-x    1 root             wheel            13031160 Oct 18 12:25 /usr/local/share/dotnet/packs/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.android-x64/7.0.0/tools/mono-aot-cross
25082132    26040 -rwxr-xr-x    1 root             wheel            13331776 Sep 26 13:33 /usr/local/share/dotnet/packs/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.android-x64/6.0.10/tools/mono-aot-cross
25084991    32104 -rwxr--r--    1 root             wheel            16434920 Feb 10 16:03 /usr/local/share/dotnet/packs/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.android-x64/8.0.0-preview.1.23110.8/tools/mono-aot-cross
25088057    25872 -rwxr-xr-x    1 root             wheel            13246104 Sep 26 14:02 /usr/local/share/dotnet/packs/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.ios-arm64/6.0.10/tools/mono-aot-cross
25085934    32184 -rwxr--r--    1 root             wheel            16476824 Feb 10 16:34 /usr/local/share/dotnet/packs/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.ios-arm64/8.0.0-preview.1.23110.8/tools/mono-aot-cross
25083553    25216 -rwxr-xr-x    1 root             wheel            12908240 Oct 18 12:34 /usr/local/share/dotnet/packs/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.android-x86/7.0.0/tools/mono-aot-cross
25082100    25864 -rwxr-xr-x    1 root             wheel            13241864 Sep 26 13:39 /usr/local/share/dotnet/packs/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.android-x86/6.0.10/tools/mono-aot-cross
25084959    31896 -rwxr--r--    1 root             wheel            16327576 Feb 10 16:10 /usr/local/share/dotnet/packs/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.android-x86/8.0.0-preview.1.23110.8/tools/mono-aot-cross
25088154    25872 -rwxr-xr-x    1 root             wheel            13244456 Sep 26 13:59 /usr/local/share/dotnet/packs/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.iossimulator-x86/6.0.10/tools/mono-aot-cross
25086059    32112 -rwxr--r--    1 root             wheel            16441040 Feb 10 16:31 /usr/local/share/dotnet/packs/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.iossimulator-x86/8.0.0-preview.1.23110.8/tools/mono-aot-cross
25091011    25944 -rwxr-xr-x    1 root             wheel            13282680 Sep 26 14:10 /usr/local/share/dotnet/packs/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.maccatalyst-arm64/6.0.10/tools/mono-aot-cross
25090264    32224 -rwxr--r--    1 root             wheel            16495416 Feb 10 16:42 /usr/local/share/dotnet/packs/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.maccatalyst-arm64/8.0.0-preview.1.23110.8/tools/mono-aot-cross
25083618    25152 -rwxr-xr-x    1 root             wheel            12877528 Oct 18 12:39 /usr/local/share/dotnet/packs/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.android-arm/7.0.0/tools/mono-aot-cross
25082164    25808 -rwxr-xr-x    1 root             wheel            13211760 Sep 26 13:41 /usr/local/share/dotnet/packs/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.android-arm/6.0.10/tools/mono-aot-cross
25085021    31832 -rwxr--r--    1 root             wheel            16296872 Feb 10 16:13 /usr/local/share/dotnet/packs/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.android-arm/8.0.0-preview.1.23110.8/tools/mono-aot-cross

All of the files have +x set!

@directhex
Copy link
Member

@radical try executing one of those as non-root 🙃

@radical
Copy link
Member

radical commented Feb 16, 2023

Lol root owned and u+x, duh! 😬

@directhex
Copy link
Member

🙈

radical added a commit that referenced this issue Feb 16, 2023
@radical
Workloads: Fix a regression in file permissions for workload packs (#…
6dfc7d3 
…82211)

Reverts
b6e2dbe
essentially.

Fixes #82201 .

This especially shows up on macOS when doing a system install. When installing the workload to a user directory, the permissions get set correctly, which is why this was not caught by Wasm.Build.Tests .
@ghost ghost removed the in-pr There is an active PR which will close this issue when it is merged label Feb 16, 2023
@rolfbjarne rolfbjarne mentioned this issue Feb 24, 2023
Closed
@ghost ghost locked as resolved and limited conversation to collaborators Mar 18, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@directhex directhex

Labels
area-Build-mono
Projects
None yet
Milestone
8.0.0
Development

Successfully merging a pull request may close this issue.

Workloads: Fix a regression in file permissions for workload packs radical/runtime
5 participants
@radical @lewing @directhex @steveisok @jonathanpeppers