Could not AOT the Assemblies - Visual Studio 16.10

[EP01]

Steps to Reproduce

1. Update to VS 16.10.0.
2. Try to archive a Xamarin Android app with AOT + LLVM enabled. I used the default Xamarin Forms template to reproduce easily by just changing those two parameters in Release config and pressing Archive. I've attached this.
3. Fails

App2.zip

[dellis1972]

This looks like a duplicate of #5764

[EP01]

Sorry @dellis1972, I missed that because it was raised in March, I assumed it would have been amongst the more recent issues.

[dellis1972]

@EP01 no problem :D

[inforithmics]

@dellis1972 Can you reopen this issue. Because #5764 is a different issue.

I discovered that llvm with Visual Studio 2019 16.10 doesn't work.
The reason is that the Android sdk path has a space and then the ndk doesn't work see comments in the other issue.

#5764 (comment)

Works:
C:\Users\[username]\AppData\Local\Android\sdk
Doesn't work:
C:\program Files (x86)\Android\android-sdk

And altough azure devops builds start failing because of this.

[AndreyBespamyatnov]

The same issue for AzureDevOps builds:

2021-05-31T21:49:31.2416389Z   [aot-compiler stderr] C:\Program Files (x86)\Android\android-sdk\ndk\21.4.7075529\toolchains\llvm\prebuilt\windows-x86_64\bin\arm-linux-androideabi-ld.EXE: error: cannot open (x86)\Android\android-sdk\ndk\21.4.7075529\toolchains\llvm\prebuilt\windows-x86_64\lib\gcc\arm-linux-androideabi\4.9.x: No such file or directory
2021-05-31T21:49:31.2417574Z   [aot-compiler stderr] C:\Program Files (x86)\Android\android-sdk\ndk\21.4.7075529\toolchains\llvm\prebuilt\windows-x86_64\bin\arm-linux-androideabi-ld.EXE: error: cannot open Files: No such file or directory
2021-05-31T21:49:31.2419262Z   [aot-compiler stderr] C:\Program Files (x86)\Android\android-sdk\ndk\21.4.7075529\toolchains\llvm\prebuilt\windows-x86_64\bin\arm-linux-androideabi-ld.EXE: error: cannot open (x86)\Android\android-sdk\ndk\21.4.7075529\platforms\android-21\arch-arm\usr\lib: No such file or directory

[VictorKochetkov]

The same issue for AzureDevOps builds:

2021-05-31T21:49:31.2416389Z   [aot-compiler stderr] C:\Program Files (x86)\Android\android-sdk\ndk\21.4.7075529\toolchains\llvm\prebuilt\windows-x86_64\bin\arm-linux-androideabi-ld.EXE: error: cannot open (x86)\Android\android-sdk\ndk\21.4.7075529\toolchains\llvm\prebuilt\windows-x86_64\lib\gcc\arm-linux-androideabi\4.9.x: No such file or directory
2021-05-31T21:49:31.2417574Z   [aot-compiler stderr] C:\Program Files (x86)\Android\android-sdk\ndk\21.4.7075529\toolchains\llvm\prebuilt\windows-x86_64\bin\arm-linux-androideabi-ld.EXE: error: cannot open Files: No such file or directory
2021-05-31T21:49:31.2419262Z   [aot-compiler stderr] C:\Program Files (x86)\Android\android-sdk\ndk\21.4.7075529\toolchains\llvm\prebuilt\windows-x86_64\bin\arm-linux-androideabi-ld.EXE: error: cannot open (x86)\Android\android-sdk\ndk\21.4.7075529\platforms\android-21\arch-arm\usr\lib: No such file or directory

Same issue on home PC and on Github actions

[jonpryor]

@AndreyBespamyatnov or @VictorKochetkov : would either of you be able to provide a complete .binlog file, or a diagnostic build log, for when your build fails?

I'm confused, because the GitHub Actions Microsoft Windows Server 2019 Datacenter Image states that the Android NDK is installed under C:\Android\android-sdk, a directory that contains no spaces. I thus wonder how $(AndroidSdkDirectory) is being detected in a manner which causes it to find an NDK path which is not documented on GitHub Actions.

[jonpryor]

While trying to investigate what's going on…

d16-10 install with the following files from 16-9: cross-*.exe:

• Fails; suggests it's not (just?) the mono bump.

d16-10 install with the following files from d16-9: opt.exe, llc.exe:

• Fails; suggests it's not due to any LLVM bumps part of the mono bump.

d16-10 install with the following files from d16-9: Xamarin.Android.Aapt2.targets, Xamarin.Android.Build.Tasks.dll, Xamarin.Android.Common.targets, Xamarin.Android.Legacy.targets, Java.Interop.Tools.JavaCallableWrappers.dll

• Works!

Which implies a xamarin-android change.

The change to Java.Interop.Tools.JavaCallableWrappers.dll doesn't seem plausible in the context of AOT.

This also allows us to ask a related question: Why isn't the path quoting in commit 6a5df16 needed? How did this work in d16-9? (I could install d16-9, but now that I have a "Frankenstein" build environment…)

The answer? -LC: does not appear in the build log, and it was the presence of -LC:\Path With Spaces\… in the ld-flags value created by the <Aot/> task and passed into mono which caused the problem. ld-flags only contains the (quoted!) paths to libgcc.a, libc.so, and libm.so.

This explains why the lack of quotes around toolchainLibDir and androidLibPath wasn't previously a problem; they weren't previously used:

https://github.com/xamarin/xamarin-android/blob/d7282f4385146469958cc922d99138553d653fe8/src/Xamarin.Android.Build.Tasks/Tasks/Aot.cs#L360-L369

They weren't previously used because, by inference, NdkUtil.UsingClangNDK was false. Now it's (presumably) true, which causes the -LC:\… paths to be used, which broke the world.

This allows us to focus on NdkUtil.UsingClangNDK semantics and possible changes…

[jonpryor]

Now we start getting to where I can't as easily provide URLs for diffs…

What changed in NdkUtils.cs?

git diff origin/d16-9...origin/d16-10 src/Xamarin.Android.Build.Tasks/Tasks/NdkUtils.cs

Nothing directly related to the UsingClangNDK property or usingClangNDK field, which means we're left with the original logic:

https://github.com/xamarin/xamarin-android/blob/0e5e06f716c7e9044e66ed27b75fcd722c546127/src/Xamarin.Android.Build.Tasks/Tasks/NdkUtils.cs#L44

NdkUtils.UsingClangNDK is true if the NDK version is >= 19.

And indeed, if I install an NDK r18 release, d16-10 is able to build with $(AotAssemblies)=True and $(EnableLLVM)=True! (Yay?)

However, my frankenstein environment was able to build with an NDK >= 19. Why?

Commit accc846 isn't present in d16-9, is present in d16-10, and has an important change:

accc846#diff-d4dcca33896738b8e073a8fde94253a72f9b76cf0923179cb06a94a8db740f41R234

		public async override System.Threading.Tasks.Task RunTaskAsync ()
		{
			// NdkUtil must always be initialized - once per thread
			if (!NdkUtil.Init (LogCodedError, AndroidNdkDirectory)) {
				LogDebugMessage ("Failed to initialize NdkUtil");
				return;
			}

d16-9 has the same method override, but lacks the NdkUtil.Init() invocation.

NdkUtil.Init() is what sets NdkUtil.UsingClangNDK, which is backed by a thread-local value:

https://github.com/xamarin/xamarin-android/blob/0e5e06f716c7e9044e66ed27b75fcd722c546127/src/Xamarin.Android.Build.Tasks/Tasks/NdkUtils.cs#L21-L23

Additionally, RunTaskAsync() is always run on a new thread:

https://github.com/xamarin/xamarin-android-tools/blob/81519fe22ec00320a4f340f4d20d415848a1b16e/src/Microsoft.Android.Build.BaseTasks/AndroidAsyncTask.cs#L28-L41

Consequently, on d16-9, NdkUtil.UsingClangNDK is always false. Thus, it never executes the libs.Add($"-L{toolchainLibDir}") statement which was responsible for the -LC:\Path With Spaces\… entry within ld-flags.

d16-10 does appropriately call NdkUtil.Init(), which ensures that the NdkUtil.usingClangNDK thread-local value is properly set for all threads, which in turn means NdkUtil.UsingClangNDK will be true…if/when the NDK is > r19, which is almost certainly the case.

The reason it never broke before d16-10 is because we never properly set the info needed to know we were on a newer NDK until d16-10!

[queequac]

Is a fix coming with the next update of VS 16.10? I am currently blocked and cannot archive apps... unfortunately switching over to a path without spaces does not work as a workarround for me, it still fails due to the same error.

And still wondering why the path in the error message has a line break and a missing character at the position where the concatenation takes places:

[aot-compiler stderr] C:\short\Android\android-sdk
dk-bundle\toolchains\llvm\prebuilt\windows-x86_64\bin\arm-linux-androideabi-ld.EXE: error: cannot open Files: No such file or directory

[Tommigun1980]

@jonpryor I'm using Visual Studio for Mac 8.10.6 and this is still broken. Did the fix not land in the latest version? If not, when will it be released? It would be nice being able to make release builds again.

Thank you.

[jonathanpeppers]

@Tommigun1980 I believe this fix is only available in VS Windows 16.11, there isn't an equivalent release for VS Mac yet.

Can you try this build and see if it solves your issue, thanks!

https://dl.internalx.com/vsts-devdiv/Xamarin.Android/public/4941337/d16-11/7776c9f1c8fac303c3aa57867825990850be0384/xamarin.android-11.4.0.5.pkg