Switch to the new Mono.Unix NuGet (#5971)

Changes: xamarin/monodroid@76c04cd...1f2ce15

  * xamarin/monodroid@1f2ce1562: [tools/msbuild] only run _GetPrimaryCpuAbi for Fast Dev (#1208)
  * xamarin/monodroid@691310ede: Bump android-sdk-installer to use Mono.Unix (#1207)
  * xamarin/monodroid@48843fcb2: [tools/msbuild] <GetPrimaryCpuAbi/> selects backup RIDs for .NET 6 (#1205)
  * xamarin/monodroid@4af48f54b: [tools/fastdev] Add error checking when writing data to disk. (#1204)
  * xamarin/monodroid@65b7b2dd4: [tools/fastdev] Rework Unix timestamp calculation code in xamarin.find. (#1202)
  * xamarin/monodroid@401f170e9: [build] fix `dotnet tool install` command (#1203)
  * xamarin/monodroid@a879b250b: Bump to 032d840, xamarin/androidtools@355d015 (#1199)
  * xamarin/monodroid@9f9ee378c: [tools/msbuild] Missing translations for XA0135 (#1198)

Changes: dotnet/android-tools@683f375...49936d6

  * dotnet/android-tools@49936d6: Merge pull request #124 from xamarin/update-libzipsharp
  * dotnet/android-tools@ef78dfc: Bump LibZipSharp to 2.0.0-alpha6
  * dotnet/android-tools@e3d708c: [BaseTasks] fix `\`-delimited paths on macOS (#122)
  * dotnet/android-tools@bdcf899: Reference the new Mono.Unix nuget (#123)
  * dotnet/android-tools@90d7621: [BaseTasks] add ABI detection for RIDs (#121)
  * dotnet/android-tools@79e3b97: [JdkInfo] handle invalid XML from /usr/libexec/java_home (#120)
  * dotnet/android-tools@81519fe: Add SECURITY.md (#119)


Xamarin.Android uses the Mono's Mono.Posix assembly on Unix machines
to perform tasks not possible with BCL classes, provided by the
[`Mono.Posix.NETStandard` NuGet][0].  The `Mono.Posix` source has been
extracted into the [mono/mono.posix repo][1], which is used to build
the new [`Mono.Unix` NuGet package][2].

Update the xamarin/xamarin-android repo -- and various dependencies --
to use the `Mono.Unix` package instead of `Mono.Posix.NETStandard`.
This includes the [`Xamarin.LibZipSharp` NuGet][3], as of
dotnet/android-libzipsharp@cf5e33c6.

`Mono.Unix` no longer uses the older `libMonoPosixHelper` dynamic
library, replaced by a new `libMono.Unix` native library.
Unfortunately, this change broke a number of tests since the
`Mono.Unix.dll` assembly was no longer able to find its companion
native shared library.  While the `Mono.Posix.NETStandard` NuGet
package provides the `libMonoPosixHelper` native library, in practice
the *actual* `libMonoPosixHelper` that was used was the "system"
library included with the system mono.

`Mono.Unix`'s new native helper library, however, must be taken from
the NuGet and both the Mono and dotnet runtimes must be told where to
load the library from once a P/Invoke into `Mono.Unix` is encountered
in managed code.  The native library is copied from the NuGet to the
referencing application's output directory and it should be loaded from
there.  This proved to be easy for the "legacy" Mono: a simple
[`dllmap` configuration][4] and everything works as it should.

With `dotnet` however, dllmap doesn't work.  `dotnet` has instead a
number of mechanisms to configure where the native libraries can be
found (5 I think).  Unfortunately, the mechanisms either require that
a main executable of the application calls the APIs on entry (e.g. in
the `Main()` method) or that a JSON configuration file is provided for
the application, telling the runtime where the native libraries reside.
In case of `Xamarin.Android.Build.Tasks` there is no main executable we
can configure, since it works in the MSBuild context, providing tasks
and utilities to build Xamarin.Android apps.  In this instance, `dotnet`
could be persuaded to find the libraries by calling one of the 5 APIs.
The problem with this approach, however, is that this action would have
to be performed at **every** possible entry point to the
`Xamarin.Android.Build.Tasks` assembly, since any of them could be used
as the first one.  While certainly possible, it would be both fragile
and an unnecessary maintenance burden.  Instead, a simpler (albeit a
bit kludgy) solution was chosen: the `src/Xamarin.Android.Build.Tasks`
build process now takes care of generating a fat (multi-architecture)
binary for macOS hosts (including `x86-64` and `arm64` architectures)
using the `lipo` utility, then it copies the resulting binary to the
same directory where `Xamarin.Android.Build.Tasks.dll` and
`Mono.Unix.dll` live.  The Linux shared library is also copied to the
same location.  The `dotnet` runtime is able to find and load shared
libraries that are in the same directory as the assembly that needs
them and everything works as expected.

[0]: https://www.nuget.org/packages/Mono.Posix.NETStandard/5.20.1-preview
[1]: https://github.com/mono/mono.posix
[2]: https://www.nuget.org/packages/Mono.Unix/
[3]: https://www.nuget.org/packages/Xamarin.LibZipSharp
[4]: https://www.mono-project.com/docs/advanced/pinvoke/dllmap/

.external

@@ -1,2 +1,2 @@
-xamarin/monodroid:main@76c04cd15eca7afc269a6d26296e9d2db6f79be2
+xamarin/monodroid:main@1f2ce156245ef1bf63ec8014882d283b3224216b
 mono/mono:2020-02@c633fe923832f0c3db3c4e6aa98e5592bf5a06e7

Directory.Build.props

@@ -26,7 +26,7 @@
 
   <!-- Common <PackageReference/> versions -->
   <PropertyGroup>
-    <LibZipSharpVersion>1.0.24</LibZipSharpVersion>
+    <LibZipSharpVersion>2.0.0-alpha6</LibZipSharpVersion>
     <MicroBuildCoreVersion>0.4.1</MicroBuildCoreVersion>
     <MonoCecilVersion>0.11.2</MonoCecilVersion>
     <NuGetApiPackageVersion>5.4.0</NuGetApiPackageVersion>

build-tools/create-packs/SignList.xml

@@ -1,7 +1,6 @@
 <Project>
   <ItemGroup>
     <!-- Do not sign files that already have a signature -->
-    <Skip Include="libzip*" />
     <Skip Include="libZipSharp*" />
   </ItemGroup>
 

build-tools/debian-metadata/rules

@@ -21,7 +21,7 @@ override_dh_install:
 	rm -f bin/*/lib/xamarin.android/xbuild/Xamarin/Android/opt.exe
 	rm -f bin/*/lib/xamarin.android/xbuild/Xamarin/Android/aapt2.exe
 	rm -f bin/*/lib/xamarin.android/xbuild/Xamarin/Android/libwinpthread-1.dll
-	rm -f bin/*/lib/xamarin.android/xbuild/Xamarin/Android/libzip.dll
+	rm -f bin/*/lib/xamarin.android/xbuild/Xamarin/Android/libZipSharpNative.dll
 
 	dh_install
 

build-tools/installers/create-installers.targets

@@ -193,10 +193,12 @@
     <_MSBuildFiles Include="@(AndroidSupportedTargetJitAbi->'$(MSBuildSrcDir)\lib\interpreter-%(Identity)\libmonosgen-2.0.so')"               ExcludeFromAndroidNETSdk="true" />
     <_MSBuildFiles Include="$(MSBuildSrcDir)\libZipSharp.dll" />
     <_MSBuildFiles Include="@(_LocalizationLanguages->'$(MSBuildSrcDir)\%(Identity)\libZipSharp.resources.dll')" />
-    <_MSBuildFiles Include="$(MSBuildSrcDir)\libZipSharp.dll.config" />
     <_MSBuildFiles Include="$(MSBuildSrcDir)\libZipSharp.pdb" />
-    <_MSBuildFiles Include="$(MSBuildSrcDir)\Mono.Posix.NETStandard.dll" />
-    <_MSBuildFiles Include="$(MSBuildSrcDir)\Mono.Posix.NETStandard.pdb" />
+    <_MSBuildFiles Include="$(MSBuildSrcDir)\Mono.Unix.dll" />
+    <_MSBuildFiles Include="$(MSBuildSrcDir)\Mono.Unix.dll.config" />
+    <_MSBuildFiles Include="$(MSBuildSrcDir)\Mono.Unix.pdb" />
+    <_MSBuildFiles Include="$(MSBuildSrcDir)\libMono.Unix.so" />
+    <_MSBuildFiles Include="$(MSBuildSrcDir)\libMono.Unix.dylib" />
     <_MSBuildFiles Include="$(MSBuildSrcDir)\Mono.Profiler.Log.dll" />
     <_MSBuildFiles Include="$(MSBuildSrcDir)\Mono.Profiler.Log.pdb" />
     <_MSBuildFiles Include="$(MSBuildSrcDir)\logcat-parse.exe"      ExcludeFromAndroidNETSdk="true" />
@@ -304,10 +306,10 @@
     <_MSBuildTargetsSrcFiles Include="$(MSBuildTargetsSrcDir)\Xamarin.Android.DefaultOutputPaths.targets" />
   </ItemGroup>
   <ItemGroup>
-    <_MSBuildFilesWin Include="$(MSBuildSrcDir)\libzip.dll" />
-    <_MSBuildFilesWin Include="$(MSBuildSrcDir)\libzip.pdb" />
-    <_MSBuildFilesWin Include="$(MSBuildSrcDir)\lib64\libzip.dll" />
-    <_MSBuildFilesWin Include="$(MSBuildSrcDir)\lib64\libzip.pdb" />
+    <_MSBuildFilesWin Include="$(MSBuildSrcDir)\libZipSharpNative.dll" />
+    <_MSBuildFilesWin Include="$(MSBuildSrcDir)\libZipSharpNative.pdb" />
+    <_MSBuildFilesWin Include="$(MSBuildSrcDir)\lib64\libZipSharpNative.dll" />
+    <_MSBuildFilesWin Include="$(MSBuildSrcDir)\lib64\libZipSharpNative.pdb" />
     <_MSBuildFilesWin Include="$(MSBuildSrcDir)\proguard\bin\proguard.bat" />
     <_MSBuildFilesWin Include="$(MSBuildSrcDir)\aapt2.exe" />
     <_MSBuildFilesWin Include="$(MSBuildSrcDir)\ndk\aarch64-linux-android-as.exe" />
@@ -369,9 +371,10 @@
     <_MSBuildFilesUnixSign Include="$(MSBuildSrcDir)\lib\host-$(HostOS)\libmonosgen-2.0.$(LibExtension)"          ExcludeFromAndroidNETSdk="true" />
     <_MSBuildFilesUnixSign Include="$(MSBuildSrcDir)\lib\host-$(HostOS)\libxamarin-app.$(LibExtension)"           ExcludeFromAndroidNETSdk="true" />
     <_MSBuildFilesUnixSign Include="$(MSBuildSrcDir)\lib\host-$(HostOS)\libxa-internal-api.$(LibExtension)"       ExcludeFromAndroidNETSdk="true" />
-    <_MSBuildFilesUnixSign Include="$(MSBuildSrcDir)\libzip.$(LibExtension)" />
-    <!-- A second libMonoPosixHelper.dylib is needed for libZipSharp.dll to load -->
-    <_MSBuildFilesUnixSign Include="$(MSBuildSrcDir)\libMonoPosixHelper.$(LibExtension)" />
+    <_MSBuildFilesUnixSign Include="$(MSBuildSrcDir)\libZipSharpNative.$(LibExtension)" />
+    <!-- A second libMono.Unix.{dylib,so} is needed for libZipSharp.dll to load -->
+    <_MSBuildFilesUnixSign Include="$(MSBuildSrcDir)\libMono.Unix.so" />
+    <_MSBuildFilesUnixSign Include="$(MSBuildSrcDir)\libMono.Unix.dylib" />
     <_MSBuildFilesUnix Include="$(MSBuildSrcDir)\proguard\bin\proguard.sh" />
   </ItemGroup>
   <!-- Allow us to exclude mono bundle files for PR builds -->

build-tools/xaprepare/xaprepare/xaprepare.csproj

@@ -52,7 +52,7 @@
     <PackageReference Include="7-Zip.CommandLine" Version="18.1.0" GeneratePathProperty="true" />
     <PackageReference Include="Kajabity.Tools.Java" Version="0.2.6862.30334" />
     <PackageReference Include="Mono.Options" Version="$(MonoOptionsVersion)" />
-    <PackageReference Include="Mono.Posix.NetStandard" Version="1.0.0" />
+    <PackageReference Include="Mono.Unix" Version="7.0.0-alpha8.21302.6" />
     <PackageReference Include="System.Net.Http" Version="4.3.4" />
     <PackageReference Include="Xamarin.LibZipSharp" Version="$(LibZipSharpVersion)" GeneratePathProperty="true" />
     <PackageReference Include="System.Text.Encoding.CodePages" Version="5.0.0" />

src/Xamarin.Android.Build.Tasks/Resources/Mono.Unix.dll.config

@@ -0,0 +1,5 @@
+<?xml version="1.0" encoding="utf-8"?>
+<configuration>
+  <dllmap dll="Mono.Unix" os="linux" cpu="x86-64" wordsize="64" target="libMono.Unix.so" />
+  <dllmap dll="Mono.Unix" os="osx" target="libMono.Unix.dylib" />
+</configuration>

src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/PackagingTest.cs

@@ -451,7 +451,7 @@ public void CheckAapt2WarningsDoNotGenerateErrors ()
 			//https://github.com/xamarin/xamarin-android/issues/3083
 			var proj = new XamarinAndroidApplicationProject () {
 				IsRelease = true,
-				TargetFrameworkVersion = Versions.Oreo_27,
+				TargetFrameworkVersion = Xamarin.ProjectTools.Versions.Oreo_27,
 				UseLatestPlatformSdk = false,
 			};
 			proj.PackageReferences.Add (KnownPackages.XamarinForms_2_3_4_231);

src/Xamarin.Android.Build.Tasks/Xamarin.Android.Build.Tasks.csproj

@@ -366,12 +366,6 @@
     <None Include="startup-xf.aotprofile">
       <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
     </None>
-    <None Include="$(PkgMono_Posix_NETStandard)\runtimes\osx\native\libMonoPosixHelper.dylib">
-      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
-    </None>
-    <None Include="$(PkgMono_Posix_NETStandard)\runtimes\linux-x64\native\libMonoPosixHelper.so">
-      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
-    </None>
   </ItemGroup>
 
   <ItemGroup>

src/Xamarin.Android.Build.Tasks/Xamarin.Android.Build.Tasks.targets

@@ -351,24 +351,57 @@
     <_ExtraPackageTarget Include="$(OutputPath)\libZipSharp.pdb" />
     <_ExtraPackageSource Include="$(PkgXamarin_Build_AsyncTask)\lib\$(TargetFrameworkNETStandard)\Xamarin.Build.AsyncTask.pdb" />
     <_ExtraPackageTarget Include="$(OutputPath)\Xamarin.Build.AsyncTask.pdb" />
-    <_ExtraPackageSource Include="$(PkgMono_Posix_NETStandard)\ref\$(TargetFrameworkNETStandard)\Mono.Posix.NETStandard.pdb" />
-    <_ExtraPackageTarget Include="$(OutputPath)\Mono.Posix.NETStandard.pdb" />
-    <_ExtraPackageSource Include="$(PkgMono_Posix_NETStandard)\ref\$(TargetFrameworkNETStandard)\Mono.Posix.NETStandard.dll" />
-    <_ExtraPackageTarget Include="$(OutputPath)\Mono.Posix.NETStandard.dll" />
+    <_ExtraPackageSource Include="$(PkgMono_Unix)\lib\$(TargetFrameworkNETStandard)\Mono.Unix.pdb" />
+    <_ExtraPackageTarget Include="$(OutputPath)\Mono.Unix.pdb" />
+
+    <_ExtraPackageSource Include="$(MSBuildThisFileDirectory)\Resources\Mono.Unix.dll.config" />
+    <_ExtraPackageTarget Include="$(OutputPath)\Mono.Unix.dll.config" />
+
+    <_ExtraPackageSource Include="$(PkgMono_Unix)\lib\$(TargetFrameworkNETStandard)\Mono.Unix.dll" />
+    <_ExtraPackageTarget Include="$(OutputPath)\Mono.Unix.dll" />
+
+    <_ExtraPackageSource Include="$(PkgMono_Unix)\runtimes\linux-x64\native\libMono.Unix.so" />
+    <_ExtraPackageTarget Include="$(OutputPath)\libMono.Unix.so" />
+  </ItemGroup>
+
+  <ItemGroup>
+    <_MonoUnixDylib Include="$(PkgMono_Unix)\runtimes\osx-arm64\native\libMono.Unix.dylib" Condition=" '$(HostOS)' == 'Darwin' " />
+    <_MonoUnixDylib Include="$(PkgMono_Unix)\runtimes\osx-x64\native\libMono.Unix.dylib" />
   </ItemGroup>
 
+  <PropertyGroup>
+    <_MonoUnixOutputDylib>$(OutputPath)\libMono.Unix.dylib</_MonoUnixOutputDylib>
+  </PropertyGroup>
+
+  <!--
+      On operating systems other than macOS we just need to copy a single .dylib, so that pack creation works
+  -->
+  <Target Name="_MakeMonoUnixFatBinariesOSX"
+      Inputs="@(_MonoUnixDylib)"
+      Outputs="$(_MonoUnixOutputDylib)">
+    <Exec
+        Command="lipo @(_MonoUnixDylib->'&quot;%(FullPath)&quot;', ' ') -create -output &quot;$(_MonoUnixOutputDylib)&quot;"
+        Condition=" '$(HostOS)' == 'Darwin' "
+    />
+    <Copy
+        SourceFiles="@(_MonoUnixDylib)"
+        DestinationFolder="$(OutputPath)"
+    />
+  </Target>
+
   <!-- Copy package ref symbols for our installers. '$(Pkg*)' props are set during NuGet restore when GeneratePathProperty="true" -->
   <Target Name="_CopyExtraPackageContent"
       Inputs="@(_ExtraPackageSource)"
-      Outputs="@(_ExtraPackageTarget)" >
+      Outputs="@(_ExtraPackageTarget)"
+      DependsOnTargets="_MakeMonoUnixFatBinariesOSX">
     <Copy
         SourceFiles="@(_ExtraPackageSource)"
         DestinationFolder="$(OutputPath)"
         SkipUnchangedFiles="True"
     />
   </Target>
 
-  <Target Name="_Remove32BitLinux" Condition="Exists ('$(OutputPath)lib32\libzip.so')" AfterTargets="CopyFilesToOutputDirectory">
-    <Delete Files="$(OutputPath)lib32\libzip.so" />
+  <Target Name="_Remove32BitLinux" Condition="Exists ('$(OutputPath)lib32\libZipSharpNative.so')" AfterTargets="CopyFilesToOutputDirectory">
+    <Delete Files="$(OutputPath)lib32\libZipSharpNative.so" />
   </Target>
 </Project>

src/Xamarin.Android.Build.Tasks/Xamarin.Android.Common.targets

@@ -91,6 +91,7 @@ Copyright (C) 2011-2012 Xamarin. All rights reserved.
 <UsingTask TaskName="Xamarin.Android.Tasks.PrepareAbiItems" AssemblyFile="Xamarin.Android.Build.Tasks.dll" />
 <UsingTask TaskName="Xamarin.Android.Tasks.WriteLockFile" AssemblyFile="Xamarin.Android.Build.Tasks.dll" />
 <UsingTask TaskName="GenerateCompressedAssembliesNativeSourceFiles" AssemblyFile="Xamarin.Android.Build.Tasks.dll" />
+
 <!--
 *******************************************
   Extensibility hook that allows VS to
@@ -102,7 +103,6 @@ Copyright (C) 2011-2012 Xamarin. All rights reserved.
         Condition="Exists('$(MSBuildThisFileDirectory)$(MSBuildThisFileName).Before.targets')"/>
 
 
-
 <!--
 *******************************************
           Code Analysis Setup

tests/MSBuildDeviceIntegration/Tests/InstantRunTest.cs

@@ -297,6 +297,10 @@ public void InstantRunNativeLibrary ([Values ("dx", "d8")] string dexTool)
 					nativeLib,
 				},
 			};
+			if (Builder.UseDotNet) {
+				//NOTE: in .NET 6 by default an x86_64 emulator would fall back to x86 if we don't set this.
+				proj.SetAndroidSupportedAbis (DeviceAbi);
+			}
 			proj.SetDefaultTargetDevice ();
 			using (var b = CreateApkBuilder (Path.Combine ("temp", TestName))) {
 				Assert.IsTrue (b.Install (proj), "install should have succeeded. 0");