[Xamarin.Android.Build.Tasks] trailing / causes missing classes.dex (#…

…4505)

Fixes: #4478
Fixes: #4486

We with the release of VS for Mac 8.5 going stable have been getting
some reports of:

	error ADB0010:  Deployment failed
	Mono.AndroidTools.InstallFailedException: Failure [INSTALL_FAILED_INVALID_APK: Package couldn't be installed in /data/app/com.contoso.androidapp1-RTZZFPyLkRI7Bk7VDgGkDg==: Package /data/app/com.contoso.androidapp1-RTZZFPyLkRI7Bk7VDgGkDg==/base.apk code is missing]
	    at Mono.AndroidTools.Internal.AdbOutputParsing.CheckInstallSuccess (System.String output, System.String packageName) [0x00159] in <65710797f19f43cc9d8f9e05353b9615>:0
	    at Mono.AndroidTools.AndroidDevice+<>c__DisplayClass95_0.<InstallPackage>b__0 (System.Threading.Tasks.Task`1[TResult] t) [0x0001c] in <65710797f19f43cc9d8f9e05353b9615>:0
	    at System.Threading.Tasks.ContinuationTaskFromResultTask`1[TAntecedentResult].InnerInvoke () [0x00024] in <f9d1b832704f410aa8ec771f4fe80552>:0
	    at System.Threading.Tasks.Task.Execute () [0x00000] in <f9d1b832704f410aa8ec771f4fe80552>:0
	    --- End of stack trace from previous location where exception was thrown ---
	    at AndroidDeviceExtensions.PushAndInstallPackage (Mono.AndroidTools.AndroidDevice device, System.String apkFile, System.String packageName, System.Boolean reinstall, Mono.AndroidTools.Adb.AdbProgressReporter notifyProgress, System.Threading.CancellationToken token) [0x001d3] in <81f66a65e9434daf9d06602cf6ac0bf9>:0
	    at AndroidDeviceExtensions.PushAndInstallPackage (Mono.AndroidTools.AndroidDevice device, System.String apkFile, System.String packageName, System.Boolean reinstall, Mono.AndroidTools.Adb.AdbProgressReporter notifyProgress, System.Threading.CancellationToken token) [0x00402] in <81f66a65e9434daf9d06602cf6ac0bf9>:0
	    at Xamarin.AndroidTools.AndroidDeploySession.InstallPackage () [0x003be] in <81f66a65e9434daf9d06602cf6ac0bf9>:0
	    at Xamarin.AndroidTools.AndroidDeploySession.RunAsync (System.Threading.CancellationToken token) [0x003f8] in <81f66a65e9434daf9d06602cf6ac0bf9>:0
	    at Xamarin.AndroidTools.AndroidDeploySession.RunLoggedAsync (System.Threading.CancellationToken token) [0x00077] in <81f66a65e9434daf9d06602cf6ac0bf9>:0  [/Users/macuser/Projects/AndroidApp1/AndroidApp1/AndroidApp1.csproj]

The error means the `.apk` is missing `classes.dex`.

It seems to affect apps where:

  * `$(AndroidDexTool)=dx`, this would need to be set explicitly or
    it would default to `d8`.
  * The issue appears to occur only on macOS.

@brendanzagaeski [was able to reproduce the issue][0]:

Using [`4478TestCase.zip`][1], there is an `obj` directory sitting
next to the `.sln` file:

  * `obj`
  * `AndroidApp1`
  * `AndroidApp1/AndroidApp1.csproj`
  * `AndroidApp1/obj`
  * `AndroidApp1.sln`

Note that the actual `$(BaseIntermediateOutputPath)` would be
`AndroidApp1/obj`.  Brendan just moved an `obj` directory next to the
`.sln` file while trying to reproduce the issue.

However, it appears the presence of this phantom `obj` triggers the
issue!  You can rename it to `obj2` to *solve* the issue.  It is a
directory with sub-directories, but no files.

If you look at the build log, the main difference is:

	CompileToDalvik
	    Parameters
	-       ClassesOutputDirectory = obj/Release/android/bin/classes/
	+       ClassesOutputDirectory = obj/Release/android/bin/classes

The presence of the trailing `/` breaks this code:

	cmd.AppendSwitchIfNotNull ("--output ", Path.GetDirectoryName (ClassesOutputDirectory));

`classes.dex` is output into `obj/Release/android/bin/classes` instead
of `obj/Release/android/bin` and doesn't make it into the final `.apk`.

Tracking back to what creates the input value of
`ClassesOutputDirectory`, it is the MSBuild property
`$(_AndroidIntermediateJavaClassDirectory)`, which is:

	<_AndroidIntermediateJavaClassDirectory>$(IntermediateOutputPath)android\bin\classes\</_AndroidIntermediateJavaClassDirectory>

But then comparing broken vs working logs:

	Broken:
	    _InitialBaseIntermediateOutputPath = obj/
	    BaseIntermediateOutputPath = obj/
	    IntermediateOutputPath = obj/Release/
	    _AndroidIntermediateJavaClassDirectory = obj/Release/android/bin/classes/
	Working:
	    _InitialBaseIntermediateOutputPath = obj\
	    BaseIntermediateOutputPath = obj\
	    IntermediateOutputPath = obj\Release\
	    _AndroidIntermediateJavaClassDirectory = obj\Release\android\bin\classes\

It appears the existence of this random `obj` directory influences
MSBuild's path combining behavior?

The trailing `\` works, but `/` does not, due to `TrimEnd`:

	<CompileToDalvik
	    ...
	    ClassesOutputDirectory="$(_AndroidIntermediateJavaClassDirectory.TrimEnd('\'))"

The breakage was introduced in 185c529, which added a trailing `\` to
the directory passed to `<CompileToDalvik/>`.  We are unsure what is
causing the strange MSBuild behavior.

However, after reviewing `<CompileToDalvik/>`, I think we should just
remove the `Path.GetDirectoryName()` call.  We should just pass in the
directory name we want to use from MSBuild targets.

I could add a test for this scenario by setting a global property:

	/p:_AndroidIntermediateJavaClassDirectory=obj/Debug/android/bin/classes/

Setting a global property could test the trailing `/` character
regardless of MSBuild behavior.

I also cleaned things up a bit by create a new
`$(_AndroidIntermediateDexOutputDirectory)` property.

[0]: #4478 (comment)
[1]: https://github.com/xamarin/xamarin-android/files/4419061/4478TestCase.zip

Documentation/release-notes/4505.md

@@ -0,0 +1,8 @@
+#### Application and library build and deployment
+
+  * [GitHub 4478](https://github.com/xamarin/xamarin-android/issues/4478),
+    [GitHub 4486](https://github.com/xamarin/xamarin-android/issues/4486):
+    On macOS, starting in Xamarin.Android 10.2, *\[INSTALL_FAILED_INVALID_APK:
+    ...  base.apk code is missing\]* and *Module 'base' has no dex files*
+    prevented deploying successfully for certain projects configured to use the
+    DX DEX compiler.

src/Xamarin.Android.Build.Tasks/Tasks/CompileToDalvik.cs

@@ -114,7 +114,7 @@ protected override string GenerateCommandLineCommands ()
 					cmd.AppendSwitchIfNotNull ("--main-dex-list=", MultiDexMainDexListFile);
 				}
 			}
-			cmd.AppendSwitchIfNotNull ("--output ", Path.GetDirectoryName (ClassesOutputDirectory));
+			cmd.AppendSwitchIfNotNull ("--output ", ClassesOutputDirectory);
 
 			using (var sw = new StreamWriter (path: inputListFile, append: false,
 					encoding: MonoAndroidHelper.UTF8withoutBOM)) {

src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/PackagingTest.cs

@@ -676,5 +676,23 @@ public void CheckTheCorrectRuntimeAssemblyIsUsedFromNuget ()
 				Assert.IsTrue (xab.LastBuildOutput.ContainsText (expected), $"Build should be using {expected}");
 			}
 		}
+
+		[Test]
+		public void MissingDexFile ()
+		{
+			//NOTE: The trailing / was breaking <CompileToDalvik /> in 16.5
+			var parameters = new [] { "_AndroidIntermediateJavaClassDirectory=obj/Debug/android/bin/classes/" };
+			var proj = new XamarinAndroidApplicationProject {
+				DexTool = "dx"
+			};
+			using (var b = CreateApkBuilder ()) {
+				Assert.IsTrue (b.Build (proj, parameters: parameters), "Build should have succeeded.");
+				var apk = Path.Combine (Root, b.ProjectDirectory,
+					proj.IntermediateOutputPath, "android", "bin", "UnnamedProject.UnnamedProject.apk");
+				using (var zip = ZipHelper.OpenZip (apk)) {
+					Assert.IsTrue (zip.ContainsEntry ("classes.dex"), "Apk should contain classes.dex");
+				}
+			}
+		}
 	}
 }

src/Xamarin.Android.Build.Tasks/Xamarin.Android.Common.targets

@@ -1124,6 +1124,7 @@ because xbuild doesn't support framework reference assemblies.
 	<MonoAndroidCodeBehindDir>$(IntermediateOutputPath)generated</MonoAndroidCodeBehindDir>
 	<MonoAndroidIntermediateResourceCache>$(IntermediateOutputPath)resourcecache</MonoAndroidIntermediateResourceCache>
 	<_AndroidIntermediateJavaSourceDirectory>$(IntermediateOutputPath)android\src\</_AndroidIntermediateJavaSourceDirectory>
+	<_AndroidIntermediateDexOutputDirectory>$(IntermediateOutputPath)android\bin\</_AndroidIntermediateDexOutputDirectory>
 	<_AndroidIntermediateJavaClassDirectory>$(IntermediateOutputPath)android\bin\classes\</_AndroidIntermediateJavaClassDirectory>
 	<_AndroidIntermediateClassesZip>$(IntermediateOutputPath)android\bin\classes.zip</_AndroidIntermediateClassesZip>
 	<_AndroidAotBinDirectory>$(IntermediateOutputPath)aot</_AndroidAotBinDirectory>
@@ -2357,8 +2358,7 @@ because xbuild doesn't support framework reference assemblies.
 <Target Name="_CompileDex"
 		DependsOnTargets="$(_BeforeCompileDex);$(_CompileDexDependsOn)">
 	<ItemGroup>
-		<_DexFile Include="$(IntermediateOutputPath)android\bin\dex\*.dex" />
-		<_DexFile Include="$(IntermediateOutputPath)android\bin\*.dex" />
+		<_DexFile Include="$(_AndroidIntermediateDexOutputDirectory)*.dex" />
 	</ItemGroup>
 </Target>
 

src/Xamarin.Android.Build.Tasks/Xamarin.Android.D8.targets

@@ -36,7 +36,7 @@ Copyright (C) 2018 Xamarin. All rights reserved.
 
     <!-- remove existing dex files that may be previous multidex outputs. -->
     <ItemGroup>
-      <_DexesToDelete Include="$(IntermediateOutputPath)android\bin\*.dex" />
+      <_DexesToDelete Include="$(_AndroidIntermediateDexOutputDirectory)*.dex" />
     </ItemGroup>
     <Delete Files="@(_DexesToDelete)" />
 
@@ -52,7 +52,7 @@ Copyright (C) 2018 Xamarin. All rights reserved.
         JavaOptions="$(JavaOptions)"
         JarPath="$(AndroidR8JarPath)"
         AndroidManifestFile="$(IntermediateOutputPath)android\AndroidManifest.xml"
-        OutputDirectory="$(IntermediateOutputPath)android\bin"
+        OutputDirectory="$(_AndroidIntermediateDexOutputDirectory)"
         Debug="$(AndroidIncludeDebugSymbols)"
         JavaPlatformJarPath="$(JavaPlatformJarPath)"
         ClassesZip="$(_AndroidIntermediateClassesZip)"
@@ -79,7 +79,7 @@ Copyright (C) 2018 Xamarin. All rights reserved.
         JavaOptions="$(JavaOptions)"
         JarPath="$(AndroidR8JarPath)"
         AndroidManifestFile="$(IntermediateOutputPath)android\AndroidManifest.xml"
-        OutputDirectory="$(IntermediateOutputPath)android\bin"
+        OutputDirectory="$(_AndroidIntermediateDexOutputDirectory)"
         Debug="$(AndroidIncludeDebugSymbols)"
         JavaPlatformJarPath="$(JavaPlatformJarPath)"
         ClassesZip="$(_AndroidIntermediateClassesZip)"
@@ -92,7 +92,7 @@ Copyright (C) 2018 Xamarin. All rights reserved.
 
     <Touch Files="$(_AndroidStampDirectory)_CompileToDalvik.stamp" AlwaysCreate="true" />
     <ItemGroup>
-      <FileWrites Include="$(IntermediateOutputPath)android\bin\*.dex" />
+      <FileWrites Include="$(_AndroidIntermediateDexOutputDirectory)*.dex" />
     </ItemGroup>
 
   </Target>

src/Xamarin.Android.Build.Tasks/Xamarin.Android.DX.targets

@@ -71,7 +71,7 @@ Copyright (C) 2018 Xamarin. All rights reserved.
 
     <!-- remove existing dex files that may be previous multidex outputs. -->
     <ItemGroup>
-      <_DexesToDelete Include="$(IntermediateOutputPath)android\bin\*.dex" />
+      <_DexesToDelete Include="$(_AndroidIntermediateDexOutputDirectory)*.dex" />
     </ItemGroup>
     <Delete Files="@(_DexesToDelete)" />
 
@@ -82,7 +82,7 @@ Copyright (C) 2018 Xamarin. All rights reserved.
         JavaToolPath="$(JavaToolPath)"
         JavaMaximumHeapSize="$(JavaMaximumHeapSize)"
         JavaOptions="$(JavaOptions)"
-        ClassesOutputDirectory="$(_AndroidIntermediateJavaClassDirectory.TrimEnd('\'))"
+        ClassesOutputDirectory="$(_AndroidIntermediateDexOutputDirectory)"
         ClassesZip="$(_AndroidIntermediateClassesZip)"
         ToolPath="$(DxToolPath)"
         ToolExe="$(DxToolExe)"
@@ -94,7 +94,7 @@ Copyright (C) 2018 Xamarin. All rights reserved.
     />
     <Touch Files="$(_AndroidStampDirectory)_CompileToDalvik.stamp" AlwaysCreate="true" />
     <ItemGroup>
-      <FileWrites Include="$(IntermediateOutputPath)android\bin\*.dex" />
+      <FileWrites Include="$(_AndroidIntermediateDexOutputDirectory)*.dex" />
     </ItemGroup>
 
   </Target>