[xaprepare] Generate SourceLink.json

Fixes: #2614

Context: https://developercommunity.visualstudio.com/t/JNINativeWrappergcs-file-Not-found/10020777
Context: https://www.nuget.org/packages/sourcelink/
Context: https://github.com/dotnet/designs/blob/main/accepted/2020/diagnostics/source-link.md#source-link-file-specification
Context: https://docs.microsoft.com/en-us/dotnet/standard/library-guidance/sourcelink
Context: https://www.hanselman.com/blog/exploring-net-cores-sourcelink-stepping-into-the-source-code-of-nuget-packages-you-dont-own

[SourceLink][0] is a set of tooling which allows debuggers to obtain
source code, allowing developers to "step into" assemblies for which
they don't have local source code to.

It operates by adding a JSON document to Portable PDB files, which
allows mapping paths within the Portable PDB file to a URL, which the
debugger can optionally download.

The JSON document contains a `documents` key, which contains a nested
dictionary which maps local filesystem path prefixes to URLs, e.g.

	{
	  "documents": {
	    "…/xamarin/xamarin-android/external/Java.Interop/*":            "https://raw.githubusercontent.com/xamarin/java.interop/a5756ca8b8764c24cc169e40a473f79302b40ca9/*",
	    "…/xamarin/xamarin-android/external/xamarin-android-tools/*":   "https://raw.githubusercontent.com/xamarin/xamarin-android-tools/9c641b3e08e56db37467a64a2c5de2c7f7ddb3ef/*",
	    "…/xamarin/xamarin-android/*":                                  "https://raw.githubusercontent.com/xamarin/xamarin-android/903ba37ce70d2840983774e1d6fb55f8002561e2/*"
	  }
	}

This JSON document can be provided to the C# compiler by setting the
`$(SourceLink)` MSBuild property to the path of the JSON document.

The [sourcelink][1] dotnet global tool can be used to check `.pdb`
file contents:

	% dotnet tool install --global SourceLink --version 3.1.1

To view the *local paths* within a `.pdb` file, use
`sourcelink print-documents`:

	% $HOME/.dotnet/tools/sourcelink print-documents bin/Debug/lib/xamarin.android/xbuild-frameworks/Microsoft.Android/33/Mono.Android.pdb
	9c5ad5a588d5a49deb98ccd63c7b0c4699c8f39d6ee1df4a8f4ae555be827e06 sha256 csharp …/xamarin-android/external/Java.Interop/src/Java.Interop.Tools.TypeNameMappings/Java.Interop.Tools.TypeNameMappings/JavaNativeTypeManager.cs
	…

To view the *remote URLs* for those paths, use `sourcelink print-urls`:

	% $HOME/.dotnet/tools/sourcelink print-urls bin/Debug/lib/xamarin.android/xbuild-frameworks/Microsoft.Android/33/Mono.Android.pdb
	9c5ad5a588d5a49deb98ccd63c7b0c4699c8f39d6ee1df4a8f4ae555be827e06 sha256 csharp …/xamarin-android/external/Java.Interop/src/Java.Interop.Tools.TypeNameMappings/Java.Interop.Tools.TypeNameMappings/JavaNativeTypeManager.cs
	https://raw.githubusercontent.com/xamarin/java.interop/a5756ca8b8764c24cc169e40a473f79302b40ca9/src/Java.Interop.Tools.TypeNameMappings/Java.Interop.Tools.TypeNameMappings/JavaNativeTypeManager.cs
	…

To view the contents of the JSON document provided to the compiler,
use `sourcelink print-json`:

	%  $HOME/.dotnet/tools/sourcelink print-json bin/Debug/lib/xamarin.android/xbuild-frameworks/Microsoft.Android/33/Mono.Android.pdb
	{
	  "documents": {
	    …
	    "…/xamarin/xamarin-android/external/Java.Interop/*": "https://raw.githubusercontent.com/xamarin/java.interop/a5756ca8b8764c24cc169e40a473f79302b40ca9/*",
	    "…/xamarin/xamarin-android/external/xamarin-android-tools/*": "https://raw.githubusercontent.com/xamarin/xamarin-android-tools/9c641b3e08e56db37467a64a2c5de2c7f7ddb3ef/*",
	    "…/xamarin/xamarin-android/*": "https://raw.githubusercontent.com/xamarin/xamarin-android/903ba37ce70d2840983774e1d6fb55f8002561e2/*"
	  }
	}

Finally, to verify that all URLs from `sourcelink print-urls` are
valid, use `sourcelink test`:

	% $HOME/.dotnet/tools/sourcelink test bin/Debug/lib/xamarin.android/xbuild-frameworks/Microsoft.Android/33/Mono.Android.pdb
	nodename nor servname provided, or not known
	# 🤔

	% $HOME/.dotnet/tools/sourcelink test bin/Debug/lib/packs/Microsoft.Android.Sdk.Darwin/33.0.0/tools/Xamarin.Android.Build.Tasks.pdb
	83 Documents with errors:
	962d50c8d14a454ff562f843051e542c22c69b63b489f1d2922d092aba3f578b sha256 csharp …/xamarin-android/src/Xamarin.Android.Build.Tasks/obj/Release/Profile.g.cs
	https://raw.githubusercontent.com/xamarin/xamarin-android/4a94e294fbf8dc85a319386d4aab3526718769c0/src/Xamarin.Android.Build.Tasks/obj/Release/Profile.g.cs
	error: url failed NotFound: Not Found
	…
	sourcelink test failed

Note: this can take awhile.

Some of these errors are to be expected, as they reference generated
files which are not available via `raw.githubusercontent.com`.

Other errors suggest that we need to "extend" this system, e.g.

	0b1fe6595fff8c1af58a6f97a6bc79227300ae7941bd113ad092c36b418c14b9 sha256 csharp …/xamarin-android/external/mono/sdks/out/android-sources/external/linker/src/tuner/Mono.Tuner/PrintStatus.cs
	https://raw.githubusercontent.com/xamarin/xamarin-android/4a94e294fbf8dc85a319386d4aab3526718769c0/external/mono/sdks/out/android-sources/external/linker/src/tuner/Mono.Tuner/PrintStatus.cs
	error: url failed NotFound: Not Found

A path of `external/…/external` suggests that we need more entries.

To make all this work, update `xaprepare` to create a
`bin/Build$(Configuration)/SourceLink.json` file, and update
`Configuration.props` so that `$(SourceLink)` is set to the generated
`SourceLink.json` file, if it exists.  `SourceLink.json` is generated
by reading `.gitmodules` to determine all of the git submodules, and
using that information to construct the "remote" URL prefix.

TODO:

`SourceLink.json` assumes that all remote URLs are GitHub URLs.
This may not be ideal, but works for most of our assemblies.

What should be done about *generated* files?

Is there a good way to deal with xamarin-android referencing mono
files which reference linker files?

[0]: https://github.com/dotnet/sourcelink
[1]: https://www.nuget.org/packages/sourcelink

Configuration.props

@@ -202,6 +202,9 @@
     <NdkBuildPath Condition=" '$(NdkBuildPath)' == '' And '$(HostOS)' == 'Windows' ">$(AndroidNdkDirectory)\ndk-build.cmd</NdkBuildPath>
     <BundleToolJarPath Condition=" '$(BundleToolJarPath)' == '' ">$(MicrosoftAndroidSdkOutDir)bundletool.jar</BundleToolJarPath>
   </PropertyGroup>
+  <PropertyGroup>
+    <SourceLink Condition=" Exists('$(MSBuildThisFileDirectory)bin/Build$(Configuration)/SourceLink.json') ">$(MSBuildThisFileDirectory)bin/Build$(Configuration)/SourceLink.json</SourceLink>
+  </PropertyGroup>
   <!--
     "Fixup" $(AndroidSupportedHostJitAbis) so that Condition attributes elsewhere
     can use `:ABI-NAME:`, to avoid substring mismatches.

build-tools/xaprepare/xaprepare/Application/GeneratedFile.cs

@@ -36,4 +36,16 @@ protected void EnsureOutputDir ()
 			Utilities.CreateDirectory (Path.GetDirectoryName (OutputPath));
 		}
 	}
+
+	sealed class SkipGeneratedFile : GeneratedFile {
+
+		public SkipGeneratedFile ()
+			: base (Path.Combine (BuildPaths.XAPrepareSourceDir, "shall-not-exist.txt"))
+		{
+		}
+
+		public override void Generate (Context context)
+		{
+		}
+	}
 }

build-tools/xaprepare/xaprepare/Application/GeneratedSourceLinkJsonFile.cs

@@ -0,0 +1,54 @@
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Text;
+
+namespace Xamarin.Android.Prepare
+{
+	class GeneratedSourceLinkJsonFile : GeneratedFile
+	{
+		IEnumerable<GitSubmoduleInfo> submodules;
+		string xaCommit;
+
+		public GeneratedSourceLinkJsonFile (IEnumerable<GitSubmoduleInfo> submodules, string xaCommit, string outputPath)
+			: base (outputPath)
+		{
+			this.submodules = submodules ?? throw new ArgumentNullException (nameof (submodules));
+			this.xaCommit   = !string.IsNullOrEmpty (xaCommit) ? xaCommit : throw new ArgumentNullException (nameof (xaCommit));
+		}
+
+		public override void Generate (Context context)
+		{
+			var json    = new StringBuilder ();
+			json.AppendLine ("{");
+			json.AppendLine ("  \"documents\": {");
+
+			foreach (var submodule in submodules.OrderBy (s => s.Name)) {
+				var localPath   = Path.Combine (BuildPaths.XamarinAndroidSourceRoot, submodule.LocalPath);
+
+				var contentUri  = new UriBuilder (submodule.RepositoryUrl);
+				contentUri.Host = "raw.githubusercontent.com";
+				contentUri.Path += $"/{submodule.CommitHash}";
+
+				json.AppendLine ($"    \"{localPath}/*\": \"{contentUri.Uri}/*\",");
+			}
+			json.AppendLine ($"    \"{BuildPaths.XamarinAndroidSourceRoot}/*\": \"https://raw.githubusercontent.com/xamarin/xamarin-android/{xaCommit}/*\"");
+			json.AppendLine ("  }");
+			json.AppendLine ("}");
+
+			EnsureOutputDir ();
+			string outputData = json.ToString ();
+			File.WriteAllText (OutputPath, outputData, Utilities.UTF8NoBOM);
+
+			if (!EchoOutput)
+				return;
+
+			Log.DebugLine ();
+			Log.DebugLine ("--------------------------------------------");
+			Log.DebugLine (outputData);
+			Log.DebugLine ("--------------------------------------------");
+			Log.DebugLine ();
+		}
+	}
+}

build-tools/xaprepare/xaprepare/Steps/Step_GenerateCGManifest.cs

@@ -149,6 +149,7 @@ sealed class GitSubmoduleInfo : CGManifestEntry {
 
 		public  string      RepositoryUrl   {get; private set;} = String.Empty;
 		public  string      CommitHash      {get; private set;} = String.Empty;
+		public  string      LocalPath       {get; private set;} = String.Empty;
 
 		GitSubmoduleInfo ()
 		{
@@ -225,6 +226,7 @@ GitSubmoduleInfo CreateSubmoduleInfo (string url, string path)
 					break;
 				}
 				return new GitSubmoduleInfo () {
+					LocalPath       = path,
 					RepositoryUrl   = url,
 					CommitHash      = hash ?? String.Empty,
 				};

build-tools/xaprepare/xaprepare/Steps/Step_GenerateFiles.cs

@@ -10,16 +10,25 @@ partial class Step_GenerateFiles : Step
 		bool atBuildStart;
 		bool onlyRequired;
 
+		IEnumerable<GitSubmoduleInfo>?  gitSubmodules;
+		string?                         xaCommit;
+
+
 		public Step_GenerateFiles (bool atBuildStart, bool onlyRequired = false)
 			: base ("Generating files required by the build")
 		{
 			this.atBuildStart = atBuildStart;
 			this.onlyRequired = onlyRequired;
 		}
 
-#pragma warning disable CS1998
 		protected override async Task<bool> Execute (Context context)
 		{
+			var git                 = new GitRunner (context);
+			xaCommit                = git.GetTopCommitHash (workingDirectory: BuildPaths.XamarinAndroidSourceRoot, shortHash: false);
+			var gitSubmoduleInfo    = await git.ConfigList (new[]{"--blob", "HEAD:.gitmodules"});
+			var gitSubmoduleStatus  = await git.SubmoduleStatus ();
+			gitSubmodules           = GitSubmoduleInfo.GetGitSubmodules (gitSubmoduleInfo, gitSubmoduleStatus);
+
 			List<GeneratedFile>? filesToGenerate = GetFilesToGenerate (context);
 			if (filesToGenerate != null && filesToGenerate.Count > 0) {
 				foreach (GeneratedFile gf in filesToGenerate) {
@@ -39,20 +48,21 @@ protected override async Task<bool> Execute (Context context)
 
 			return true;
 		}
-#pragma warning restore CS1998
 
 		List<GeneratedFile>? GetFilesToGenerate (Context context)
 		{
 			if (atBuildStart) {
 				if (onlyRequired) {
 					return new List<GeneratedFile> {
+						Get_SourceLink_Json (context),
 						Get_MonoGitHash_props (context),
 						Get_Configuration_Generated_Props (context),
 						Get_Cmake_XA_Build_Configuration (context),
 						new GeneratedMonodroidCmakeFiles (Configurables.Paths.BuildBinDir),
 					};
 				} else {
 					return new List <GeneratedFile> {
+						Get_SourceLink_Json (context),
 						Get_Configuration_OperatingSystem_props (context),
 						Get_Configuration_Generated_Props (context),
 						Get_Cmake_XA_Build_Configuration (context),
@@ -254,5 +264,17 @@ public GeneratedFile Get_Omnisharp_Json (Context context)
 				Path.Combine (BuildPaths.XamarinAndroidSourceRoot, OutputFileName)
 			);
 		}
+
+		public GeneratedFile Get_SourceLink_Json (Context context)
+		{
+			if (gitSubmodules == null || xaCommit == null) {
+				return new SkipGeneratedFile ();
+			}
+			return new GeneratedSourceLinkJsonFile (
+					gitSubmodules!,
+					xaCommit!,
+					Path.Combine (Configurables.Paths.BuildBinDir, "SourceLink.json")
+			);
+		}
 	}
 }