Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[xaprepare] Generate SourceLink.json #7298

Merged
merged 1 commit into from
Aug 23, 2022

Conversation

jonpryor
Copy link
Member

Fixes: #2614

Context: https://developercommunity.visualstudio.com/t/JNINativeWrappergcs-file-Not-found/10020777
Context: https://www.nuget.org/packages/sourcelink/
Context: https://github.com/dotnet/designs/blob/main/accepted/2020/diagnostics/source-link.md#source-link-file-specification
Context: https://docs.microsoft.com/en-us/dotnet/standard/library-guidance/sourcelink
Context: https://www.hanselman.com/blog/exploring-net-cores-sourcelink-stepping-into-the-source-code-of-nuget-packages-you-dont-own

SourceLink is a set of tooling which allows debuggers to obtain
source code, allowing developers to "step into" assemblies for which
they don't have local source code to.

It operates by adding a JSON document to Portable PDB files, which
allows mapping paths within the Portable PDB file to a URL, which the
debugger can optionally download.

The JSON document contains a documents key, which contains a nested
dictionary which maps local filesystem path prefixes to URLs, e.g.

{
  "documents": {
    "…/xamarin/xamarin-android/external/Java.Interop/*":            "https://raw.githubusercontent.com/xamarin/java.interop/a5756ca8b8764c24cc169e40a473f79302b40ca9/*",
    "…/xamarin/xamarin-android/external/xamarin-android-tools/*":   "https://raw.githubusercontent.com/xamarin/xamarin-android-tools/9c641b3e08e56db37467a64a2c5de2c7f7ddb3ef/*",
    "…/xamarin/xamarin-android/*":                                  "https://raw.githubusercontent.com/xamarin/xamarin-android/903ba37ce70d2840983774e1d6fb55f8002561e2/*"
  }
}

This JSON document can be provided to the C# compiler by setting the
$(SourceLink) MSBuild property to the path of the JSON document.

The sourcelink dotnet global tool can be used to check .pdb
file contents:

% dotnet tool install --global SourceLink --version 3.1.1

To view the local paths within a .pdb file, use
sourcelink print-documents:

% $HOME/.dotnet/tools/sourcelink print-documents bin/Debug/lib/xamarin.android/xbuild-frameworks/Microsoft.Android/33/Mono.Android.pdb
9c5ad5a588d5a49deb98ccd63c7b0c4699c8f39d6ee1df4a8f4ae555be827e06 sha256 csharp …/xamarin-android/external/Java.Interop/src/Java.Interop.Tools.TypeNameMappings/Java.Interop.Tools.TypeNameMappings/JavaNativeTypeManager.cs
…

To view the remote URLs for those paths, use sourcelink print-urls:

% $HOME/.dotnet/tools/sourcelink print-urls bin/Debug/lib/xamarin.android/xbuild-frameworks/Microsoft.Android/33/Mono.Android.pdb
9c5ad5a588d5a49deb98ccd63c7b0c4699c8f39d6ee1df4a8f4ae555be827e06 sha256 csharp …/xamarin-android/external/Java.Interop/src/Java.Interop.Tools.TypeNameMappings/Java.Interop.Tools.TypeNameMappings/JavaNativeTypeManager.cs
https://raw.githubusercontent.com/xamarin/java.interop/a5756ca8b8764c24cc169e40a473f79302b40ca9/src/Java.Interop.Tools.TypeNameMappings/Java.Interop.Tools.TypeNameMappings/JavaNativeTypeManager.cs
…

To view the contents of the JSON document provided to the compiler,
use sourcelink print-json:

%  $HOME/.dotnet/tools/sourcelink print-json bin/Debug/lib/xamarin.android/xbuild-frameworks/Microsoft.Android/33/Mono.Android.pdb
{
  "documents": {
    …
    "…/xamarin/xamarin-android/external/Java.Interop/*": "https://raw.githubusercontent.com/xamarin/java.interop/a5756ca8b8764c24cc169e40a473f79302b40ca9/*",
    "…/xamarin/xamarin-android/external/xamarin-android-tools/*": "https://raw.githubusercontent.com/xamarin/xamarin-android-tools/9c641b3e08e56db37467a64a2c5de2c7f7ddb3ef/*",
    "…/xamarin/xamarin-android/*": "https://raw.githubusercontent.com/xamarin/xamarin-android/903ba37ce70d2840983774e1d6fb55f8002561e2/*"
  }
}

Finally, to verify that all URLs from sourcelink print-urls are
valid, use sourcelink test:

% $HOME/.dotnet/tools/sourcelink test bin/Debug/lib/xamarin.android/xbuild-frameworks/Microsoft.Android/33/Mono.Android.pdb

Note: this can take awhile.

To make all this work, update xaprepare to create a
bin/Build$(Configuration)/SourceLink.json file, and update
Configuration.props so that $(SourceLink) is set to the generated
SourceLink.json file, if it exists. SourceLink.json is generated
by reading .gitmodules to determine all of the git submodules, and
using that information to construct the "remote" URL prefix.

TODO:

SourceLink.json assumes that all remote URLs are GitHub URLs.
This may not be ideal, but works for most of our assemblies.

What should be done about generated files?


public GeneratedFile Get_SourceLink_Json (Context context)
{
return new GeneratedSourceLinkJsonFile (
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the PR is missing the GeneratedSourceLinkJsonFile sources :(

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doh!

@jonpryor jonpryor force-pushed the jonp-mono.android-sourcelink branch 2 times, most recently from 8a1af6a to dcab9a0 Compare August 23, 2022 12:13
{
if (submodules == null)
throw new ArgumentNullException (nameof (submodules));
this.submodules = submodules;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

turn that to

this.submodules = submodules ?? throw new ArgumentNullException(nameof (submodules));

to save some vertical space? :)


json.AppendLine ($" \"{localPath}/*\": \"{contentUri.Uri}/*\",");
}
json.AppendLine ($" \"{BuildPaths.XamarinAndroidSourceRoot}/*\": \"https://raw.githubusercontent.com/xamarin/xamarin-android/{xaCommit}/*\"");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Constructor doesn't check whether xaCommit is valid and given that it is used to form a URL, perhaps it should?

Fixes: dotnet#2614

Context: https://developercommunity.visualstudio.com/t/JNINativeWrappergcs-file-Not-found/10020777
Context: https://www.nuget.org/packages/sourcelink/
Context: https://github.com/dotnet/designs/blob/main/accepted/2020/diagnostics/source-link.md#source-link-file-specification
Context: https://docs.microsoft.com/en-us/dotnet/standard/library-guidance/sourcelink
Context: https://www.hanselman.com/blog/exploring-net-cores-sourcelink-stepping-into-the-source-code-of-nuget-packages-you-dont-own

[SourceLink][0] is a set of tooling which allows debuggers to obtain
source code, allowing developers to "step into" assemblies for which
they don't have local source code to.

It operates by adding a JSON document to Portable PDB files, which
allows mapping paths within the Portable PDB file to a URL, which the
debugger can optionally download.

The JSON document contains a `documents` key, which contains a nested
dictionary which maps local filesystem path prefixes to URLs, e.g.

	{
	  "documents": {
	    "…/xamarin/xamarin-android/external/Java.Interop/*":            "https://raw.githubusercontent.com/xamarin/java.interop/a5756ca8b8764c24cc169e40a473f79302b40ca9/*",
	    "…/xamarin/xamarin-android/external/xamarin-android-tools/*":   "https://raw.githubusercontent.com/xamarin/xamarin-android-tools/9c641b3e08e56db37467a64a2c5de2c7f7ddb3ef/*",
	    "…/xamarin/xamarin-android/*":                                  "https://raw.githubusercontent.com/xamarin/xamarin-android/903ba37ce70d2840983774e1d6fb55f8002561e2/*"
	  }
	}

This JSON document can be provided to the C# compiler by setting the
`$(SourceLink)` MSBuild property to the path of the JSON document.

The [sourcelink][1] dotnet global tool can be used to check `.pdb`
file contents:

	% dotnet tool install --global SourceLink --version 3.1.1

To view the *local paths* within a `.pdb` file, use
`sourcelink print-documents`:

	% $HOME/.dotnet/tools/sourcelink print-documents bin/Debug/lib/xamarin.android/xbuild-frameworks/Microsoft.Android/33/Mono.Android.pdb
	9c5ad5a588d5a49deb98ccd63c7b0c4699c8f39d6ee1df4a8f4ae555be827e06 sha256 csharp …/xamarin-android/external/Java.Interop/src/Java.Interop.Tools.TypeNameMappings/Java.Interop.Tools.TypeNameMappings/JavaNativeTypeManager.cs
	…

To view the *remote URLs* for those paths, use `sourcelink print-urls`:

	% $HOME/.dotnet/tools/sourcelink print-urls bin/Debug/lib/xamarin.android/xbuild-frameworks/Microsoft.Android/33/Mono.Android.pdb
	9c5ad5a588d5a49deb98ccd63c7b0c4699c8f39d6ee1df4a8f4ae555be827e06 sha256 csharp …/xamarin-android/external/Java.Interop/src/Java.Interop.Tools.TypeNameMappings/Java.Interop.Tools.TypeNameMappings/JavaNativeTypeManager.cs
	https://raw.githubusercontent.com/xamarin/java.interop/a5756ca8b8764c24cc169e40a473f79302b40ca9/src/Java.Interop.Tools.TypeNameMappings/Java.Interop.Tools.TypeNameMappings/JavaNativeTypeManager.cs
	…

To view the contents of the JSON document provided to the compiler,
use `sourcelink print-json`:

	%  $HOME/.dotnet/tools/sourcelink print-json bin/Debug/lib/xamarin.android/xbuild-frameworks/Microsoft.Android/33/Mono.Android.pdb
	{
	  "documents": {
	    …
	    "…/xamarin/xamarin-android/external/Java.Interop/*": "https://raw.githubusercontent.com/xamarin/java.interop/a5756ca8b8764c24cc169e40a473f79302b40ca9/*",
	    "…/xamarin/xamarin-android/external/xamarin-android-tools/*": "https://raw.githubusercontent.com/xamarin/xamarin-android-tools/9c641b3e08e56db37467a64a2c5de2c7f7ddb3ef/*",
	    "…/xamarin/xamarin-android/*": "https://raw.githubusercontent.com/xamarin/xamarin-android/903ba37ce70d2840983774e1d6fb55f8002561e2/*"
	  }
	}

Finally, to verify that all URLs from `sourcelink print-urls` are
valid, use `sourcelink test`:

	% $HOME/.dotnet/tools/sourcelink test bin/Debug/lib/xamarin.android/xbuild-frameworks/Microsoft.Android/33/Mono.Android.pdb
	nodename nor servname provided, or not known
	# 🤔

	% $HOME/.dotnet/tools/sourcelink test bin/Debug/lib/packs/Microsoft.Android.Sdk.Darwin/33.0.0/tools/Xamarin.Android.Build.Tasks.pdb
	83 Documents with errors:
	962d50c8d14a454ff562f843051e542c22c69b63b489f1d2922d092aba3f578b sha256 csharp …/xamarin-android/src/Xamarin.Android.Build.Tasks/obj/Release/Profile.g.cs
	https://raw.githubusercontent.com/xamarin/xamarin-android/4a94e294fbf8dc85a319386d4aab3526718769c0/src/Xamarin.Android.Build.Tasks/obj/Release/Profile.g.cs
	error: url failed NotFound: Not Found
	…
	sourcelink test failed

Note: this can take awhile.

Some of these errors are to be expected, as they reference generated
files which are not available via `raw.githubusercontent.com`.

Other errors suggest that we need to "extend" this system, e.g.

	0b1fe6595fff8c1af58a6f97a6bc79227300ae7941bd113ad092c36b418c14b9 sha256 csharp …/xamarin-android/external/mono/sdks/out/android-sources/external/linker/src/tuner/Mono.Tuner/PrintStatus.cs
	https://raw.githubusercontent.com/xamarin/xamarin-android/4a94e294fbf8dc85a319386d4aab3526718769c0/external/mono/sdks/out/android-sources/external/linker/src/tuner/Mono.Tuner/PrintStatus.cs
	error: url failed NotFound: Not Found

A path of `external/…/external` suggests that we need more entries.

To make all this work, update `xaprepare` to create a
`bin/Build$(Configuration)/SourceLink.json` file, and update
`Configuration.props` so that `$(SourceLink)` is set to the generated
`SourceLink.json` file, if it exists.  `SourceLink.json` is generated
by reading `.gitmodules` to determine all of the git submodules, and
using that information to construct the "remote" URL prefix.

TODO:

`SourceLink.json` assumes that all remote URLs are GitHub URLs.
This may not be ideal, but works for most of our assemblies.

What should be done about *generated* files?

Is there a good way to deal with xamarin-android referencing mono
files which reference linker files?

[0]: https://github.com/dotnet/sourcelink
[1]: https://www.nuget.org/packages/sourcelink
@jonpryor jonpryor force-pushed the jonp-mono.android-sourcelink branch from dcab9a0 to 5a6c179 Compare August 23, 2022 15:14
@jonpryor jonpryor merged commit 7b4d4b8 into dotnet:main Aug 23, 2022
grendello added a commit to grendello/xamarin-android that referenced this pull request Aug 24, 2022
* main:
  [xaprepare] Generate SourceLink.json (dotnet#7298)
grendello added a commit to grendello/xamarin-android that referenced this pull request Aug 24, 2022
* mm-runtime:
  [xaprepare] Generate SourceLink.json (dotnet#7298)
@github-actions github-actions bot locked and limited conversation to collaborators Jan 24, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

SourceLink support
2 participants