[Xamarin.Android.Build.Tasks] minimal Reference Assemblies support #2474

jonathanpeppers · 2018-11-30T17:42:09Z

This is the beginning of our support for MSBuild/Roslyn "Reference
Assemblies".

Adding the following to a csproj file:

<ProduceReferenceAssembly>True</ProduceReferenceAssembly>

Causes a bin\Debug\ref\MyLibrary.dll to exist alongside
bin\Debug\MyLibrary.dll.

Two item groups, @(ReferenceCopyLocalPaths) and @(ReferencePath)
will have new %(ReferenceAssembly) metadata:

bin\Debug\MyLibrary.dll
    ReferenceAssembly = C:\full\path\to\bin\Debug\ref\MyLibrary.dll

If an assembly does not include a reference assembly, it will still
maintain %(ReferenceAssembly) metadata pointing to itself:

bin\Debug\OtherLibrary.dll
    ReferenceAssembly = C:\full\path\to\bin\Debug\OtherLibrary.dll

In some cases this metadata might not be there, such as using a path
as input to ResolveAssemblies ($(OutDir)$(TargetFileName)), so our
ResolveAssemblies MSBuild task should produce the value if it does
not exist.

Changes

The biggest change here is to rework ResolveAssemblies so that it
preserves item metadata. The metadata was getting lost in various
ways:

Use of %(ReferencePath.Identity) instead of @(ReferencePath).
Any new TaskItem instances created should use the ctor taking in
an existing ITaskItem. This preserves the metadata from the
original item.

We also needed to make some changes to support %(ReferenceAssembly):

New TaskItem objects created by a file path should set
%(ReferenceAssembly).
We should check if %(ReferenceAssembly) is missing, and set it
where appropriate.

I looked into reworking _GenerateJavaStubs so it can be skipped
when a reference assembly didn't change.

I merely changed a single input: @(_ResolvedAssemblies) to
@(_ResolvedAssemblies->'%(ReferenceAssembly)').

Unfortunately, this would break! A developer could write this
completely valid C# class:

internal class Example : Java.Lang.Object { }

However, since we have preserved item metadata, we can now use
%(ReferenceAssembly) throughout the build. We can take advantage of
it in the future.

I added some documentation about what I found out with MSBuild
"Reference Assembly" support, in general. I also added a unit test
verifying which targets skip when $(ProduceReferenceAssembly) is set
in a library project.

I also reworked our Xamarin.Forms integration project so that it is
netstandard instead of a shared project.

This will help us test $(ProduceReference) assembly, and also has a
much simpler project file!

Other changes:

I let VS Windows fixup the Xamarin.Android-Tests.sln file. It
looks like there were a few mismatched guids and other things.
We did not have XamlC enabled:
[assembly: XamlCompilation (XamlCompilationOptions.Compile)]
this will greatly improve startup times!
We will need to call /t:Restore on the Xamarin.Forms netstandard
project now.

jonathanpeppers · 2018-11-30T17:42:43Z

Documentation/guides/MSBuildReferenceAssemblies.md

+to be rebuilt *less often*.
+
+[msbuild_refassemblies]: https://github.com/dotnet/roslyn/blob/master/docs/features/refout.md
+


The above section was the relevant good stuff from here, written by @jonpryor : https://github.com/xamarin/xamarin-android/wiki/Build-Performance-Ideas#embrace-reference-assemblies

jonpryor · 2018-11-30T20:45:52Z

Should this also set $(ProduceReferenceAssembly)=True in e.g. tests/CodeGen-Binding//Xamarin.Android.LibraryProjectZip-LibBinding/Xamarin.Android.LibraryProjectZip-LibBinding.csproj?

jonathanpeppers · 2018-11-30T21:39:30Z

Hmm, that project would be good, but I don’t think anything would test an incremental build on it...

I think it might actually make more sense to convert the Xamarin.Forms integration project to use NetStandard instead of a shared project. Then enable it there.

@radekdoulik would that be ok with you? Would that hurt our plots?

jonathanpeppers · 2018-12-04T22:57:23Z

Well looks like calling /t:Restore directly isn't a good idea on macOS?

/Library/Frameworks/Mono.framework/Versions/5.16.0/lib/mono/msbuild/15.0/bin/MSBuild.dll /binaryLogger:/Users/builder/jenkins/workspace/xamarin-android-pr-builder-release/xamarin-android//bin/BuildRelease/msbuild-20181204T120645-prepare-external.binlog /p:Configuration=Release /p:AutoProvision=True /p:AutoProvisionUsesSudo=True /p:IgnoreMaxMonoVersion=False /p:Configuration=Release /p:AutoProvision=True /p:AutoProvisionUsesSudo=True /p:IgnoreMaxMonoVersion=False /t:Restore /v:normal Xamarin.Android-Tests.sln
...
TestRunner.xUnit.csproj : error MSB4057: The target "Restore" does not exist in the project.

I'm not sure why, it seemed to work for me locally?

Makefile

build-tools/scripts/PrepareWindows.targets

build-tools/timing/timing.targets

jonathanpeppers · 2018-12-05T17:11:19Z

Windows is green:

assets/subfolder/asset2.txt should be in the apk.

This is fixed in #2480

src/Xamarin.Android.Build.Tasks/Xamarin.Android.Common.targets

jonpryor · 2018-12-05T18:03:58Z

Suggested Squash-and-merge commit message:

[Xamarin.Android.Build.Tasks] Start Reference Assemblies support (#2474)

This is the beginning of our support for MSBuild/Roslyn
"Reference Assemblies".

Adding the following to a `.csproj` file:

	<ProduceReferenceAssembly>True</ProduceReferenceAssembly>

Causes a `bin\Debug\ref\MyLibrary.dll` to be built alongside
`bin\Debug\MyLibrary.dll`.  The new `bin\Debug\ref\MyLibrary.dll`
file is *only* changed whenever *`public` and `protected`* API in
`MyLibrary.dll` is changed.

MSBuild creates two item groups, `@(ReferenceCopyLocalPaths)` and
`@(ReferencePath)`, which have `%(ReferenceAssembly)` metadata:

	bin\Debug\MyLibrary.dll
	    ReferenceAssembly = C:\full\path\to\bin\Debug\ref\MyLibrary.dll

If an assembly does *not* include a reference assembly, it will still
maintain `%(ReferenceAssembly)` metadata pointing to itself:

	bin\Debug\OtherLibrary.dll
	    ReferenceAssembly = C:\full\path\to\bin\Debug\OtherLibrary.dll

In some cases this metadata might not be there, such as using a path
as input to `<ResolveAssemblies/>` (via `$(OutDir)$(TargetFileName)`),
so our `<ResolveAssemblies/>` task will produce the value if it does
not exist.

~~ Changes ~~

The biggest change here is to rework `<ResolveAssemblies/>` so that
it preserves item metadata.  The metadata was getting lost in
various ways:

  - Use of `%(ReferencePath.Identity)` instead of `@(ReferencePath)`.
  - Any new `TaskItem` instances created should use the ctor taking
    in an existing `ITaskItem`.  This preserves the metadata from the
    original item.

We also needed to make some changes to support `%(ReferenceAssembly)`:

  - New `TaskItem` objects created by a file path should set
    `%(ReferenceAssembly)`.
  - We should check if `%(ReferenceAssembly)` is missing, and set it
    where appropriate.

I added some documentation about what I found out with MSBuild
"Reference Assembly" support, in general.  I also added a unit test
verifying which targets skip when `$(ProduceReferenceAssembly)` is
set in a library project.

To help test Reference Assembly support,
`tests/Xamarin.Forms-Performance-Integration` has been updated so a
.NET Standard 2 project is used instead of a Shared Project.

This will help us test `$(ProduceReferenceAssembly)`, and also has a
much simpler project file!

Other changes:

  - I let VS Windows fixup the `Xamarin.Android-Tests.sln file`.
    It looks like there were a few mismatched GUIDs and other things.
  - We did not have `<XamlC/>` enabled:
    `[assembly: XamlCompilation (XamlCompilationOptions.Compile)]`
    should greatly improve startup times!

This is the beginning of our support for MSBuild/Roslyn "Reference Assemblies". Adding the following to a `csproj` file: <ProduceReferenceAssembly>True</ProduceReferenceAssembly> Causes a `bin\Debug\ref\MyLibrary.dll` to exist alongside `bin\Debug\MyLibrary.dll`. Two item groups, `@(ReferenceCopyLocalPaths)` and `@(ReferencePath)` will have new `%(ReferenceAssembly)` metadata: bin\Debug\MyLibrary.dll ReferenceAssembly = C:\full\path\to\bin\Debug\ref\MyLibrary.dll If an assembly does *not* include a reference assembly, it will still maintain `%(ReferenceAssembly)` metadata pointing to itself: bin\Debug\OtherLibrary.dll ReferenceAssembly = C:\full\path\to\bin\Debug\OtherLibrary.dll In some cases this metadata might not be there, such as using a path as input to `ResolveAssemblies` (`$(OutDir)$(TargetFileName)`), so our `ResolveAssemblies` MSBuild task should produce the value if it does not exist. ~~ Changes ~~ The biggest change here is to rework `ResolveAssemblies` so that it preserves item metadata. The metadata was getting lost in various ways: - Use of `%(ReferencePath.Identity)` instead of `@(ReferencePath)`. - Any new `TaskItem` instances created should use the ctor taking in an existing `ITaskItem`. This preserves the metadata from the original item. We also needed to make some changes to support `%(ReferenceAssembly)`: - New `TaskItem` objects created by a file path should set `%(ReferenceAssembly)`. - We should check if `%(ReferenceAssembly)` is missing, and set it where appropriate. I looked into reworking `_GenerateJavaStubs` so it can be skipped when a reference assembly didn't change. I merely changed a single input: `@(_ResolvedAssemblies)` to `@(_ResolvedAssemblies->'%(ReferenceAssembly)')`. Unfortunately, this would break! A developer could write this completely valid C# class: internal class Example : Java.Lang.Object { } However, since we have preserved item metadata, we can now use `%(ReferenceAssembly)` throughout the build. We can take advantage of it in the future. I added some documentation about what I found out with MSBuild "Reference Assembly" support, in general. I also added a unit test verifying which targets skip when `$(ProduceReferenceAssembly)` is set in a library project. I also reworked our Xamarin.Forms integration project so that it is netstandard instead of a shared project. This will help us test `$(ProduceReference)` assembly, and also has a much simpler project file! Other changes: - I let VS Windows fixup the `Xamarin.Android-Tests.sln file`. It looks like there were a few mismatched guids and other things. - We did not have XamlC enabled: `[assembly: XamlCompilation (XamlCompilationOptions.Compile)]` this will greatly improve startup times! - We will need to call `/t:Restore` on the Xamarin.Forms netstandard project now.

MagicAndre1981 · 2019-08-13T07:03:10Z

@jonathanpeppers

in which version is this included? In VS2017 15.9, I always get the message that the ref dll is modified even if I only change one value in XAML file.

"not up to date as bin\Debug\ref\FooApp.dll is modified after output file"

jonathanpeppers · 2019-08-13T11:36:22Z

You need VS 2019 16.0 for XAML files to work properly in reference assemblies. I reported a bug for Roslyn around EmbeddedResource, they fixed and shipped in 16.0 GA.

dotnet/roslyn#31197

MagicAndre1981 · 2019-08-13T12:29:40Z

@jonathanpeppers thanks for the information. This explains it.

MagicAndre1981 · 2019-08-13T14:47:45Z

ok, works in 16.3 Preview1

Reference assembly bin\Debug\ref\Foo.App.dll" already has latest information. Leaving it untouched.

@jonathanpeppers

please update your blogpost "Optimize your Xamarin.Android Builds" and add this information.

jonathanpeppers · 2019-08-13T15:04:05Z

That blog post is assuming you are using 16.2 stable. Did that not work for you?

MagicAndre1981 · 2019-08-13T15:06:29Z

@jonathanpeppers this was not clear. I hate VS2019 because of the broken UI and so I'll stay at VS2017 and will likely skip 2019 at all. I only have a VM with Preview channel to test if UI designers repaired the broken UI mess.

jonathanpeppers · 2019-08-13T15:14:08Z

I added a sentence to the intro paragraph, thanks: https://devblogs.microsoft.com/xamarin/optimize-xamarin-android-builds/

jonathanpeppers requested review from jonpryor and dellis1972 November 30, 2018 17:42

jonathanpeppers commented Nov 30, 2018

View reviewed changes

jonathanpeppers force-pushed the referenceassemblies-babysteps branch from f053ff8 to 72f065e Compare December 3, 2018 19:14

jonathanpeppers force-pushed the referenceassemblies-babysteps branch 2 times, most recently from 3e04390 to 89558df Compare December 5, 2018 04:57

jonathanpeppers requested a review from radekdoulik as a code owner December 5, 2018 04:57

dellis1972 approved these changes Dec 5, 2018

View reviewed changes

jonpryor reviewed Dec 5, 2018

View reviewed changes

Makefile Outdated Show resolved Hide resolved

jonpryor reviewed Dec 5, 2018

View reviewed changes

build-tools/scripts/PrepareWindows.targets Outdated Show resolved Hide resolved

jonpryor reviewed Dec 5, 2018

View reviewed changes

build-tools/timing/timing.targets Outdated Show resolved Hide resolved

jonathanpeppers force-pushed the referenceassemblies-babysteps branch 3 times, most recently from 12b3310 to c382404 Compare December 5, 2018 15:42

jonathanpeppers force-pushed the referenceassemblies-babysteps branch from c382404 to ad7d1c8 Compare December 5, 2018 17:44

jonpryor reviewed Dec 5, 2018

View reviewed changes

src/Xamarin.Android.Build.Tasks/Xamarin.Android.Common.targets Outdated Show resolved Hide resolved

jonathanpeppers force-pushed the referenceassemblies-babysteps branch from ad7d1c8 to 61ccb05 Compare December 5, 2018 18:22

jonpryor merged commit 83559e2 into xamarin:master Dec 5, 2018

jonathanpeppers deleted the referenceassemblies-babysteps branch August 13, 2019 11:34

github-actions bot locked and limited conversation to collaborators Feb 1, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Xamarin.Android.Build.Tasks] minimal Reference Assemblies support #2474

[Xamarin.Android.Build.Tasks] minimal Reference Assemblies support #2474

jonathanpeppers commented Nov 30, 2018 •

edited

jonathanpeppers Nov 30, 2018 •

edited

jonpryor commented Nov 30, 2018

jonathanpeppers commented Nov 30, 2018

jonathanpeppers commented Dec 4, 2018 •

edited

jonathanpeppers commented Dec 5, 2018

jonpryor commented Dec 5, 2018

MagicAndre1981 commented Aug 13, 2019

jonathanpeppers commented Aug 13, 2019

MagicAndre1981 commented Aug 13, 2019

MagicAndre1981 commented Aug 13, 2019

jonathanpeppers commented Aug 13, 2019

MagicAndre1981 commented Aug 13, 2019

jonathanpeppers commented Aug 13, 2019

		to be rebuilt less often.

		[msbuild_refassemblies]: https://github.com/dotnet/roslyn/blob/master/docs/features/refout.md

[Xamarin.Android.Build.Tasks] minimal Reference Assemblies support #2474

[Xamarin.Android.Build.Tasks] minimal Reference Assemblies support #2474

Conversation

jonathanpeppers commented Nov 30, 2018 • edited

Changes

jonathanpeppers Nov 30, 2018 • edited

Choose a reason for hiding this comment

jonpryor commented Nov 30, 2018

jonathanpeppers commented Nov 30, 2018

jonathanpeppers commented Dec 4, 2018 • edited

jonathanpeppers commented Dec 5, 2018

jonpryor commented Dec 5, 2018

MagicAndre1981 commented Aug 13, 2019

jonathanpeppers commented Aug 13, 2019

MagicAndre1981 commented Aug 13, 2019

MagicAndre1981 commented Aug 13, 2019

jonathanpeppers commented Aug 13, 2019

MagicAndre1981 commented Aug 13, 2019

jonathanpeppers commented Aug 13, 2019

jonathanpeppers commented Nov 30, 2018 •

edited

jonathanpeppers Nov 30, 2018 •

edited

jonathanpeppers commented Dec 4, 2018 •

edited