Skip to content

Commit

Permalink
[Xamarin.Android.Build.Tasks] mechanism to skip ConvertResourcesCases
Browse files Browse the repository at this point in the history 
Currently, we run the `ConvertResourcesCases` MSBuild task for all
assemblies (on each assemby's `ResolveLibraryProjectImports` output).
`ConvertResourcesCases` is one of our slower targets.

Luckily, there are some "well-known" number of assemblies that we
could skip, so we have decided to "whitelist" certain assemblies such
as the Android support libraries, Google Play Services, Firebase, etc.

So a new `@(_AndroidAssemblySkipCases)` item group has been added such
as:

    <_AndroidAssemblySkipCases Include="Xamarin.Android.Support.v7.AppCompat" />
    <_AndroidAssemblySkipCases Include="Xamarin.Android.Support.v7.CardView" />

This `<ItemGroup/>` would be an indicator for `ConvertResourcesCases`
to just completely skip these assemblies.

Additionally, we can flat out skip `aar` files in the same manner.

To make this work:
- Added support to put `ITaskItem` metadata in the cache file produced
  by `ResolveLibraryProjectImports`
- Added item metadata for `SkipAndroidResourceProcessing` and
  `OriginalFile`.
- `ConvertResourcesCases` now skips these directories and logs
  `OriginalFile`.
- `CollectNonEmptyDirectories` needs to preserve item metadata for
  `$(AndroidUseAapt2)` to take advantage of the functionality.

The results appear to be well worth the effort!

Results with `$(AndroidUseAapt2)` enabled (note this is not the
default):

    Before:
    7841 ms  ConvertResourcesCases                      9 calls
    1588 ms  ResolveLibraryProjectImports               1 calls

    After:
      69 ms  ConvertResourcesCases                      9 calls
    1875 ms  ResolveLibraryProjectImports               1 calls

Results with `$(AndroidUseAapt2)` disabled:

    Before:
    1216 ms  ConvertResourcesCases                      1 calls
    1521 ms  ResolveLibraryProjectImports               1 calls

    After:
      24 ms  ConvertResourcesCases                      1 calls
    1578 ms  ResolveLibraryProjectImports               1 calls

This was the Xamarin.Forms-Integration project in this repo, an
initial clean build. It is basically a "Hello World" Xamarin.Forms
project. These updated numbers are from a `Release` build of
Xamarin.Android.

Overall this will save ~1.2 seconds of `ConvertResourcesCases`, which
happens on an initial build or incremental builds when Android
resources have changes. There is also a slight slowdown in
`ResolveLibraryProjectImports` due to the changes.
  • Loading branch information
@jonathanpeppers
jonathanpeppers committed Oct 29, 2018
1 parent 3dc914d commit 4de096d
Show file tree
Hide file tree
Showing 8 changed files with 323 additions and 36 deletions.
6 changes: 4 additions & 2 deletions src/Xamarin.Android.Build.Tasks/Tasks/CollectNonEmptyDirectories.cs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,11 @@ public override bool Execute ()
foreach (var directory in Directories) {
var firstFile = Directory.EnumerateFiles(directory.ItemSpec, "*.*", SearchOption.AllDirectories).FirstOrDefault ();
if (firstFile != null) {
output.Add (new TaskItem (directory.ItemSpec, new Dictionary<string, string> () {
var taskItem = new TaskItem (directory.ItemSpec, new Dictionary<string, string> () {
{"FileFound", firstFile}
}));
});
directory.CopyMetadataTo (taskItem);
output.Add (taskItem);
}
}
return !Log.HasLoggedErrors;
Expand Down
11 changes: 9 additions & 2 deletions src/Xamarin.Android.Build.Tasks/Tasks/ConvertResourcesCases.cs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -47,11 +47,18 @@ public override bool Execute ()
return true;
}


void FixupResources (Dictionary<string, string> acwMap)
{
foreach (var dir in ResourceDirectories)
foreach (var dir in ResourceDirectories) {
var skipResourceProcessing = dir.GetMetadata (ResolveLibraryProjectImports.SkipAndroidResourceProcessing);
if (skipResourceProcessing != null && skipResourceProcessing.Equals ("true", StringComparison.OrdinalIgnoreCase)) {
var originalFile = dir.GetMetadata (ResolveLibraryProjectImports.OriginalFile);
Log.LogDebugMessage ($"Skipping: `{dir.ItemSpec}` via `{ResolveLibraryProjectImports.SkipAndroidResourceProcessing}`, original file: `{originalFile}`...");
continue;
}

FixupResources (dir, acwMap);
}
}

void FixupResources (ITaskItem item, Dictionary<string, string> acwMap)
Expand Down
76 changes: 49 additions & 27 deletions src/Xamarin.Android.Build.Tasks/Tasks/ResolveLibraryProjectImports.cs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,8 @@ public class ResolveLibraryProjectImports : Task

public string CacheFile { get; set; }

public string [] AssembliesToSkipCases { get; set; }

[Required]
public bool DesignTimeBuild { get; set; }

Expand All @@ -60,6 +62,13 @@ public class ResolveLibraryProjectImports : Task
[Output]
public ITaskItem [] ResolvedResourceDirectoryStamps { get; set; }

internal const string OriginalFile = "OriginalFile";
internal const string SkipAndroidResourceProcessing = "SkipAndroidResourceProcessing";
static readonly string [] knownMetadata = new [] {
OriginalFile,
SkipAndroidResourceProcessing
};

AssemblyIdentityMap assemblyMap = new AssemblyIdentityMap();

public ResolveLibraryProjectImports ()
Expand All @@ -70,16 +79,8 @@ public ResolveLibraryProjectImports ()
// Extracts library project contents under e.g. obj/Debug/[lp/*.jar | res/*/*]
public override bool Execute ()
{
Log.LogDebugMessage ("ResolveLibraryProjectImports Task");
Log.LogDebugMessage (" ImportsDirectory: {0}", ImportsDirectory);
Log.LogDebugMessage (" OutputDirectory: {0}", OutputDirectory);
Log.LogDebugMessage (" OutputImportDirectory: {0}", OutputImportDirectory);
Log.LogDebugMessage (" UseShortFileNames: {0}", UseShortFileNames);
Log.LogDebugTaskItems (" Assemblies: ", Assemblies);
Log.LogDebugTaskItems (" AarLibraries: ", AarLibraries);

var jars = new List<string> ();
var resolvedResourceDirectories = new List<string> ();
var resolvedResourceDirectories = new List<ITaskItem> ();
var resolvedAssetDirectories = new List<string> ();
var resolvedEnvironmentFiles = new List<string> ();

Expand All @@ -95,9 +96,7 @@ public override bool Execute ()
}

Jars = jars.ToArray ();
ResolvedResourceDirectories = resolvedResourceDirectories
.Select (s => new TaskItem (Path.GetFullPath (s)))
.ToArray ();
ResolvedResourceDirectories = resolvedResourceDirectories.ToArray ();
ResolvedAssetDirectories = resolvedAssetDirectories.ToArray ();
ResolvedEnvironmentFiles = resolvedEnvironmentFiles.ToArray ();

Expand All @@ -120,7 +119,15 @@ public override bool Execute ()
new XElement ("Jars",
Jars.Select(e => new XElement ("Jar", e))),
new XElement ("ResolvedResourceDirectories",
ResolvedResourceDirectories.Select(e => new XElement ("ResolvedResourceDirectory", e))),
ResolvedResourceDirectories.Select(dir => {
var e = new XElement ("ResolvedResourceDirectory", dir.ItemSpec);
foreach (var name in knownMetadata) {
var value = dir.GetMetadata (name);
if (!string.IsNullOrEmpty (value))
e.SetAttributeValue (name, value);
}
return e;
})),
new XElement ("ResolvedAssetDirectories",
ResolvedAssetDirectories.Select(e => new XElement ("ResolvedAssetDirectory", e))),
new XElement ("ResolvedEnvironmentFiles",
Expand All @@ -133,12 +140,6 @@ public override bool Execute ()

assemblyMap.Save (AssemblyIdentityMapFile);

Log.LogDebugTaskItems (" Jars: ", Jars.Select (s => new TaskItem (s)).ToArray ());
Log.LogDebugTaskItems (" ResolvedResourceDirectories: ", ResolvedResourceDirectories.Select (s => new TaskItem (s)).ToArray ());
Log.LogDebugTaskItems (" ResolvedAssetDirectories: ", ResolvedAssetDirectories.Select (s => new TaskItem (s)).ToArray ());
Log.LogDebugTaskItems (" ResolvedEnvironmentFiles: ", ResolvedEnvironmentFiles.Select (s => new TaskItem (s)).ToArray ());
Log.LogDebugTaskItems (" ResolvedResourceDirectoryStamps: ", ResolvedResourceDirectoryStamps);

return !Log.HasLoggedErrors;
}

Expand All @@ -160,7 +161,7 @@ static string GetTargetAssembly (ITaskItem assemblyName)
void Extract (
DirectoryAssemblyResolver res,
ICollection<string> jars,
ICollection<string> resolvedResourceDirectories,
ICollection<ITaskItem> resolvedResourceDirectories,
ICollection<string> resolvedAssetDirectories,
ICollection<string> resolvedEnvironments)
{
Expand All @@ -183,9 +184,10 @@ void Extract (
.Select (a => GetTargetAssembly (a))
.Where (a => a != null)
.Distinct ()) {
string assemblyIdentName = Path.GetFileNameWithoutExtension (assemblyPath);
string assemblyFileName = Path.GetFileNameWithoutExtension (assemblyPath);
string assemblyIdentName = assemblyFileName;
if (UseShortFileNames) {
assemblyIdentName = assemblyMap.GetLibraryImportDirectoryNameForAssembly (assemblyIdentName);
assemblyIdentName = assemblyMap.GetLibraryImportDirectoryNameForAssembly (assemblyFileName);
}
string outDirForDll = Path.Combine (OutputImportDirectory, assemblyIdentName);
string importsDir = Path.Combine (outDirForDll, ImportsDirectory);
Expand All @@ -210,8 +212,14 @@ void Extract (
if (Directory.Exists (binAssemblyDir))
resolvedAssetDirectories.Add (binAssemblyDir);
#endif
if (Directory.Exists (resDir))
resolvedResourceDirectories.Add (resDir);
if (Directory.Exists (resDir)) {
var taskItem = new TaskItem (resDir, new Dictionary<string, string> {
{ OriginalFile, assemblyPath },
});
if (ShouldSkipAssembly (assemblyFileName))
taskItem.SetMetadata (SkipAndroidResourceProcessing, "True");
resolvedResourceDirectories.Add (taskItem);
}
if (Directory.Exists (assemblyDir))
resolvedAssetDirectories.Add (assemblyDir);
foreach (var env in Directory.EnumerateFiles (outDirForDll, "__AndroidEnvironment__*", SearchOption.TopDirectoryOnly)) {
Expand Down Expand Up @@ -308,8 +316,14 @@ void Extract (
if (Directory.Exists (binAssemblyDir))
resolvedAssetDirectories.Add (binAssemblyDir);
#endif
if (Directory.Exists (resDir))
resolvedResourceDirectories.Add (resDir);
if (Directory.Exists (resDir)) {
var taskItem = new TaskItem (resDir, new Dictionary<string, string> {
{ OriginalFile, assemblyPath }
});
if (ShouldSkipAssembly (assemblyFileName))
taskItem.SetMetadata (SkipAndroidResourceProcessing, "True");
resolvedResourceDirectories.Add (taskItem);
}
if (Directory.Exists (assemblyDir))
resolvedAssetDirectories.Add (assemblyDir);

Expand Down Expand Up @@ -362,7 +376,10 @@ void Extract (
}
}
if (Directory.Exists (resDir))
resolvedResourceDirectories.Add (resDir);
resolvedResourceDirectories.Add (new TaskItem (resDir, new Dictionary<string, string> {
{ OriginalFile, Path.GetFullPath (aarFile.ItemSpec) },
{ SkipAndroidResourceProcessing, "True" },
}));
if (Directory.Exists (assetsDir))
resolvedAssetDirectories.Add (assetsDir);
}
Expand All @@ -373,5 +390,10 @@ void Extract (
jars.Add (f);
}
}

bool ShouldSkipAssembly (string assemblyFileName)
{
return AssembliesToSkipCases != null && AssembliesToSkipCases.Contains (assemblyFileName, StringComparer.OrdinalIgnoreCase);
}
}
}
50 changes: 50 additions & 0 deletions src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/BuildTest.cs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,56 @@ public void BuildBasicApplicationReleaseFSharp ()
}
}

[Test]
public void SkipConvertResourcesCases ()
{
var target = "ConvertResourcesCases";
var proj = new XamarinFormsAndroidApplicationProject ();
proj.OtherBuildItems.Add (new BuildItem ("AndroidAarLibrary", "Jars\\material-menu-1.1.0.aar") {
WebContent = "https://repo.jfrog.org/artifactory/libs-release-bintray/com/balysv/material-menu/1.1.0/material-menu-1.1.0.aar"
});
using (var b = CreateApkBuilder (Path.Combine ("temp", TestName))) {
Assert.IsTrue (b.Build (proj), "Build should have succeeded.");
Assert.IsFalse (b.Output.IsTargetSkipped (target), $"`{target}` should not be skipped.");

int skipped = 0, processed = 0;
bool convertResourcesCases = false;
foreach (var line in b.LastBuildOutput) {
if (!convertResourcesCases) {
convertResourcesCases = line.StartsWith ($"Task \"{target}\"", StringComparison.OrdinalIgnoreCase);
} else if (line.StartsWith ($"Done executing task \"{target}\"", StringComparison.OrdinalIgnoreCase)) {
break; //end of target
} else if (line.IndexOf ("Processing:", StringComparison.OrdinalIgnoreCase) >= 0) {
processed++;
} else if (line.IndexOf ("Skipping:", StringComparison.OrdinalIgnoreCase) >= 0) {
skipped++;
}
}

/*
Processing: obj\Debug\res\layout\main.xml 10/29/2018 8:19:36 PM > 1/1/0001 12:00:00 AM
Processing: obj\Debug\res\layout\tabbar.xml 10/29/2018 8:19:36 PM > 1/1/0001 12:00:00 AM
Processing: obj\Debug\res\layout\toolbar.xml 10/29/2018 8:19:36 PM > 1/1/0001 12:00:00 AM
Processing: obj\Debug\res\values\colors.xml 10/29/2018 8:19:36 PM > 1/1/0001 12:00:00 AM
Processing: obj\Debug\res\values\strings.xml 10/29/2018 8:19:36 PM > 1/1/0001 12:00:00 AM
Processing: obj\Debug\res\values\styles.xml 10/29/2018 8:19:36 PM > 1/1/0001 12:00:00 AM
*/
Assert.AreEqual (6, processed, "A total of 6 layouts should be processed.");

/*
Skipping: `obj\Debug\lp\5\jl\res` via `SkipAndroidResourceProcessing`, original file: `bin\TestDebug\temp\packages\Xamarin.Android.Support.Compat.27.0.2.1\lib\MonoAndroid81\Xamarin.Android.Support.Compat.dll`...
Skipping: `obj\Debug\lp\8\jl\res` via `SkipAndroidResourceProcessing`, original file: `bin\TestDebug\temp\packages\Xamarin.Android.Support.Design.27.0.2.1\lib\MonoAndroid81\Xamarin.Android.Support.Design.dll`...
Skipping: `obj\Debug\lp\10\jl\res` via `SkipAndroidResourceProcessing`, original file: `bin\TestDebug\temp\packages\Xamarin.Android.Support.Media.Compat.27.0.2.1\lib\MonoAndroid81\Xamarin.Android.Support.Media.Compat.dll`...
Skipping: `obj\Debug\lp\11\jl\res` via `SkipAndroidResourceProcessing`, original file: `bin\TestDebug\temp\packages\Xamarin.Android.Support.v7.AppCompat.27.0.2.1\lib\MonoAndroid81\Xamarin.Android.Support.v7.AppCompat.dll`...
Skipping: `obj\Debug\lp\12\jl\res` via `SkipAndroidResourceProcessing`, original file: `bin\TestDebug\temp\packages\Xamarin.Android.Support.v7.CardView.27.0.2.1\lib\MonoAndroid81\Xamarin.Android.Support.v7.CardView.dll`...
Skipping: `obj\Debug\lp\13\jl\res` via `SkipAndroidResourceProcessing`, original file: `bin\TestDebug\temp\packages\Xamarin.Android.Support.v7.MediaRouter.27.0.2.1\lib\MonoAndroid81\Xamarin.Android.Support.v7.MediaRouter.dll`...
Skipping: `obj\Debug\lp\14\jl\res` via `SkipAndroidResourceProcessing`, original file: `bin\TestDebug\temp\packages\Xamarin.Android.Support.v7.RecyclerView.27.0.2.1\lib\MonoAndroid81\Xamarin.Android.Support.v7.RecyclerView.dll`...
Skipping: `obj\Debug\lp\20\jl\res` via `SkipAndroidResourceProcessing`, original file: `bin\TestDebug\temp\SkipConvertResourcesCases\Jars\material-menu-1.1.0.aar`...
*/
Assert.AreEqual (8, skipped, "A total of 8 resource directories should be skipped.");
}
}

[Test]
public void BuildInParallel ()
{
Expand Down
20 changes: 16 additions & 4 deletions src/Xamarin.Android.Build.Tasks/Utilities/XDocumentExtensions.cs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,16 +10,28 @@ namespace Xamarin.Android.Tasks
{
public static class XDocumentExtensions
{
const string PathsElementName = "Paths";

public static ITaskItem[] GetPathsAsTaskItems (this XDocument doc, params string[] paths)
{
return doc.GetPaths (paths)
.Select(x => new TaskItem(x))
.ToArray ();
var e = doc.Elements (PathsElementName);
foreach (var p in paths)
e = e.Elements (p);
return e.Select (ToTaskItem).ToArray ();
}

static ITaskItem ToTaskItem (XElement element)
{
var taskItem = new TaskItem (element.Value);
foreach (var attribute in element.Attributes ()) {
taskItem.SetMetadata (attribute.Name.LocalName, attribute.Value);
}
return taskItem;
}

public static string[] GetPaths (this XDocument doc, params string[] paths)
{
var e = doc.Elements ("Paths");
var e = doc.Elements (PathsElementName);
foreach (var p in paths)
e = e.Elements (p);
return e.Select (p => p.Value).ToArray ();
Expand Down
3 changes: 3 additions & 0 deletions src/Xamarin.Android.Build.Tasks/Xamarin.Android.Build.Tasks.targets
View file
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,9 @@
<None Include="Xamarin.Android.VisualBasic.targets">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>
<None Include="Xamarin.Android.SkipCases.projitems">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>
</ItemGroup>
<ItemGroup>
<_SharedRuntimeAssemblies Include="@(MonoProfileAssembly->'$(_SharedRuntimeBuildPath)v1.0\%(Identity)')" />
Expand Down
11 changes: 10 additions & 1 deletion src/Xamarin.Android.Build.Tasks/Xamarin.Android.Common.targets
View file
Original file line number Diff line number Diff line change
Expand Up @@ -353,6 +353,14 @@ Copyright (C) 2011-2012 Xamarin. All rights reserved.
</AllowedReferenceRelatedFileExtensions>
</PropertyGroup>

<!--
*******************************************
Imports
*******************************************
-->
<!-- As we split up/refactor this file, put new imports here -->
<Import Project="$(MSBuildThisFileDirectory)Xamarin.Android.SkipCases.projitems" />

<Target Name="_SeparateAppExtensionReferences">
<CreateItem Include="@(ProjectReference)" PreserveExistingMetadata="true" Condition="'%(Identity)' != '' AND '%(ProjectReference.IsAppExtension)' == 'true'">
<Output ItemName="_AppExtensionReference" TaskParameter="Include" />
Expand Down Expand Up @@ -1316,7 +1324,8 @@ because xbuild doesn't support framework reference assemblies.
UseShortFileNames="$(UseShortFileNames)"
OutputDirectory="$(IntermediateOutputPath)"
AssemblyIdentityMapFile="$(_AndroidLibrayProjectAssemblyMapFile)"
OutputImportDirectory="$(_AndroidLibrayProjectIntermediatePath)">
OutputImportDirectory="$(_AndroidLibrayProjectIntermediatePath)"
AssembliesToSkipCases="@(_AndroidAssemblySkipCases)">
</ResolveLibraryProjectImports>
<Touch Files="$(_AndroidStampDirectory)_ResolveLibraryProjectImports.stamp" AlwaysCreate="True" />
</Target>
Expand Down
Loading

0 comments on commit 4de096d

Please sign in to comment.