Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[Xamarin.Android.Build.Tasks] generate R.java like Android Studio
Context: https://android.googlesource.com/platform/tools/base/+/refs/heads/master/build-system/builder/ Fixes: xamarin#2680 Fixes: xamarin#2836 The current behavior in the `_GenerateJavaDesignerForComponent` MSBuild target does the following: * For each library that has Android resources... (in parallel) * Run an instance of aapt/aapt2 to generate the `R.java` file for each library. * This actually creates an `R.java` file that contains *every* resource id for *every* library. These libraries are not using most of these ids. This has a few problems: * xamarin#2680 notes a problem where a file is locked on Windows during `_GenerateJavaDesignerForComponent`. Xamarin.Android.Common.targets(1541,2): The process cannot access the file 'C:\repos\msalnet\tests\devapps\XForms\XForms.Android\obj\Debug\90\lp\26\jl\manifest\AndroidManifest.xml' because it is being used by another process. at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) at System.IO.StreamWriter.CreateFile(String path, Boolean append, Boolean checkHost) at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding, Int32 bufferSize, Boolean checkHost) at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding) at Xamarin.Android.Tasks.ManifestDocument.Save(String filename) at Xamarin.Android.Tasks.Aapt.GenerateCommandLineCommands(String ManifestFile, String currentAbi, String currentResourceOutputFile) at Xamarin.Android.Tasks.Aapt.ProcessManifest(ITaskItem manifestFile) at System.Threading.Tasks.Parallel.<>c__DisplayClass30_0`2.<ForEachWorker>b__0(Int32 i) at System.Threading.Tasks.Parallel.<>c__DisplayClass17_0`1.<ForWorker>b__1() at System.Threading.Tasks.Task.InnerInvoke() at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask) at System.Threading.Tasks.Task.<>c__DisplayClass176_0.<ExecuteSelfReplicating>b__0(Object ) [C:\repos\msalnet\tests\devapps\XForms\XForms.Android\XForms.Android.csproj] * We are hugely contributing to the dex limit for fields. Apps contain exponentially more fields for each library with resources. An example from @PureWeen: 1> trouble writing output: Too many field references to fit in one dex file: 70468; max is 65536. * Quite a few instances of aapt/aapt2 startup on developer's machines: this pegs the CPU. We have had a few general complaints about it. Reviewing the source code for the Android gradle plugin, here is what they do: * Build the main app's "full" `R.txt` file. * For each library, load its `R.txt` file. * Map each resource in the library's `R.txt` back to the main app * Write a small `R.java` file for each library: containing *only* the lines from the `R.txt` and updated integer values from the main app `R.txt` file. Looking into this, we can do the exact same thing? We have the `R.txt` file one directory above where we extract resources for each library. We already had code parsing `R.txt` files I could repurpose, the only thing *new* is a `R.java` writer: a pretty simple port from java. The results are great! Before: 3173 ms _GenerateJavaDesignerForComponentAapt2 1 calls After: 20 ms GenerateLibraryResources 1 calls `_GenerateJavaDesignerForComponent` is now completely gone. This is a total savings of ~3 seconds on first build and incremental builds with library changes. To compare APKs, I used: $ ~/android-toolchain/sdk/tools/bin/apkanalyzer dex packages Xamarin.Forms_Performance_Integration-Before.apk | grep ^F Which omits a line for each field such as: F d 0 0 16 xamarin.forms_performance_integration.R$color int abc_background_cache_hint_selector_material_dark So then, before these changes: $ ~/android-toolchain/sdk/tools/bin/apkanalyzer dex packages Xamarin.Forms_Performance_Integration-Before.apk | grep ^F | wc -l 29681 After: $ ~/android-toolchain/sdk/tools/bin/apkanalyzer dex packages Xamarin.Forms_Performance_Integration-After.apk | grep ^F | wc -l 17210 12K less fields in a "Hello World" Xamarin.Forms app! Comparing file sizes seems good, too: $ zipinfo Xamarin.Forms_Performance_Integration-Before.apk | grep classes.dex -rw-rw-r-- 6.3 unx 3657872 b- defX 19-Mar-28 16:37 classes.dex $ zipinfo Xamarin.Forms_Performance_Integration-After.apk | grep classes.dex -rw-rw-r-- 6.3 unx 3533120 b- defX 19-Mar-28 16:20 classes.dex Dex file in the APK is ~120KB smaller. ~~ What if R.txt is missing? ~~ I found this was the case when the `<GetAdditionalResourcesFromAssemblies/>` MSBuild task runs. This is an old codepath that allowed old support libraries to work. In this case, a directory is created such as: * `obj\Debug\resourcecache\CF390EBB0064FDA00BB090E733D37E89` * `adil` * `assets` * `libs` * `res` * `AndroidManifest.xml` * `classes.jar` No `R.txt` file? Checking the zip files we download: $ for z in ~/.local/share/Xamarin/zips/*.zip; do zipinfo $z; done | grep R.txt # no results This actually makes sense, since the zip file contains the *actual resources*. To make this case work properly, we should just process the main app's `R.txt` file when no library `R.txt` file is found. This will still be faster than invoking `aapt`, even though we have more fields than needed. ~~ Tests ~~ I added a set of unit tests for the `<GenerateLibraryResources/>` MSBuild task. I also had to remove a few assertions that looked for the `_GenerateJavaDesignerForComponent` MSBuild target. Lastly, I added some assertions to a test that uses an old support library to verify it's main `R.java` reasonably matches the library `R.java` we generate.
- Loading branch information
1 parent
2660230
commit 22d43cb
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -138,32 +138,4 @@ Copyright (C) 2011-2012 Xamarin. All rights reserved. | |
| </ItemGroup> | ||
| </Target> | ||
|
|
||
| </Project> | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,234 @@ | ||
| using Microsoft.Build.Framework; | ||
| using System; | ||
| using System.Collections.Generic; | ||
| using System.IO; | ||
| using System.Linq; | ||
| using System.Text; | ||
| using Xamarin.Android.Tools; | ||
| using Xamarin.Build; | ||
|
|
||
| namespace Xamarin.Android.Tasks | ||
| { | ||
| /// <summary> | ||
| /// We used to invoke aapt/aapt2 per library (many times!), this task does the work to generate R.java for libraries without calling aapt/aapt2. | ||
| /// </summary> | ||
| public class GenerateLibraryResources : AsyncTask | ||
| { | ||
| /// <summary> | ||
| /// The main R.txt for the app | ||
| /// </summary> | ||
| [Required] | ||
| public string ResourceSymbolsTextFile { get; set; } | ||
|
|
||
| /// <summary> | ||
| /// The output directory for Java source code, such as: $(IntermediateOutputPath)android\src | ||
| /// </summary> | ||
| [Required] | ||
| public string OutputDirectory { get; set; } | ||
|
|
||
| /// <summary> | ||
| /// The list of R.txt files for each library | ||
| /// </summary> | ||
| public string [] LibraryTextFiles { get; set; } | ||
|
|
||
| /// <summary> | ||
| /// The accompanying manifest file for each library | ||
| /// </summary> | ||
| public string [] ManifestFiles { get; set; } | ||
|
|
||
| public override bool Execute () | ||
| { | ||
| Yield (); | ||
| try { | ||
| this.RunTask (DoExecute).ContinueWith (Complete); | ||
|
|
||
| base.Execute (); | ||
| } finally { | ||
| Reacquire (); | ||
| } | ||
|
|
||
| return !Log.HasLoggedErrors; | ||
| } | ||
|
|
||
| string main_r_txt; | ||
| string output_directory; | ||
| Dictionary<Tuple<string, string>, string> r_txt_mapping; | ||
|
|
||
| void DoExecute () | ||
| { | ||
| if (LibraryTextFiles == null || LibraryTextFiles.Length == 0) | ||
| return; | ||
|
|
||
| // Load the "main" R.txt file into a dictionary | ||
| main_r_txt = Path.GetFullPath (ResourceSymbolsTextFile); | ||
| r_txt_mapping = new Dictionary<Tuple<string, string>, string> (); | ||
| using (var reader = File.OpenText (main_r_txt)) { | ||
| foreach (var line in ParseFile (reader)) { | ||
| var key = new Tuple<string, string> (line [Index.Class], line [Index.Name]); | ||
| r_txt_mapping [key] = line [Index.Value]; | ||
| } | ||
| } | ||
|
|
||
| Directory.CreateDirectory (OutputDirectory); | ||
| output_directory = Path.GetFullPath (OutputDirectory); | ||
|
|
||
| var libraries = LibraryTextFiles.Zip (ManifestFiles, (textFile, manifestFile) => new Library (textFile, manifestFile)); | ||
| this.ParallelForEach (libraries, GenerateJava); | ||
| } | ||
|
|
||
| /// <summary> | ||
| /// A quick class to combine the paths to R.txt and Manifest | ||
| /// </summary> | ||
| class Library | ||
| { | ||
| public Library (string textFile, string manifestFile) | ||
| { | ||
| TextFile = Path.GetFullPath (textFile); | ||
| ManifestFile = Path.GetFullPath (manifestFile); | ||
| } | ||
|
|
||
| /// <summary> | ||
| /// A full path to the R.txt file | ||
| /// </summary> | ||
| public string TextFile { get; } | ||
|
|
||
| /// <summary> | ||
| /// A full path to the AndroidManifest.xml file | ||
| /// </summary> | ||
| public string ManifestFile { get; } | ||
| } | ||
|
|
||
| /// <summary> | ||
| /// NOTE: all file paths used in this method should be full paths. (Or use AsyncTask.WorkingDirectory) | ||
| /// </summary> | ||
| void GenerateJava (Library library) | ||
| { | ||
| // In some cases (such as ancient support libraries), R.txt does not exist. | ||
| // We can just use the main app's R.txt file and write *all fields* in this case. | ||
| bool using_main_r_txt = false; | ||
| var r_txt = library.TextFile; | ||
| if (!File.Exists (r_txt)) { | ||
| LogDebugMessage ($"Using main R.txt, R.txt does not exist: {r_txt}"); | ||
| using_main_r_txt = true; | ||
| r_txt = main_r_txt; | ||
| } | ||
|
|
||
| var manifestFile = library.ManifestFile; | ||
| if (!File.Exists (manifestFile)) { | ||
| LogDebugMessage ($"Skipping, AndroidManifest.xml does not exist: {manifestFile}"); | ||
| return; | ||
| } | ||
|
|
||
| var manifest = AndroidAppManifest.Load (manifestFile, MonoAndroidHelper.SupportedVersions); | ||
|
|
||
| using (var memory = new MemoryStream ()) | ||
| using (var writer = new StreamWriter (memory, Encoding)) { | ||
| // This code is based on the Android gradle plugin | ||
| // https://android.googlesource.com/platform/tools/base/+/908b391a9c006af569dfaff08b37f8fdd6c4da89/build-system/builder/src/main/java/com/android/builder/internal/SymbolWriter.java | ||
|
|
||
| writer.WriteLine ("/* AUTO-GENERATED FILE. DO NOT MODIFY."); | ||
| writer.WriteLine (" *"); | ||
| writer.WriteLine (" * This class was automatically generated by"); | ||
| writer.WriteLine (" * Xamarin.Android from the resource data it found."); | ||
| writer.WriteLine (" * It should not be modified by hand."); | ||
| writer.WriteLine (" */"); | ||
|
|
||
| writer.Write ("package "); | ||
| writer.Write (manifest.PackageName); | ||
| writer.WriteLine (';'); | ||
| writer.WriteLine (); | ||
| writer.WriteLine ("public final class R {"); | ||
|
|
||
| using (var reader = File.OpenText (r_txt)) { | ||
| string currentClass = null; | ||
| foreach (var line in ParseFile (reader)) { | ||
| var type = line [Index.Type]; | ||
| var clazz = line [Index.Class]; | ||
| var name = line [Index.Name]; | ||
| if (GetValue (clazz, name, line, using_main_r_txt, out string value)) { | ||
| if (clazz != currentClass) { | ||
| // If not the first inner class | ||
| if (currentClass != null) { | ||
| writer.WriteLine ("\t}"); | ||
| } | ||
|
|
||
| currentClass = clazz; | ||
| writer.Write ("\tpublic static final class "); | ||
| writer.Write (currentClass); | ||
| writer.WriteLine (" {"); | ||
| } | ||
|
|
||
| writer.Write ("\t\tpublic static final "); | ||
| writer.Write (type); | ||
| writer.Write (' '); | ||
| writer.Write (name); | ||
| writer.Write (" = "); | ||
| // It may be an int[] | ||
| if (value.StartsWith ("{", StringComparison.Ordinal)) { | ||
| writer.Write ("new "); | ||
| writer.Write (type); | ||
| writer.Write (' '); | ||
| } | ||
| writer.Write (value); | ||
| writer.WriteLine (';'); | ||
| } else { | ||
| LogDebugMessage ($"{r_txt}: `{type} {clazz} {name}` value not found"); | ||
This comment has been minimized.
Sorry, something went wrong.
This comment has been minimized.
Sorry, something went wrong. 
jonathanpeppers
Author
Owner
|
||
| } | ||
| } | ||
|
|
||
| // If we wrote at least one inner class | ||
| if (currentClass != null) { | ||
| writer.WriteLine ("\t}"); | ||
| } | ||
| writer.WriteLine ('}'); | ||
| } | ||
|
|
||
| writer.Flush (); | ||
| var r_java = Path.Combine (output_directory, manifest.PackageName.Replace ('.', Path.DirectorySeparatorChar), "R.java"); | ||
| if (MonoAndroidHelper.CopyIfStreamChanged (memory, r_java)) { | ||
| LogDebugMessage ($"Writing: {r_java}"); | ||
| } else { | ||
| LogDebugMessage ($"Up to date: {r_java}"); | ||
| } | ||
| } | ||
| } | ||
|
|
||
| bool GetValue (string clazz, string name, string[] line, bool using_main_r_txt, out string value) | ||
| { | ||
| // If this is the main R.txt file, we don't need to do a lookup | ||
| if (using_main_r_txt) { | ||
| value = line [Index.Value]; | ||
| return true; | ||
| } | ||
|
|
||
| var key = new Tuple<string, string> (clazz, name); | ||
| return r_txt_mapping.TryGetValue (key, out value); | ||
| } | ||
|
|
||
| static readonly Encoding Encoding = new UTF8Encoding (encoderShouldEmitUTF8Identifier: false); | ||
| static readonly char [] Delimiter = new [] { ' ' }; | ||
|
|
||
| class Index | ||
| { | ||
| public const int Type = 0; | ||
| public const int Class = 1; | ||
| public const int Name = 2; | ||
| public const int Value = 3; | ||
| } | ||
|
|
||
| /// <summary> | ||
| /// R.txt is of the format: | ||
| /// int id icon 0x7f0c000a | ||
| /// int[] styleable ViewStubCompat { 0x010100d0, 0x010100f2, 0x010100f3 } | ||
| /// This returns a 4-length string[] of the parts. | ||
| /// </summary> | ||
| IEnumerable<string []> ParseFile (StreamReader reader) | ||
| { | ||
| while (!reader.EndOfStream) { | ||
| var line = reader.ReadLine (); | ||
| var items = line.Split (Delimiter, 4); | ||
| yield return items; | ||
| } | ||
| } | ||
| } | ||
| } | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
@jonathanpeppers I'm getting a lot of these entries in my build logs. What could it mean?