Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Android build cannot access AndroidManifest.xml because file is in use #2680

Closed
MarkZuber opened this issue Feb 1, 2019 · 7 comments
Closed
Assignees
Labels
Area: App+Library Build Issues when building Library projects or Application projects. need-info Issues that need more information from the author.
Milestone

Comments

@MarkZuber
Copy link

Steps to Reproduce

  1. Clone Update nuget packages, MonoAndroid9, Xamarin 28, XForms 3.x AzureAD/microsoft-authentication-library-for-dotnet#838 and build within VS2017 using LibsAndSamples.sln
  2. Rebuild All from within VS

Expected Behavior

Build completes successfully

Actual Behavior

------ Rebuild All started: Project: XForms.Android, Configuration: Debug Any CPU ------
C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\Xamarin\Android\Xamarin.Android.Common.targets(1541,2): error : The process cannot access the file 'C:\repos\msalnet\tests\devapps\XForms\XForms.Android\obj\Debug\90\lp\21\jl\manifest\AndroidManifest.xml' because it is being used by another process.
C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\Xamarin\Android\Xamarin.Android.Common.targets(1541,2): error : at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\Xamarin\Android\Xamarin.Android.Common.targets(1541,2): error : at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\Xamarin\Android\Xamarin.Android.Common.targets(1541,2): error : at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\Xamarin\Android\Xamarin.Android.Common.targets(1541,2): error : at System.IO.StreamWriter.CreateFile(String path, Boolean append, Boolean checkHost)
C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\Xamarin\Android\Xamarin.Android.Common.targets(1541,2): error : at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding, Int32 bufferSize, Boolean checkHost)
C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\Xamarin\Android\Xamarin.Android.Common.targets(1541,2): error : at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding)
C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\Xamarin\Android\Xamarin.Android.Common.targets(1541,2): error : at Xamarin.Android.Tasks.ManifestDocument.Save(String filename)
C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\Xamarin\Android\Xamarin.Android.Common.targets(1541,2): error : at Xamarin.Android.Tasks.Aapt.GenerateCommandLineCommands(String ManifestFile, String currentAbi, String currentResourceOutputFile)
C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\Xamarin\Android\Xamarin.Android.Common.targets(1541,2): error : at Xamarin.Android.Tasks.Aapt.ProcessManifest(ITaskItem manifestFile)
C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\Xamarin\Android\Xamarin.Android.Common.targets(1541,2): error : at System.Threading.Tasks.Parallel.<>c__DisplayClass30_02.<ForEachWorker>b__0(Int32 i) C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\Xamarin\Android\Xamarin.Android.Common.targets(1541,2): error : at System.Threading.Tasks.Parallel.<>c__DisplayClass17_01.b__1()
C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\Xamarin\Android\Xamarin.Android.Common.targets(1541,2): error : at System.Threading.Tasks.Task.InnerInvoke()
C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\Xamarin\Android\Xamarin.Android.Common.targets(1541,2): error : at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)
C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\Xamarin\Android\Xamarin.Android.Common.targets(1541,2): error : at System.Threading.Tasks.Task.<>c__DisplayClass176_0.b__0(Object )

Version Information

Microsoft Visual Studio Enterprise 2017
Version 15.9.6
VisualStudio.15.Release/15.9.6+28307.344
Microsoft .NET Framework
Version 4.7.03190

Installed Version: Enterprise

Visual C++ 2017 00369-60000-00001-AA587
Microsoft Visual C++ 2017

ADL Tools Service Provider 1.0
This package contains services used by Data Lake tools

Application Insights Tools for Visual Studio Package 8.14.11009.1
Application Insights Tools for Visual Studio

ASP.NET and Web Tools 2017 15.9.04012.0
ASP.NET and Web Tools 2017

ASP.NET Core Razor Language Services 15.8.31590
Provides languages services for ASP.NET Core Razor.

ASP.NET Web Frameworks and Tools 2017 5.2.60913.0
For additional information, visit https://www.asp.net/

Azure App Service Tools v3.0.0 15.9.03024.0
Azure App Service Tools v3.0.0

Azure Data Lake Node 1.0
This package contains the Data Lake integration nodes for Server Explorer.

Azure Data Lake Tools for Visual Studio 2.3.7000.0
Microsoft Azure Data Lake Tools for Visual Studio

Azure Functions and Web Jobs Tools 15.9.02046.0
Azure Functions and Web Jobs Tools

Azure Stream Analytics Tools for Visual Studio 2.3.7000.0
Microsoft Azure Stream Analytics Tools for Visual Studio

C# Tools 2.10.0-beta2-63501-03+b9fb1610c87cccc8ceb74a770dba261a58e39c4a
C# components used in the IDE. Depending on your project type and settings, a different version of the compiler may be used.

Common Azure Tools 1.10
Provides common services for use by Azure Mobile Services and Microsoft Azure Tools.

Cookiecutter 15.9.18254.1
Provides tools for finding, instantiating and customizing templates in cookiecutter format.

Extensibility Message Bus 1.1.49 (remotes/origin/d15-8@ee674f3)
Provides common messaging-based MEF services for loosely coupled Visual Studio extension components communication and integration.

Fabric.DiagnosticEvents 1.0
Fabric Diagnostic Events

GitHub.VisualStudio 2.7.1.6591
A Visual Studio Extension that brings the GitHub Flow into Visual Studio.

JavaScript Language Service 2.0
JavaScript Language Service

JavaScript Project System 2.0
JavaScript Project System

JavaScript UWP Project System 2.0
JavaScript UWP Project System

JetBrains ReSharper Ultimate 2018.2.3 Build 182.0.20180912.70621
JetBrains ReSharper Ultimate package for Microsoft Visual Studio. For more information about ReSharper Ultimate, visit http://www.jetbrains.com/resharper. Copyright © 2019 JetBrains, Inc.

Microsoft Azure HDInsight Azure Node 2.3.7000.0
HDInsight Node under Azure Node

Microsoft Azure Hive Query Language Service 2.3.7000.0
Language service for Hive query

Microsoft Azure Service Fabric Tools for Visual Studio 2.4
Microsoft Azure Service Fabric Tools for Visual Studio

Microsoft Azure Stream Analytics Language Service 2.3.7000.0
Language service for Azure Stream Analytics

Microsoft Azure Stream Analytics Node 1.0
Azure Stream Analytics Node under Azure Node

Microsoft Azure Tools 2.9
Microsoft Azure Tools for Microsoft Visual Studio 2017 - v2.9.10730.2

Microsoft Continuous Delivery Tools for Visual Studio 0.4
Simplifying the configuration of Azure DevOps pipelines from within the Visual Studio IDE.

Microsoft JVM Debugger 1.0
Provides support for connecting the Visual Studio debugger to JDWP compatible Java Virtual Machines

Microsoft Library Manager 1.0
Install client-side libraries easily to any web project

Microsoft MI-Based Debugger 1.0
Provides support for connecting Visual Studio to MI compatible debuggers

Microsoft Visual C++ Wizards 1.0
Microsoft Visual C++ Wizards

Microsoft Visual Studio Tools for Containers 1.1
Develop, run, validate your ASP.NET Core applications in the target environment. F5 your application directly into a container with debugging, or CTRL + F5 to edit & refresh your app without having to rebuild the container.

Microsoft Visual Studio VC Package 1.0
Microsoft Visual Studio VC Package

MLGen Package Extension 1.0
MLGen Package Visual Studio Extension Detailed Info

Mono Debugging for Visual Studio 4.13.12-pre (9bc9548)
Support for debugging Mono processes with Visual Studio.

Node.js Tools 1.4.21001.1 Commit Hash:8dd15923800d931b153ab9e4de74e42a74eba5e6
Adds support for developing and debugging Node.js apps in Visual Studio

NuGet Package Manager 4.6.0
NuGet Package Manager in Visual Studio. For more information about NuGet, visit http://docs.nuget.org/.

ProjectServicesPackage Extension 1.0
ProjectServicesPackage Visual Studio Extension Detailed Info

Python 15.9.18254.1
Provides IntelliSense, projects, templates, debugging, interactive windows, and other support for Python developers.

Python - Django support 15.9.18254.1
Provides templates and integration for the Django web framework.

Python - IronPython support 15.9.18254.1
Provides templates and integration for IronPython-based projects.

Python - Profiling support 15.9.18254.1
Profiling support for Python projects.

ResourcePackage Extension 1.0
ResourcePackage Visual Studio Extension Detailed Info

ResourcePackage Extension 1.0
ResourcePackage Visual Studio Extension Detailed Info

Snapshot Debugging Extension 1.0
Snapshot Debugging Visual Studio Extension Detailed Info

SQL Server Data Tools 15.1.61901.03220
Microsoft SQL Server Data Tools

Test Adapter for Boost.Test 1.0
Enables Visual Studio's testing tools with unit tests written for Boost.Test. The use terms and Third Party Notices are available in the extension installation directory.

Test Adapter for Google Test 1.0
Enables Visual Studio's testing tools with unit tests written for Google Test. The use terms and Third Party Notices are available in the extension installation directory.

ToolWindowHostedEditor 1.0
Hosting json editor into a tool window

TypeScript Tools 15.9.20918.2001
TypeScript Tools for Microsoft Visual Studio

Visual Basic Tools 2.10.0-beta2-63501-03+b9fb1610c87cccc8ceb74a770dba261a58e39c4a
Visual Basic components used in the IDE. Depending on your project type and settings, a different version of the compiler may be used.

Visual C++ for Linux Development 1.0.9.28218
Visual C++ for Linux Development

Visual F# Tools 10.2 for F# 4.5 15.8.0.0. Commit Hash: 6e26c5bacc8c4201e962f5bdde0a177f82f88691.
Microsoft Visual F# Tools 10.2 for F# 4.5

Visual Studio Code Debug Adapter Host Package 1.0
Interop layer for hosting Visual Studio Code debug adapters in Visual Studio

Visual Studio Tools for CMake 1.0
Visual Studio Tools for CMake

Visual Studio Tools for Containers 1.0
Visual Studio Tools for Containers

Visual Studio Tools for Universal Windows Apps 15.0.28307.208
The Visual Studio Tools for Universal Windows apps allow you to build a single universal app experience that can reach every device running Windows 10: phone, tablet, PC, and more. It includes the Microsoft Windows 10 Software Development Kit.

VisualStudio.Mac 1.0
Mac Extension for Visual Studio

Xamarin 4.12.3.79 (d15-9@260fa6a34)
Visual Studio extension to enable development for Xamarin.iOS and Xamarin.Android.

Xamarin Designer 4.16.13 (45a16efd4)
Visual Studio extension to enable Xamarin Designer tools in Visual Studio.

Xamarin Templates 1.1.128 (6f5ebb2)
Templates for building iOS, Android, and Windows apps with Xamarin and Xamarin.Forms.

Xamarin.Android SDK 9.1.5.0 (HEAD/4b951a3e7)
Xamarin.Android Reference Assemblies and MSBuild support.

Xamarin.iOS and Xamarin.Mac SDK 12.2.1.12 (65ec520)
Xamarin.iOS and Xamarin.Mac Reference Assemblies and MSBuild support.

Log File

------ Rebuild All started: Project: XForms.Android, Configuration: Debug Any CPU ------
C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\Xamarin\Android\Xamarin.Android.Common.targets(1541,2): error : The process cannot access the file 'C:\repos\msalnet\tests\devapps\XForms\XForms.Android\obj\Debug\90\lp\21\jl\manifest\AndroidManifest.xml' because it is being used by another process.
C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\Xamarin\Android\Xamarin.Android.Common.targets(1541,2): error : at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\Xamarin\Android\Xamarin.Android.Common.targets(1541,2): error : at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\Xamarin\Android\Xamarin.Android.Common.targets(1541,2): error : at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\Xamarin\Android\Xamarin.Android.Common.targets(1541,2): error : at System.IO.StreamWriter.CreateFile(String path, Boolean append, Boolean checkHost)
C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\Xamarin\Android\Xamarin.Android.Common.targets(1541,2): error : at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding, Int32 bufferSize, Boolean checkHost)
C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\Xamarin\Android\Xamarin.Android.Common.targets(1541,2): error : at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding)
C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\Xamarin\Android\Xamarin.Android.Common.targets(1541,2): error : at Xamarin.Android.Tasks.ManifestDocument.Save(String filename)
C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\Xamarin\Android\Xamarin.Android.Common.targets(1541,2): error : at Xamarin.Android.Tasks.Aapt.GenerateCommandLineCommands(String ManifestFile, String currentAbi, String currentResourceOutputFile)
C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\Xamarin\Android\Xamarin.Android.Common.targets(1541,2): error : at Xamarin.Android.Tasks.Aapt.ProcessManifest(ITaskItem manifestFile)
C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\Xamarin\Android\Xamarin.Android.Common.targets(1541,2): error : at System.Threading.Tasks.Parallel.<>c__DisplayClass30_02.<ForEachWorker>b__0(Int32 i) C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\Xamarin\Android\Xamarin.Android.Common.targets(1541,2): error : at System.Threading.Tasks.Parallel.<>c__DisplayClass17_01.b__1()
C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\Xamarin\Android\Xamarin.Android.Common.targets(1541,2): error : at System.Threading.Tasks.Task.InnerInvoke()
C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\Xamarin\Android\Xamarin.Android.Common.targets(1541,2): error : at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)
C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\Xamarin\Android\Xamarin.Android.Common.targets(1541,2): error : at System.Threading.Tasks.Task.<>c__DisplayClass176_0.b__0(Object )

@jonathanpeppers
Copy link
Member

@MarkZuber I saw an issue about this project internally, and I think the problem is the use of the /m switch in the build script: https://github.com/AzureAD/microsoft-authentication-library-for-dotnet/blob/591050231dbc0ec498af4bac98cebf715aea82d6/buildVS2017.cmd#L27-L31
(this tells MSBuild to build everything in parallel)

I saw a similar error in a non-Xamarin.Android project when I built command line:

"C:\src\msal\LibsNoSamples.sln" (build target) (1) ->
       "C:\src\msal\src\Microsoft.Identity.Client.Ref\Microsoft.Identity.Client.Ref.csproj" (default target) (7) ->
       "C:\src\msal\src\Microsoft.Identity.Client.Ref\Microsoft.Identity.Client.Ref.csproj" (Build target) (7:30) ->
       (GenerateSourceLinkFile target) ->
         C:\Users\jopepper\.nuget\packages\microsoft.sourcelink.common\1.0.0-beta2-18618-05\build\Microsoft.SourceLink.Common.targets(53,5): error : Error writing to source link file 'obj\Debug\xamarinmac20\Microsoft.Identity.Client.Ref.sourcelink.json'
       : The process cannot access the file 'C:\src\msal\src\Microsoft.Identity.Client.Ref\obj\Debug\xamarinmac20\Microsoft.Identity.Client.Ref.sourcelink.json' because it is being used by another process. [C:\src\msal\src\Microsoft.Identity.Client.Ref\
       Microsoft.Identity.Client.Ref.csproj]

But it looks like your log might be coming from Visual Studio? I'll try it in the IDE.

@jonathanpeppers jonathanpeppers self-assigned this Feb 5, 2019
@jonathanpeppers jonathanpeppers added the Area: App+Library Build Issues when building Library projects or Application projects. label Feb 5, 2019
@jonathanpeppers jonathanpeppers added this to the d16-1 milestone Feb 5, 2019
@jonathanpeppers
Copy link
Member

It is working inside the IDE for me, were you seeing the failure command line?

@jonathanpeppers jonathanpeppers added the need-info Issues that need more information from the author. label Feb 5, 2019
@MarkZuber
Copy link
Author

I was seeing this intermittently within the IDE.

https://identitydivision.visualstudio.com/IDDP/_build/results?buildId=284384. our CI build (as we're trying to work on getting this merged) just failed again with this issue. I can get you access to our VSTS CI system if you need me to.

@MarkZuber
Copy link
Author

Our build has the main assembly (Microsoft.Identity.Client.dll) and we also build a reference assembly (Microsoft.Identity.Client.ref.dll) for each platform (see https://oren.codes/2018/07/03/create-and-pack-reference-assemblies/).

It looks like if these run at the same time, we can hit this.

We also have a base project, XForms.csproj, which contains our core logic for Xamarin sample apps. We then have XForms.Android.csproj and XForms.ios.csproj which depend on this.

We are able to work around this by setting explicit build dependencies in our solution to ensure that only one of these builds at a time. Even setting "don't use parallel builds" doesn't seem to be enough.

So, we are able to mitigate this issue, but it would be great if we didn't have to do it manually.

@jonathanpeppers
Copy link
Member

@MarkZuber so I'm confused how this is happening in your project, if I look at:

https://github.com/AzureAD/microsoft-authentication-library-for-dotnet/blob/591050231dbc0ec498af4bac98cebf715aea82d6/tests/devapps/XForms/XForms.Android/XForms.Android.csproj

From the stacktrace, it looks like the <Aapt/> MSBuild task was running concurrently. It writes a file in obj, and two instances of this task tried to do that at once.

Is your build, somehow building XForms.Android.csproj multiple times concurrently?

@MarkZuber
Copy link
Author

Hi @jonathanpeppers thanks for your response. We're just building via the solution so I don't understand why it would build a leaf node project multiple times. But I have seen cases where the dependent projects (e.g. XForms.csproj, or the ref assemblies) get built multiple times because VS/solution doesn't think they're up to date. I'll dig into our SLN again to see if there's something else weird there that's referencing it twice in the same build configuration somehow.

@trwalke
Copy link

trwalke commented Mar 5, 2019

@jonathanpeppers Any updates on this? our Azure devops build has now started to fail with the same error. I specifically set '-maxcpucount:1' to disable parallel builds.

jonathanpeppers added a commit to jonathanpeppers/xamarin-android that referenced this issue Mar 29, 2019
Context: https://android.googlesource.com/platform/tools/base/+/refs/heads/master/build-system/builder/
Fixes: xamarin#2680
Fixes: xamarin#2836

The current behavior in the `_GenerateJavaDesignerForComponent`
MSBuild target does the following:

* For each library that has Android resources... (in parallel)
* Run an instance of aapt/aapt2 to generate the `R.java` file for each
  library.
* This actually creates an `R.java` file that contains *every*
  resource id for *every* library. These libraries are not using most
  of these ids.

This has a few problems:

* xamarin#2680 notes a problem where a file is locked on Windows during
  `_GenerateJavaDesignerForComponent`.

    Xamarin.Android.Common.targets(1541,2): The process cannot access the file 'C:\repos\msalnet\tests\devapps\XForms\XForms.Android\obj\Debug\90\lp\26\jl\manifest\AndroidManifest.xml' because it is being used by another process.
        at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
        at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
        at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
        at System.IO.StreamWriter.CreateFile(String path, Boolean append, Boolean checkHost)
        at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding, Int32 bufferSize, Boolean checkHost)
        at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding)
        at Xamarin.Android.Tasks.ManifestDocument.Save(String filename)
        at Xamarin.Android.Tasks.Aapt.GenerateCommandLineCommands(String ManifestFile, String currentAbi, String currentResourceOutputFile)
        at Xamarin.Android.Tasks.Aapt.ProcessManifest(ITaskItem manifestFile)
        at System.Threading.Tasks.Parallel.<>c__DisplayClass30_0`2.<ForEachWorker>b__0(Int32 i)
        at System.Threading.Tasks.Parallel.<>c__DisplayClass17_0`1.<ForWorker>b__1()
        at System.Threading.Tasks.Task.InnerInvoke()
        at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)
        at System.Threading.Tasks.Task.<>c__DisplayClass176_0.<ExecuteSelfReplicating>b__0(Object ) [C:\repos\msalnet\tests\devapps\XForms\XForms.Android\XForms.Android.csproj]

* We are hugely contributing to the dex limit for fields. Apps contain
  exponentially more fields for each library with resources.

An example from @PureWeen:

    1>  trouble writing output: Too many field references to fit in one dex file: 70468; max is 65536.

* Quite a few instances of aapt/aapt2 startup on developer's machines:
  this pegs the CPU. We have had a few general complaints about it.

Reviewing the source code for the Android gradle plugin, here is what
they do:

* Build the main app's "full" `R.txt` file.
* For each library, load its `R.txt` file.
* Map each resource in the library's `R.txt` back to the main app
* Write a small `R.java` file for each library: containing *only* the
  lines from the `R.txt` and updated integer values from the main app
  `R.txt` file.

Looking into this, we can do the exact same thing? We have the `R.txt`
file one directory above where we extract resources for each library.
We already had code parsing `R.txt` files I could repurpose, the only
thing *new* is a `R.java` writer: a pretty simple port from java.

The results are great!

    Before:
    3173 ms  _GenerateJavaDesignerForComponentAapt2     1 calls
    After:
      20 ms  GenerateLibraryResources                   1 calls

`_GenerateJavaDesignerForComponent` is now completely gone. This is a
total savings of ~3 seconds on first build and incremental builds
with library changes.

To compare APKs, I used:

    $ ~/android-toolchain/sdk/tools/bin/apkanalyzer dex packages Xamarin.Forms_Performance_Integration-Before.apk | grep ^F

Which omits a line for each field such as:

    F d 0	0	16	xamarin.forms_performance_integration.R$color int abc_background_cache_hint_selector_material_dark

So then, before these changes:

    $ ~/android-toolchain/sdk/tools/bin/apkanalyzer dex packages Xamarin.Forms_Performance_Integration-Before.apk | grep ^F | wc -l
    29681

After:

    $ ~/android-toolchain/sdk/tools/bin/apkanalyzer dex packages Xamarin.Forms_Performance_Integration-After.apk | grep ^F | wc -l
    17210

12K less fields in a "Hello World" Xamarin.Forms app!

Comparing file sizes seems good, too:

    $ zipinfo Xamarin.Forms_Performance_Integration-Before.apk | grep classes.dex
    -rw-rw-r--  6.3 unx  3657872 b- defX 19-Mar-28 16:37 classes.dex
    $ zipinfo Xamarin.Forms_Performance_Integration-After.apk | grep classes.dex
    -rw-rw-r--  6.3 unx  3533120 b- defX 19-Mar-28 16:20 classes.dex

Dex file in the APK is ~120KB smaller.
jonathanpeppers added a commit to jonathanpeppers/xamarin-android that referenced this issue Mar 29, 2019
Context: https://android.googlesource.com/platform/tools/base/+/refs/heads/master/build-system/builder/
Fixes: xamarin#2680
Fixes: xamarin#2836

The current behavior in the `_GenerateJavaDesignerForComponent`
MSBuild target does the following:

* For each library that has Android resources... (in parallel)
* Run an instance of aapt/aapt2 to generate the `R.java` file for each
  library.
* This actually creates an `R.java` file that contains *every*
  resource id for *every* library. These libraries are not using most
  of these ids.

This has a few problems:

* xamarin#2680 notes a problem where a file is locked on Windows during
  `_GenerateJavaDesignerForComponent`.

    Xamarin.Android.Common.targets(1541,2): The process cannot access the file 'C:\repos\msalnet\tests\devapps\XForms\XForms.Android\obj\Debug\90\lp\26\jl\manifest\AndroidManifest.xml' because it is being used by another process.
        at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
        at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
        at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
        at System.IO.StreamWriter.CreateFile(String path, Boolean append, Boolean checkHost)
        at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding, Int32 bufferSize, Boolean checkHost)
        at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding)
        at Xamarin.Android.Tasks.ManifestDocument.Save(String filename)
        at Xamarin.Android.Tasks.Aapt.GenerateCommandLineCommands(String ManifestFile, String currentAbi, String currentResourceOutputFile)
        at Xamarin.Android.Tasks.Aapt.ProcessManifest(ITaskItem manifestFile)
        at System.Threading.Tasks.Parallel.<>c__DisplayClass30_0`2.<ForEachWorker>b__0(Int32 i)
        at System.Threading.Tasks.Parallel.<>c__DisplayClass17_0`1.<ForWorker>b__1()
        at System.Threading.Tasks.Task.InnerInvoke()
        at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)
        at System.Threading.Tasks.Task.<>c__DisplayClass176_0.<ExecuteSelfReplicating>b__0(Object ) [C:\repos\msalnet\tests\devapps\XForms\XForms.Android\XForms.Android.csproj]

* We are hugely contributing to the dex limit for fields. Apps contain
  exponentially more fields for each library with resources.

An example from @PureWeen:

    1>  trouble writing output: Too many field references to fit in one dex file: 70468; max is 65536.

* Quite a few instances of aapt/aapt2 startup on developer's machines:
  this pegs the CPU. We have had a few general complaints about it.

Reviewing the source code for the Android gradle plugin, here is what
they do:

* Build the main app's "full" `R.txt` file.
* For each library, load its `R.txt` file.
* Map each resource in the library's `R.txt` back to the main app
* Write a small `R.java` file for each library: containing *only* the
  lines from the `R.txt` and updated integer values from the main app
  `R.txt` file.

Looking into this, we can do the exact same thing? We have the `R.txt`
file one directory above where we extract resources for each library.
We already had code parsing `R.txt` files I could repurpose, the only
thing *new* is a `R.java` writer: a pretty simple port from java.

The results are great!

    Before:
    3173 ms  _GenerateJavaDesignerForComponentAapt2     1 calls
    After:
      20 ms  GenerateLibraryResources                   1 calls

`_GenerateJavaDesignerForComponent` is now completely gone. This is a
total savings of ~3 seconds on first build and incremental builds
with library changes.

To compare APKs, I used:

    $ ~/android-toolchain/sdk/tools/bin/apkanalyzer dex packages Xamarin.Forms_Performance_Integration-Before.apk | grep ^F

Which omits a line for each field such as:

    F d 0	0	16	xamarin.forms_performance_integration.R$color int abc_background_cache_hint_selector_material_dark

So then, before these changes:

    $ ~/android-toolchain/sdk/tools/bin/apkanalyzer dex packages Xamarin.Forms_Performance_Integration-Before.apk | grep ^F | wc -l
    29681

After:

    $ ~/android-toolchain/sdk/tools/bin/apkanalyzer dex packages Xamarin.Forms_Performance_Integration-After.apk | grep ^F | wc -l
    17210

12K less fields in a "Hello World" Xamarin.Forms app!

Comparing file sizes seems good, too:

    $ zipinfo Xamarin.Forms_Performance_Integration-Before.apk | grep classes.dex
    -rw-rw-r--  6.3 unx  3657872 b- defX 19-Mar-28 16:37 classes.dex
    $ zipinfo Xamarin.Forms_Performance_Integration-After.apk | grep classes.dex
    -rw-rw-r--  6.3 unx  3533120 b- defX 19-Mar-28 16:20 classes.dex

Dex file in the APK is ~120KB smaller.

~~ What if R.txt is missing? ~~

I found this was the case when the
`<GetAdditionalResourcesFromAssemblies/>` MSBuild task runs. This is
an old codepath that allowed old support libraries to work.

In this case, a directory is created such as:

* `obj\Debug\resourcecache\CF390EBB0064FDA00BB090E733D37E89`
  * `adil`
  * `assets`
  * `libs`
  * `res`
  * `AndroidManifest.xml`
  * `classes.jar`

No `R.txt` file?

Checking the zip files we download:

    $ for z in ~/.local/share/Xamarin/zips/*.zip; do zipinfo $z; done | grep R.txt
    # no results

This actually makes sense, since the zip file contains the *actual
resources*.

To make this case work properly, we should just process the main app's
`R.txt` file when no library `R.txt` file is found. This will still be
faster than invoking `aapt`, even though we have more fields than
needed.

~~ Tests ~~

I added a set of unit tests for the `<GenerateLibraryResources/>`
MSBuild task. I also had to remove a few assertions that looked for
the `_GenerateJavaDesignerForComponent` MSBuild target.

Lastly, I added some assertions to a test that uses an old support
library to verify it's main `R.java` reasonably matches the library
`R.java` we generate.
jonathanpeppers added a commit to jonathanpeppers/xamarin-android that referenced this issue Apr 1, 2019
Context: https://android.googlesource.com/platform/tools/base/+/refs/heads/master/build-system/builder/
Fixes: xamarin#2680
Fixes: xamarin#2836

The current behavior in the `_GenerateJavaDesignerForComponent`
MSBuild target does the following:

* For each library that has Android resources... (in parallel)
* Run an instance of aapt/aapt2 to generate the `R.java` file for each
  library.
* This actually creates an `R.java` file that contains *every*
  resource id for *every* library. These libraries are not using most
  of these ids.

This has a few problems:

* xamarin#2680 notes a problem where a file is locked on Windows during
  `_GenerateJavaDesignerForComponent`.

    Xamarin.Android.Common.targets(1541,2): The process cannot access the file 'C:\repos\msalnet\tests\devapps\XForms\XForms.Android\obj\Debug\90\lp\26\jl\manifest\AndroidManifest.xml' because it is being used by another process.
        at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
        at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
        at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
        at System.IO.StreamWriter.CreateFile(String path, Boolean append, Boolean checkHost)
        at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding, Int32 bufferSize, Boolean checkHost)
        at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding)
        at Xamarin.Android.Tasks.ManifestDocument.Save(String filename)
        at Xamarin.Android.Tasks.Aapt.GenerateCommandLineCommands(String ManifestFile, String currentAbi, String currentResourceOutputFile)
        at Xamarin.Android.Tasks.Aapt.ProcessManifest(ITaskItem manifestFile)
        at System.Threading.Tasks.Parallel.<>c__DisplayClass30_0`2.<ForEachWorker>b__0(Int32 i)
        at System.Threading.Tasks.Parallel.<>c__DisplayClass17_0`1.<ForWorker>b__1()
        at System.Threading.Tasks.Task.InnerInvoke()
        at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)
        at System.Threading.Tasks.Task.<>c__DisplayClass176_0.<ExecuteSelfReplicating>b__0(Object ) [C:\repos\msalnet\tests\devapps\XForms\XForms.Android\XForms.Android.csproj]

* We are hugely contributing to the dex limit for fields. Apps contain
  exponentially more fields for each library with resources.

An example from @PureWeen:

    1>  trouble writing output: Too many field references to fit in one dex file: 70468; max is 65536.

* Quite a few instances of aapt/aapt2 startup on developer's machines:
  this pegs the CPU. We have had a few general complaints about it.

Reviewing the source code for the Android gradle plugin, here is what
they do:

* Build the main app's "full" `R.txt` file.
* For each library, load its `R.txt` file.
* Map each resource in the library's `R.txt` back to the main app
* Write a small `R.java` file for each library: containing *only* the
  lines from the `R.txt` and updated integer values from the main app
  `R.txt` file.

Looking into this, we can do the exact same thing? We have the `R.txt`
file one directory above where we extract resources for each library.
We already had code parsing `R.txt` files I could repurpose, the only
thing *new* is a `R.java` writer: a pretty simple port from java.

The results are great!

    Before:
    3173 ms  _GenerateJavaDesignerForComponentAapt2     1 calls
    After:
      20 ms  GenerateLibraryResources                   1 calls

`_GenerateJavaDesignerForComponent` is now completely gone. This is a
total savings of ~3 seconds on first build and incremental builds
with library changes.

To compare APKs, I used:

    $ ~/android-toolchain/sdk/tools/bin/apkanalyzer dex packages Xamarin.Forms_Performance_Integration-Before.apk | grep ^F

Which omits a line for each field such as:

    F d 0	0	16	xamarin.forms_performance_integration.R$color int abc_background_cache_hint_selector_material_dark

So then, before these changes:

    $ ~/android-toolchain/sdk/tools/bin/apkanalyzer dex packages Xamarin.Forms_Performance_Integration-Before.apk | grep ^F | wc -l
    29681

After:

    $ ~/android-toolchain/sdk/tools/bin/apkanalyzer dex packages Xamarin.Forms_Performance_Integration-After.apk | grep ^F | wc -l
    17210

12K less fields in a "Hello World" Xamarin.Forms app!

Comparing file sizes seems good, too:

    $ zipinfo Xamarin.Forms_Performance_Integration-Before.apk | grep classes.dex
    -rw-rw-r--  6.3 unx  3657872 b- defX 19-Mar-28 16:37 classes.dex
    $ zipinfo Xamarin.Forms_Performance_Integration-After.apk | grep classes.dex
    -rw-rw-r--  6.3 unx  3533120 b- defX 19-Mar-28 16:20 classes.dex

Dex file in the APK is ~120KB smaller.

~~ What if R.txt is missing? ~~

I found this was the case when the
`<GetAdditionalResourcesFromAssemblies/>` MSBuild task runs. This is
an old codepath that allowed old support libraries to work.

In this case, a directory is created such as:

* `obj\Debug\resourcecache\CF390EBB0064FDA00BB090E733D37E89`
  * `adil`
  * `assets`
  * `libs`
  * `res`
  * `AndroidManifest.xml`
  * `classes.jar`

No `R.txt` file?

Checking the zip files we download:

    $ for z in ~/.local/share/Xamarin/zips/*.zip; do zipinfo $z; done | grep R.txt
    # no results

This actually makes sense, since the zip file contains the *actual
resources*.

To make this case work properly, we should just process the main app's
`R.txt` file when no library `R.txt` file is found. This will still be
faster than invoking `aapt`, even though we have more fields than
needed.

~~ Tests ~~

I added a set of unit tests for the `<GenerateLibraryResources/>`
MSBuild task. I also had to remove a few assertions that looked for
the `_GenerateJavaDesignerForComponent` MSBuild target.

Lastly, I added some assertions to a test that uses an old support
library to verify it's main `R.java` reasonably matches the library
`R.java` we generate.
jonathanpeppers added a commit to jonathanpeppers/xamarin-android that referenced this issue Apr 1, 2019
Context: https://android.googlesource.com/platform/tools/base/+/refs/heads/master/build-system/builder/
Fixes: xamarin#2680
Fixes: xamarin#2836

The current behavior in the `_GenerateJavaDesignerForComponent`
MSBuild target does the following:

* For each library that has Android resources... (in parallel)
* Run an instance of aapt/aapt2 to generate the `R.java` file for each
  library.
* This actually creates an `R.java` file that contains *every*
  resource id for *every* library. These libraries are not using most
  of these ids.

This has a few problems:

* xamarin#2680 notes a problem where a file is locked on Windows during
  `_GenerateJavaDesignerForComponent`.

    Xamarin.Android.Common.targets(1541,2): The process cannot access the file 'C:\repos\msalnet\tests\devapps\XForms\XForms.Android\obj\Debug\90\lp\26\jl\manifest\AndroidManifest.xml' because it is being used by another process.
        at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
        at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
        at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
        at System.IO.StreamWriter.CreateFile(String path, Boolean append, Boolean checkHost)
        at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding, Int32 bufferSize, Boolean checkHost)
        at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding)
        at Xamarin.Android.Tasks.ManifestDocument.Save(String filename)
        at Xamarin.Android.Tasks.Aapt.GenerateCommandLineCommands(String ManifestFile, String currentAbi, String currentResourceOutputFile)
        at Xamarin.Android.Tasks.Aapt.ProcessManifest(ITaskItem manifestFile)
        at System.Threading.Tasks.Parallel.<>c__DisplayClass30_0`2.<ForEachWorker>b__0(Int32 i)
        at System.Threading.Tasks.Parallel.<>c__DisplayClass17_0`1.<ForWorker>b__1()
        at System.Threading.Tasks.Task.InnerInvoke()
        at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)
        at System.Threading.Tasks.Task.<>c__DisplayClass176_0.<ExecuteSelfReplicating>b__0(Object ) [C:\repos\msalnet\tests\devapps\XForms\XForms.Android\XForms.Android.csproj]

* We are hugely contributing to the dex limit for fields. Apps contain
  exponentially more fields for each library with resources.

An example from @PureWeen:

    1>  trouble writing output: Too many field references to fit in one dex file: 70468; max is 65536.

* Quite a few instances of aapt/aapt2 startup on developer's machines:
  this pegs the CPU. We have had a few general complaints about it.

Reviewing the source code for the Android gradle plugin, here is what
they do:

* Build the main app's "full" `R.txt` file.
* For each library, load its `R.txt` file.
* Map each resource in the library's `R.txt` back to the main app
* Write a small `R.java` file for each library: containing *only* the
  lines from the `R.txt` and updated integer values from the main app
  `R.txt` file.

Looking into this, we can do the exact same thing? We have the `R.txt`
file one directory above where we extract resources for each library.
We already had code parsing `R.txt` files I could repurpose, the only
thing *new* is a `R.java` writer: a pretty simple port from java.

The results are great!

    Before:
    3173 ms  _GenerateJavaDesignerForComponentAapt2     1 calls
    After:
      20 ms  GenerateLibraryResources                   1 calls

`_GenerateJavaDesignerForComponent` is now completely gone. This is a
total savings of ~3 seconds on first build and incremental builds
with library changes.

To compare APKs, I used:

    $ ~/android-toolchain/sdk/tools/bin/apkanalyzer dex packages Xamarin.Forms_Performance_Integration-Before.apk | grep ^F

Which omits a line for each field such as:

    F d 0	0	16	xamarin.forms_performance_integration.R$color int abc_background_cache_hint_selector_material_dark

So then, before these changes:

    $ ~/android-toolchain/sdk/tools/bin/apkanalyzer dex packages Xamarin.Forms_Performance_Integration-Before.apk | grep ^F | wc -l
    29681

After:

    $ ~/android-toolchain/sdk/tools/bin/apkanalyzer dex packages Xamarin.Forms_Performance_Integration-After.apk | grep ^F | wc -l
    17210

12K less fields in a "Hello World" Xamarin.Forms app!

Comparing file sizes seems good, too:

    $ zipinfo Xamarin.Forms_Performance_Integration-Before.apk | grep classes.dex
    -rw-rw-r--  6.3 unx  3657872 b- defX 19-Mar-28 16:37 classes.dex
    $ zipinfo Xamarin.Forms_Performance_Integration-After.apk | grep classes.dex
    -rw-rw-r--  6.3 unx  3533120 b- defX 19-Mar-28 16:20 classes.dex

Dex file in the APK is ~120KB smaller.

~~ What if R.txt is missing? ~~

I found this was the case when the
`<GetAdditionalResourcesFromAssemblies/>` MSBuild task runs. This is
an old codepath that allowed old support libraries to work.

In this case, a directory is created such as:

* `obj\Debug\resourcecache\CF390EBB0064FDA00BB090E733D37E89`
  * `adil`
  * `assets`
  * `libs`
  * `res`
  * `AndroidManifest.xml`
  * `classes.jar`

No `R.txt` file?

Checking the zip files we download:

    $ for z in ~/.local/share/Xamarin/zips/*.zip; do zipinfo $z; done | grep R.txt
    # no results

This actually makes sense, since the zip file contains the *actual
resources*.

To make this case work properly, we should just process the main app's
`R.txt` file when no library `R.txt` file is found. This will still be
faster than invoking `aapt`, even though we have more fields than
needed.

~~ Tests ~~

I added a set of unit tests for the `<GenerateLibraryResources/>`
MSBuild task. I also had to remove a few assertions that looked for
the `_GenerateJavaDesignerForComponent` MSBuild target.

Lastly, I added some assertions to a test that uses an old support
library to verify it's main `R.java` reasonably matches the library
`R.java` we generate.
jonpryor pushed a commit that referenced this issue Apr 1, 2019
)

Context: https://android.googlesource.com/platform/tools/base/+/refs/heads/master/build-system/builder/
Fixes: #2680
Fixes: #2836

The current behavior in the `_GenerateJavaDesignerForComponent`
MSBuild target does the following:

  * For each library that has Android resources... (in parallel)
  * Run an instance of `aapt`/`aapt2` to generate the `R.java` file
    for each library.
  * This actually creates an `R.java` file that contains *every*
    resource id for *every* library.  These libraries are not using
    most of these ids.

This has a few problems:

  * Issue #2680 notes a problem where a file is locked on Windows
    during `_GenerateJavaDesignerForComponent`:

        Xamarin.Android.Common.targets(1541,2): The process cannot access the file 'C:\repos\msalnet\tests\devapps\XForms\XForms.Android\obj\Debug\90\lp\26\jl\manifest\AndroidManifest.xml' because it is being used by another process.
            at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
            at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
            at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
            at System.IO.StreamWriter.CreateFile(String path, Boolean append, Boolean checkHost)
            at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding, Int32 bufferSize, Boolean checkHost)
            at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding)
            at Xamarin.Android.Tasks.ManifestDocument.Save(String filename)
            at Xamarin.Android.Tasks.Aapt.GenerateCommandLineCommands(String ManifestFile, String currentAbi, String currentResourceOutputFile)
            at Xamarin.Android.Tasks.Aapt.ProcessManifest(ITaskItem manifestFile)
            at System.Threading.Tasks.Parallel.<>c__DisplayClass30_0`2.<ForEachWorker>b__0(Int32 i)
            at System.Threading.Tasks.Parallel.<>c__DisplayClass17_0`1.<ForWorker>b__1()
            at System.Threading.Tasks.Task.InnerInvoke()
            at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)
            at System.Threading.Tasks.Task.<>c__DisplayClass176_0.<ExecuteSelfReplicating>b__0(Object ) [C:\repos\msalnet\tests\devapps\XForms\XForms.Android\XForms.Android.csproj]

  * We are hugely contributing to the dex limit for fields.  Apps
    contain exponentially more fields for each library with resources.

    An example from @PureWeen:

        1>  trouble writing output: Too many field references to fit in one dex file: 70468; max is 65536.

  * Quite a few instances of `aapt`/`aapt2` startup on developer's
    machines: this pegs the CPU.  We have had a few general
    complaints about it.

Reviewing the source code for the Android gradle plugin, here is what
they do:

 1. Build the main app's "full" `R.txt` file.
 2. For each library, load its `R.txt` file.
 3. Map each resource in the library's `R.txt` back to the main app
 4. Write a small `R.java` file for each library containing *only*
    the lines from the `R.txt` and updated integer values from the
    main app `R.txt` file.

Looking into this, can we do the exact same thing?  We have the
`R.txt` file one directory above where we extract resources for each
library.  We already had code parsing `R.txt` files we could
repurpose.  The only thing *new* is a `R.java` writer: a pretty
simple port from java.

The results are great!

	Before:
	3173 ms  _GenerateJavaDesignerForComponentAapt2     1 calls
	After:
	  20 ms  GenerateLibraryResources                   1 calls

`_GenerateJavaDesignerForComponent` is now completely gone.  This is
a total savings of ~3 seconds on first build and incremental builds
with library changes.

To compare APKs, I used:

	$ ~/android-toolchain/sdk/tools/bin/apkanalyzer dex packages Xamarin.Forms_Performance_Integration-Before.apk | grep ^F

Which omits a line for each field such as:

	F d 0	0	16	xamarin.forms_performance_integration.R$color int abc_background_cache_hint_selector_material_dark

So then, before these changes there were ~30000 fields:

	$ ~/android-toolchain/sdk/tools/bin/apkanalyzer dex packages Xamarin.Forms_Performance_Integration-Before.apk | grep ^F | wc -l
	29681

After, there are less than 18000 (58%!):

	$ ~/android-toolchain/sdk/tools/bin/apkanalyzer dex packages Xamarin.Forms_Performance_Integration-After.apk | grep ^F | wc -l
	17210

12K less fields in a "Hello World" Xamarin.Forms app!

Comparing file sizes seems good, too:

	$ zipinfo Xamarin.Forms_Performance_Integration-Before.apk | grep classes.dex
	-rw-rw-r--  6.3 unx  3657872 b- defX 19-Mar-28 16:37 classes.dex
	$ zipinfo Xamarin.Forms_Performance_Integration-After.apk | grep classes.dex
	-rw-rw-r--  6.3 unx  3533120 b- defX 19-Mar-28 16:20 classes.dex

The `.dex` file in the `.apk` is ~120KB smaller.


~~ What if R.txt is missing? ~~

I found this was the case when the
`<GetAdditionalResourcesFromAssemblies/>` MSBuild task runs.  This is
an old codepath that allowed old support libraries to work.

In this case, a directory is created such as:

  * `obj\Debug\resourcecache\CF390EBB0064FDA00BB090E733D37E89`
      * `aidl`
      * `AndroidManifest.xml`
      * `assets`
      * `classes.jar`
      * `libs`
      * `res`

No `R.txt` file?

Checking the zip files we download:

	$ for z in ~/.local/share/Xamarin/zips/*.zip; do zipinfo $z; done | grep R.txt
	# no results

This actually makes sense, since the zip file contains the
*actual resources*.

To make this case work properly, we should just process the main
app's `R.txt` file when no library `R.txt` file is found.  This will
still be faster than invoking `aapt`, even though we have more
fields than needed.


~~ Tests ~~

I added a set of unit tests for the `<GenerateLibraryResources/>`
MSBuild task.  I also had to remove a few assertions that looked for
the `_GenerateJavaDesignerForComponent` MSBuild target.

Lastly, I added some assertions to a test that uses an old support
library to verify it's main `R.java` reasonably matches the library
`R.java` we generate.
jonathanpeppers added a commit that referenced this issue Apr 1, 2019
)

Context: https://android.googlesource.com/platform/tools/base/+/refs/heads/master/build-system/builder/
Fixes: #2680
Fixes: #2836

The current behavior in the `_GenerateJavaDesignerForComponent`
MSBuild target does the following:

  * For each library that has Android resources... (in parallel)
  * Run an instance of `aapt`/`aapt2` to generate the `R.java` file
    for each library.
  * This actually creates an `R.java` file that contains *every*
    resource id for *every* library.  These libraries are not using
    most of these ids.

This has a few problems:

  * Issue #2680 notes a problem where a file is locked on Windows
    during `_GenerateJavaDesignerForComponent`:

        Xamarin.Android.Common.targets(1541,2): The process cannot access the file 'C:\repos\msalnet\tests\devapps\XForms\XForms.Android\obj\Debug\90\lp\26\jl\manifest\AndroidManifest.xml' because it is being used by another process.
            at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
            at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
            at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
            at System.IO.StreamWriter.CreateFile(String path, Boolean append, Boolean checkHost)
            at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding, Int32 bufferSize, Boolean checkHost)
            at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding)
            at Xamarin.Android.Tasks.ManifestDocument.Save(String filename)
            at Xamarin.Android.Tasks.Aapt.GenerateCommandLineCommands(String ManifestFile, String currentAbi, String currentResourceOutputFile)
            at Xamarin.Android.Tasks.Aapt.ProcessManifest(ITaskItem manifestFile)
            at System.Threading.Tasks.Parallel.<>c__DisplayClass30_0`2.<ForEachWorker>b__0(Int32 i)
            at System.Threading.Tasks.Parallel.<>c__DisplayClass17_0`1.<ForWorker>b__1()
            at System.Threading.Tasks.Task.InnerInvoke()
            at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)
            at System.Threading.Tasks.Task.<>c__DisplayClass176_0.<ExecuteSelfReplicating>b__0(Object ) [C:\repos\msalnet\tests\devapps\XForms\XForms.Android\XForms.Android.csproj]

  * We are hugely contributing to the dex limit for fields.  Apps
    contain exponentially more fields for each library with resources.

    An example from @PureWeen:

        1>  trouble writing output: Too many field references to fit in one dex file: 70468; max is 65536.

  * Quite a few instances of `aapt`/`aapt2` startup on developer's
    machines: this pegs the CPU.  We have had a few general
    complaints about it.

Reviewing the source code for the Android gradle plugin, here is what
they do:

 1. Build the main app's "full" `R.txt` file.
 2. For each library, load its `R.txt` file.
 3. Map each resource in the library's `R.txt` back to the main app
 4. Write a small `R.java` file for each library containing *only*
    the lines from the `R.txt` and updated integer values from the
    main app `R.txt` file.

Looking into this, can we do the exact same thing?  We have the
`R.txt` file one directory above where we extract resources for each
library.  We already had code parsing `R.txt` files we could
repurpose.  The only thing *new* is a `R.java` writer: a pretty
simple port from java.

The results are great!

	Before:
	3173 ms  _GenerateJavaDesignerForComponentAapt2     1 calls
	After:
	  20 ms  GenerateLibraryResources                   1 calls

`_GenerateJavaDesignerForComponent` is now completely gone.  This is
a total savings of ~3 seconds on first build and incremental builds
with library changes.

To compare APKs, I used:

	$ ~/android-toolchain/sdk/tools/bin/apkanalyzer dex packages Xamarin.Forms_Performance_Integration-Before.apk | grep ^F

Which omits a line for each field such as:

	F d 0	0	16	xamarin.forms_performance_integration.R$color int abc_background_cache_hint_selector_material_dark

So then, before these changes there were ~30000 fields:

	$ ~/android-toolchain/sdk/tools/bin/apkanalyzer dex packages Xamarin.Forms_Performance_Integration-Before.apk | grep ^F | wc -l
	29681

After, there are less than 18000 (58%!):

	$ ~/android-toolchain/sdk/tools/bin/apkanalyzer dex packages Xamarin.Forms_Performance_Integration-After.apk | grep ^F | wc -l
	17210

12K less fields in a "Hello World" Xamarin.Forms app!

Comparing file sizes seems good, too:

	$ zipinfo Xamarin.Forms_Performance_Integration-Before.apk | grep classes.dex
	-rw-rw-r--  6.3 unx  3657872 b- defX 19-Mar-28 16:37 classes.dex
	$ zipinfo Xamarin.Forms_Performance_Integration-After.apk | grep classes.dex
	-rw-rw-r--  6.3 unx  3533120 b- defX 19-Mar-28 16:20 classes.dex

The `.dex` file in the `.apk` is ~120KB smaller.

~~ What if R.txt is missing? ~~

I found this was the case when the
`<GetAdditionalResourcesFromAssemblies/>` MSBuild task runs.  This is
an old codepath that allowed old support libraries to work.

In this case, a directory is created such as:

  * `obj\Debug\resourcecache\CF390EBB0064FDA00BB090E733D37E89`
      * `aidl`
      * `AndroidManifest.xml`
      * `assets`
      * `classes.jar`
      * `libs`
      * `res`

No `R.txt` file?

Checking the zip files we download:

	$ for z in ~/.local/share/Xamarin/zips/*.zip; do zipinfo $z; done | grep R.txt
	# no results

This actually makes sense, since the zip file contains the
*actual resources*.

To make this case work properly, we should just process the main
app's `R.txt` file when no library `R.txt` file is found.  This will
still be faster than invoking `aapt`, even though we have more
fields than needed.

~~ Tests ~~

I added a set of unit tests for the `<GenerateLibraryResources/>`
MSBuild task.  I also had to remove a few assertions that looked for
the `_GenerateJavaDesignerForComponent` MSBuild target.

Lastly, I added some assertions to a test that uses an old support
library to verify it's main `R.java` reasonably matches the library
`R.java` we generate.
@xamarin xamarin locked as resolved and limited conversation to collaborators Jun 6, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Area: App+Library Build Issues when building Library projects or Application projects. need-info Issues that need more information from the author.
Projects
None yet
Development

No branches or pull requests

3 participants