Skip to content

Commit

Permalink
[monodroid] Embedded assemblies store (#6311)
Browse files Browse the repository at this point in the history 
What do we want?  Faster (Release) App Startup!

How do we get that?  Assembly Stores!

"In the beginning", assemblies were stored in the `assemblies`
directory within the `.apk`.  App startup would open the `.apk`,
traverse all entries within the `.apk` looking for `assemblies/*.dll`,
`assemblies/*.dll.config`, and `assemblies/*.pdb` files.  When a
"supported" `assemblies/*` entry was encountered, the entry would be
**mmap**(2)'d so that it could be used; see also commit c195683.

Of particular note is:

 1. The need to enumerate *all* entries within the `.apk`, as there
    is no guarantee of entry ordering, and

 2. The need for *N* `mmap()` invocations, one per assembly included
    in the app, *plus* additional `mmap()` invocations for the `.pdb`
    and `.dll.config` files, if present.

    Useful contextual note: a "modern" AndroidX-using app could pull
    in dozens to over 200 assemblies without really trying.

    There will be *lots* of `mmap()` invocations.

Instead of adding (compressed! d236af5) data for each assembly
separately, instead add a small set of "Assembly Store" files which
contain the assembly & related data to use within the app:

  * `assemblies/assemblies.blob`
  * `assemblies/assemblies.[ARCHITECTURE].blob`

`assemblies.[ARCHITECTURE].blob` contains architecture-specific
assemblies, e.g. `System.Private.CoreLib.dll` built for x86 would be
placed within `assemblies.x86.blob`.  `ARCHITECTURE` is one of `x86`,
`x86_64`, `armeabi_v7a`, or `arm64_v8a`; note use of `_` instead of
`-`, which is different from the `lib/ARCHITECTURE` convention within
`.apk` files.  This is done because this is apparently what Android
and `bundletool` do, e.g. creating `split_config.armeabi_v7a.apk`.

Once the architecture-neutral `assemblies.blob` and appropriate
(singular!) `assemblies.[ARCHITECTURE].blob` for the current
architecture is found and `mmap()`'d, `.apk` entry traversal can end.
There is no longer a need to parse the entire `.apk` during startup.

The reduction in the number of `mmap()` system calls required can
have a noticeable impact on process startup, particularly with
.NET SDK for Android & MAUI; see below for timing details.

The assembly store format uses the followings structures:

	struct AssemblyStoreHeader {
	    uint32_t magic, version;
	    uint32_t local_entry_count;                    // Number of AssemblyStoreAssemblyDescriptor entries
	    uint32_t global_entry_count;                   // Number of AssemblyStoreAssemblyDescriptor entries in entire app, across all *.blob files
	    uint32_t store_id;
	};
	struct AssemblyStoreAssemblyDescriptor {
	    uint32_t data_offset, data_size;                // Offset from beginning of file for .dll data
	    uint32_t debug_data_offset, debug_data_size;    // Offset from beginning of file for .pdb data
	    uint32_t config_data_offset, config_data_size;  // Offset from beginning of file for .dll.config data
	};
	struct AssemblyStoreHashEntry {
	    union {
	        uint64_t hash64;                            // 64-bit xxhash of assembly filename
	        uint32_t hash64;                            // 32-bit xxhash of assembly filename
	    };
	    uint32_t mapping_index, local_store_index, store_id;
	};

The assembly store format is roughly as follows:

	AssemblyStoreHeader                 header {…};
	AssemblyStoreAssemblyDescriptor     assemblies [header.local_entry_count];

	// The following two entries exist only when header.store_id == 0
	AssemblyStoreHashEntry              hashes32[header.global_entry_count];
	AssemblyStoreHashEntry              hashes64[header.global_entry_count];

	uint8_t data[];

Note that `AssemblyStoreFileFormat::hashes32` and
`AssemblyStoreFileFormat::hashes64` are *sorted by their hash*.
Further note that assembly *filenames* are not present.
`EmbeddedAssemblies::blob_assemblies_open_from_bundles()` will hash
the filename, then binary search the appropriate `hashes*` array to
get the appropriate assembly information.

As the assembly store format doesn't include assembly names, `.apk`
and `.aab` files will also contain an `assemblies.manifest` file,
which contains the assembly names and other information in a human-
readable format; it is also used by `assembly-store-reader`:

	Hash 32     Hash 64             Blob ID  Blob idx  Name
	0xa2e0939b  0x4288cfb749e4c631  000      0000      Xamarin.AndroidX.Activity
	…
	0xad6f1e8a  0x6b0ff375198b9c17  001      0000      System.Private.CoreLib

Add a new `tools/assembly-store-reader` utility which can read the
new `assemblies*.blob` files:

	% tools/scripts/read-assembly-store path/to/app.apk
	Store set 'base_assemblies':
	  Is complete set? yes
	  Number of stores in the set: 5

	Assemblies:
	  0:
	    Name: Xamarin.AndroidX.Activity
	    Store ID: 0 (shared)
	    Hashes: 32-bit == 0xa2e0939b; 64-bit == 0x4288cfb749e4c631
	    Assembly image: offset == 1084; size == 14493
	    Debug data: absent
	    Config file: absent
	  …
	  16:
	    Name: System.Private.CoreLib
	    Store ID: 1 (x86)
	    Hashes: 32-bit == 0xad6f1e8a; 64-bit == 0x6b0ff375198b9c17
	    Assembly image: offset == 44; size == 530029
	    Debug data: absent
	    Config file: absent
	  …

On a Pixel 3 XL (arm64-v8a) running Android 12 with MAUI
6.0.101-preview.10.1952, we observe:

~~ MAUI: Displayed Time ~~

| Before ms |  After ms |            Δ | Notes                                 |
| ---------:| --------: | -----------: | ------------------------------------- |
|  1016.800 |   892.600 |    -12.21% ✓ | defaults; profiled AOT; 32-bit build  |
|  1016.100 |   894.700 |    -11.95% ✓ | defaults; profiled AOT; 64-bit build  |
|  1104.200 |   922.000 |    -16.50% ✓ | defaults; full AOT+LLVM; 64-bit build |
|  1102.700 |   926.100 |    -16.02% ✓ | defaults; full AOT; 32-bit build      |
|  1108.400 |   932.600 |    -15.86% ✓ | defaults; full AOT; 64-bit build      |
|  1106.300 |   932.600 |    -15.70% ✓ | defaults; full AOT+LLVM; 32-bit build |
|  1292.000 |  1271.800 |     -1.56% ✓ | defaults; 64-bit build                |
|  1307.000 |  1275.400 |     -2.42% ✓ | defaults; 32-bit build                |

Displayed time reduces by ~12% when Profiled AOT is used.

It is interesting to note that **Displayed time** is nearly identical
for the default (JIT) settings case. It's most probably caused by the
amount of JIT-ed code between `OnCreate()` and the time when the
application screen is presented, most likely the time is spent JIT-ing
MAUI rendering code.


~~ MAUI: Total native init time (before `OnCreate()`) ~~

| Before ms |  After ms |            Δ | Notes                                 |
| --------: | --------: | -----------: | ------------------------------------- |
|    96.727 |    88.921 |     -8.07% ✓ | defaults; 32-bit build                |
|    97.236 |    89.693 |     -7.76% ✓ | defaults; 64-bit build                |
|   169.315 |   108.845 |    -35.71% ✓ | defaults; profiled AOT; 32-bit build  |
|   170.061 |   109.071 |    -35.86% ✓ | defaults; profiled AOT; 64-bit build  |
|   363.864 |   208.949 |    -42.57% ✓ | defaults; full AOT; 64-bit build      |
|   363.629 |   209.092 |    -42.50% ✓ | defaults; full AOT; 32-bit build      |
|   373.203 |   218.289 |    -41.51% ✓ | defaults; full AOT+LLVM; 64-bit build |
|   372.783 |   219.003 |    -41.25% ✓ | defaults; full AOT+LLVM; 32-bit build |

Note that "native init time" includes running `JNIEnv.Initialize()`,
which requires loading `Mono.Android.dll` + dependencies such as
`System.Private.CoreLib.dll`, which in turn means that the AOT DSOs
such as `libaot-System.Private.CoreLib.dll.so` must *also* be loaded.
The loading of the AOT DSOs is why JIT is fastest here (no AOT DSOs),
and why Profiled AOT is faster than Full AOT (smaller DSOs).


~~ Plain Xamarin.Android: Displayed Time ~~

| Before ms |  After ms |            Δ | Notes                                 |
| --------: | --------: | -----------: | ------------------------------------- |
|   289.300 |   251.000 |    -13.24% ✓ | defaults; full AOT+LLVM; 64-bit build |
|   286.300 |   252.900 |    -11.67% ✓ | defaults; full AOT; 64-bit build      |
|   285.700 |   255.300 |    -10.64% ✓ | defaults; profiled AOT; 32-bit build  |
|   282.900 |   255.800 |     -9.58% ✓ | defaults; full AOT+LLVM; 32-bit build |
|   286.100 |   256.500 |    -10.35% ✓ | defaults; full AOT; 32-bit build      |
|   286.100 |   258.000 |     -9.82% ✓ | defaults; profiled AOT; 64-bit build  |
|   328.900 |   310.600 |     -5.56% ✓ | defaults; 32-bit build                |
|   319.300 |   313.000 |     -1.97% ✓ | defaults; 64-bit build                |


~~ Plain Xamarin.Android: Total native init time (before `OnCreate()`) ~~

| Before ms |  After ms |            Δ | Notes                                 |
| --------: | --------: | -----------: | ------------------------------------- |
|    59.768 |    42.694 |    -28.57% ✓ | defaults; profiled AOT; 64-bit build  |
|    60.056 |    42.990 |    -28.42% ✓ | defaults; profiled AOT; 32-bit build  |
|    65.829 |    48.684 |    -26.05% ✓ | defaults; full AOT; 64-bit build      |
|    65.688 |    48.713 |    -25.84% ✓ | defaults; full AOT; 32-bit build      |
|    67.159 |    49.938 |    -25.64% ✓ | defaults; full AOT+LLVM; 64-bit build |
|    67.514 |    50.465 |    -25.25% ✓ | defaults; full AOT+LLVM; 32-bit build |
|    66.758 |    62.531 |     -6.33% ✓ | defaults; 32-bit build                |
|    67.252 |    62.829 |     -6.58% ✓ | defaults; 64-bit build                |
  • Loading branch information
@grendello
grendello committed Oct 25, 2021
1 parent e0f3683 commit c927026
Show file tree
Hide file tree
Showing 72 changed files with 3,847 additions and 533 deletions.
1 change: 1 addition & 0 deletions Configuration.props
View file
Expand Up @@ -25,6 +25,7 @@
<!-- Should correspond to the first value from `$(API_LEVELS)` in `build-tools/api-xml-adjuster/Makefile` -->
<AndroidFirstFrameworkVersion Condition="'$(AndroidFirstFrameworkVersion)' == ''">v4.4</AndroidFirstFrameworkVersion>
<AndroidFirstApiLevel Condition="'$(AndroidFirstApiLevel)' == ''">19</AndroidFirstApiLevel>
<AndroidJavaRuntimeApiLevel Condition="'$(AndroidJavaRuntimeApiLevel)' == ''">21</AndroidJavaRuntimeApiLevel>
<!-- The min API level supported by Microsoft.Android.Sdk, should refactor/remove when this value is the same as $(AndroidFirstApiLevel) -->
<AndroidMinimumDotNetApiLevel Condition="'$(AndroidMinimumDotNetApiLevel)' == ''">21</AndroidMinimumDotNetApiLevel>
<AndroidFirstPlatformId Condition="'$(AndroidFirstPlatformId)' == ''">$(AndroidFirstApiLevel)</AndroidFirstPlatformId>
Expand Down
1 change: 1 addition & 0 deletions Documentation/README.md
View file
Expand Up @@ -11,6 +11,7 @@

* [Submitting Bugs, Feature Requests, and Pull Requests][bugs]
* [Directory Structure](project-docs/ExploringSources.md)
* [Assembly store format](project-docs/AssemblyStores.md)

[bugs]: https://github.com/xamarin/xamarin-android/wiki/Submitting-Bugs,-Feature-Requests,-and-Pull-Requests

Expand Down
224 changes: 224 additions & 0 deletions Documentation/project-docs/AssemblyStores.md
View file
@@ -0,0 +1,224 @@
<!-- markdown-toc start - Don't edit this section. Run M-x markdown-toc-refresh-toc -->
**Table of Contents**

- [Assembly Store format and purpose](#assembly-store-format-and-purpose)
- [Rationale](#rationale)
- [Store kinds and locations](#store-kinds-and-locations)
- [Store format](#store-format)
- [Common header](#common-header)
- [Assembly descriptor table](#assembly-descriptor-table)
- [Index store](#index-store)
- [Hash table format](#hash-table-format)

<!-- markdown-toc end -->

# Assembly Store format and purpose

Assembly stores are binary files which contain the managed
assemblies, their debug data (optionally) and the associated config
file (optionally). They are placed inside the Android APK/AAB
archives, replacing individual assemblies/pdb/config files.

Assembly stores are an optional form of assembly storage in the
archive, they can be used in all build configurations **except** when
Fast Deployment is in effect (in which case assemblies aren't placed
in the archives at all, they are instead synchronized from the host to
the device/emulator filesystem)

## Rationale

During native startup, the Xamarin.Android runtime looks inside the
application APK file for the managed assemblies (and their associated
pdb and config files, if applicable) in order to map them (using the
`mmap(2)` call) into memory so that they can be given to the Mono
runtime when it requests a given assembly is loaded. The reason for
the memory mapping is that, as far as Android is concerned, managed
assembly files are just data/resources and, thus, aren't extracted to
the filesystem. As a result, Mono wouldn't be able to find the
assemblies by scanning the filesystem - the host application
(Xamarin.Android) must give it a hand in finding them.

Applications can contain hundreds of assemblies (for instance a Hello
World MAUI application currently contains over 120 assemblies) and
each of them would have to be mmapped at startup, together with its
pdb and config files, if found. This not only costs time (each `mmap`
invocation is a system call) but it also makes the assembly discovery
an O(n) algorithm, which takes more time as more assemblies are added
to the APK/AAB archive.

An assembly store, however, needs to be mapped only once and any
further operations are merely pointer arithmetic, making the process
not only faster but also reducing the algorithm complexity to O(1).

# Store kinds and locations

Each application will contain at least a single assembly store, with
assemblies that are architecture-agnostics and any number of
architecture-specific stores. dotnet ships with a handful of
assemblies that **are** architecture-specific - those assemblies are
placed in an architecture specific store, one per architecture
supported by and enabled for the application. On the execution time,
the Xamarin.Android runtime will always map the architecture-agnostic
store and one, and **only** one, of the architecture-specific stores.

Stores are placed in the same location in the APK/AAB archive where the
individual assemblies traditionally live, the `assemblies/` (for APK)
and `base/root/assemblies/` (for AAB) folders.

The architecture agnostic store is always named `assemblies.blob` while
the architecture-specific one is called `assemblies.[ARCH].blob`.

Each APK in the application (e.g. the future Feature APKs) **may**
contain the above two assembly store files (some APKs may contain only
resources, other may contain only native libraries etc)

Currently, Xamarin.Android applications will produce only one set of
stores but when Xamarin.Android adds support for Android Features, each
feature APK will contain its own set of stores. All of the APKs will
follow the location, format and naming conventions described above.

# Store format

Each store is a structured binary file, using little-endian byte order
and aligned to a byte boundary. Each store consists of a header, an
assembly descriptor table and, optionally (see below), two tables with
assembly name hashes. All the stores are assigned a unique ID, with
the store having ID equal to `0` being the [Index store](#index-store)

Assemblies are stored as adjacent byte streams:

- **Image data**
Required to be present for all assemblies, contains the actual
assembly PE image.
- **Debug data**
Optional. Contains the assembly's PDB or MDB debug data.
- **Config data**
Optional. Contains the assembly's .config file. Config data
**must** be terminated with a `NUL` character (`0`), this is to
make runtime code slightly more efficient.

All the structures described here are defined in the
[`xamarin-app.hh`](../../src/monodroid/jni/xamarin-app.hh) file.
Should there be any difference between this document and the
structures in the header file, the information from the header is the
one that should be trusted.

## Common header

All kinds of stores share the following header format:

struct AssemblyStoreHeader
{
uint32_t magic;
uint32_t version;
uint32_t local_entry_count;
uint32_t global_entry_count;
uint32_t store_id;
;

Individual fields have the following meanings:

- `magic`: has the value of 0x41424158 (`XABA`)
- `version`: a value increased every time assembly store format changes.
- `local_entry_count`: number of assemblies stored in this assembly
store (also the number of entries in the assembly descriptor
table, see below)
- `global_entry_count`: number of entries in the index store's (see
below) hash tables and, thus, the number of assemblies stored in
**all** of the assembly stores across **all** of the application's
APK files, all the other assembly stores have `0` in this field
since they do **not** have the hash tables.
- `store_id`: a unique ID of this store.

## Assembly descriptor table

Each store header is followed by a table of
`AssemblyStoreHeader.local_entry_count` entries, each entry
defined by the following structure:

struct AssemblyStoreAssemblyDescriptor
{
uint32_t data_offset;
uint32_t data_size;
uint32_t debug_data_offset;
uint32_t debug_data_size;
uint32_t config_data_offset;
uint32_t config_data_size;
};

Only the `data_offset` and `data_size` fields must have a non-zero
value, other fields describe optional data and can be set to `0`.

Individual fields have the following meanings:

- `data_offset`: offset of the assembly image data from the
beginning of the store file
- `data_size`: number of bytes of the image data
- `debug_data_offset`: offset of the assembly's debug data from the
beginning of the store file. A value of `0` indicates there's no
debug data for this assembly.
- `debug_data_size`: number of bytes of debug data. Can be `0` only
if `debug_data_offset` is `0`
- `config_data_offset`: offset of the assembly's config file data
from the beginning of the store file. A value of `0` indicates
there's no config file data for this assembly.
- `config_data_size`: number of bytes of config file data. Can be
`0` only if `config_data_offset` is `0`

## Index store

Each application will contain exactly one store with a global index -
two tables with assembly name hashes. All the other stores **do not**
contain these tables. Two hash tables are necessary because hashes
for 32-bit and 64-bit devices are different.

The hash tables follow the [Assembly descriptor
table](#assembly-descriptor-table) and precede the individual assembly
streams.

Placing the hash tables in a single index store, while "wasting" a
certain amount of memory (since 32-bit devices won't use the 64-bit
table and vice versa), makes for simpler and faster runtime
implementation and the amount of memory wasted isn't big (1000
two tables which are 8kb long each, this being the amount of memory
wasted)

### Hash table format

Both tables share the same format, despite the hashes themselves being
of different sizes. This is done to make handling of the tables
easier on the runtime.

Each entry contains, among other fields, the assembly name hash. In
case of satellite assemblies, the assembly culture (e.g. `en/` or
`fr/`) is treated as part of the assembly name, thus resulting in a
unique hash. The hash value is obtained using the
[xxHash](https://cyan4973.github.io/xxHash/) algorithm and is
calculated **without** including the `.dll` extension. This is done
for runtime efficiency as the vast majority of Mono requests to load
an assembly does not include the `.dll` suffix, thus saving us time of
appending it in order to generate the hash for index lookup.

Each entry is represented by the following structure:

struct AssemblyStoreHashEntry
{
union {
uint64_t hash64;
uint32_t hash32;
};
uint32_t mapping_index;
uint32_t local_store_index;
uint32_t store_id;
};

Individual fields have the following meanings:

- `hash64`/`hash32`: the 32-bit or 64-bit hash of the assembly's name
**without** the `.dll` suffix
- `mapping_index`: index into a compile-time generated array of
assembly data pointers. This is a global index, unique across
**all** the APK files comprising the application.
- `local_store_index`: index into assembly store [Assembly descriptor table](#assembly-descriptor-table)
describing the assembly.
- `store_id`: ID of the assembly store containing the assembly
7 changes: 7 additions & 0 deletions Xamarin.Android.sln
View file
Expand Up @@ -148,6 +148,8 @@ Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "decompress-assemblies", "to
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "tmt", "tools\tmt\tmt.csproj", "{1A273ED2-AE84-48E9-9C23-E978C2D0CB34}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "assembly-store-reader", "tools\assembly-store-reader\assembly-store-reader.csproj", "{DA50FC92-7FE7-48B5-BDB6-CDA57B37BB51}"
EndProject
Global
GlobalSection(SharedMSBuildProjectFiles) = preSolution
src\Xamarin.Android.NamingCustomAttributes\Xamarin.Android.NamingCustomAttributes.projitems*{3f1f2f50-af1a-4a5a-bedb-193372f068d7}*SharedItemsImports = 5
Expand Down Expand Up @@ -408,6 +410,10 @@ Global
{1FED3F23-1175-42AA-BE87-EF1E8DB52F8B}.Debug|AnyCPU.Build.0 = Debug|Any CPU
{1FED3F23-1175-42AA-BE87-EF1E8DB52F8B}.Release|AnyCPU.ActiveCfg = Release|Any CPU
{1FED3F23-1175-42AA-BE87-EF1E8DB52F8B}.Release|AnyCPU.Build.0 = Release|Any CPU
{DA50FC92-7FE7-48B5-BDB6-CDA57B37BB51}.Debug|AnyCPU.ActiveCfg = Debug|anycpu
{DA50FC92-7FE7-48B5-BDB6-CDA57B37BB51}.Debug|AnyCPU.Build.0 = Debug|anycpu
{DA50FC92-7FE7-48B5-BDB6-CDA57B37BB51}.Release|AnyCPU.ActiveCfg = Release|anycpu
{DA50FC92-7FE7-48B5-BDB6-CDA57B37BB51}.Release|AnyCPU.Build.0 = Release|anycpu
EndGlobalSection
GlobalSection(SolutionProperties) = preSolution
HideSolutionNode = FALSE
Expand Down Expand Up @@ -474,6 +480,7 @@ Global
{37FCD325-1077-4603-98E7-4509CAD648D6} = {864062D3-A415-4A6F-9324-5820237BA058}
{88B746FF-8D6E-464D-9D66-FF2ECCF148E0} = {864062D3-A415-4A6F-9324-5820237BA058}
{1A273ED2-AE84-48E9-9C23-E978C2D0CB34} = {864062D3-A415-4A6F-9324-5820237BA058}
{DA50FC92-7FE7-48B5-BDB6-CDA57B37BB51} = {864062D3-A415-4A6F-9324-5820237BA058}
{1FED3F23-1175-42AA-BE87-EF1E8DB52F8B} = {04E3E11E-B47D-4599-8AFC-50515A95E715}
EndGlobalSection
GlobalSection(ExtensibilityGlobals) = postSolution
Expand Down
3 changes: 3 additions & 0 deletions build-tools/automation/azure-pipelines.yaml
View file
Expand Up @@ -397,6 +397,9 @@ stages:
cancelTimeoutInMinutes: 2
workspace:
clean: all
variables:
CXX: g++-10
CC: gcc-10
steps:
- checkout: self
clean: true
Expand Down
1 change: 1 addition & 0 deletions build-tools/installers/create-installers.targets
View file
Expand Up @@ -290,6 +290,7 @@
<_MSBuildFiles Include="$(MSBuildSrcDir)\Xamarin.SourceWriter.dll" />
<_MSBuildFiles Include="$(MSBuildSrcDir)\Xamarin.SourceWriter.pdb" />
<_MSBuildFiles Include="$(MSBuildSrcDir)\K4os.Compression.LZ4.dll" />
<_MSBuildFiles Include="$(MSBuildSrcDir)\K4os.Hash.xxHash.dll" />
</ItemGroup>
<ItemGroup>
<_MSBuildTargetsSrcFiles Include="$(MSBuildTargetsSrcDir)\Xamarin.Android.AvailableItems.targets" />
Expand Down

0 comments on commit c927026

Please sign in to comment.