Skip to content

Commit

Permalink
[Xamarin.Android.Build.Tasks, monodroid] LLVM Marshal Methods Infra (#…
Browse files Browse the repository at this point in the history
…7004)

Context: https://github.com/xamarin/xamarin-android/wiki/Blueprint#java-type-registration

Introduce low-level "plumbing" for future JNI marshal method work.

A JNI marshal method is a [JNI-callable][0] function pointer provided
to [`JNIEnv::RegisterNatives()`][1].  Currently, JNI marshal methods
are provided via the interaction between `generator` and
`JNINativeWrapper.CreateDelegate()`:

  * `generator` emits the "actual" JNI-callable method.
  * `JNINativeWrapper.CreateDelegate()` uses System.Reflection.Emit
    to *wrap* the `generator`-emitted for exception marshaling.
    (Though see also 32cff43.)

JNI marshal methods are needed for all Java-to-C# transitions.

Consider the virtual `Activity.OnCreate()` method:

	partial class Activity {
	  static Delegate? cb_onCreate_Landroid_os_Bundle_;
	  static Delegate GetOnCreate_Landroid_os_Bundle_Handler ()
	  {
	    if (cb_onCreate_Landroid_os_Bundle_ == null)
	      cb_onCreate_Landroid_os_Bundle_ = JNINativeWrapper.CreateDelegate ((_JniMarshal_PPL_V) n_OnCreate_Landroid_os_Bundle_);
	    return cb_onCreate_Landroid_os_Bundle_;
	  }
	
	  static void n_OnCreate_Landroid_os_Bundle_ (IntPtr jnienv, IntPtr native__this, IntPtr native_savedInstanceState)
	  {
	    var __this = global::Java.Lang.Object.GetObject<Android.App.Activity> (jnienv, native__this, JniHandleOwnership.DoNotTransfer)!;
	    var savedInstanceState = global::Java.Lang.Object.GetObject<Android.OS.Bundle> (native_savedInstanceState, JniHandleOwnership.DoNotTransfer);
	    __this.OnCreate (savedInstanceState);
	  }
	
	  // Metadata.xml XPath method reference: path="/api/package[@name='android.app']/class[@name='Activity']/method[@name='onCreate' and count(parameter)=1 and parameter[1][@type='android.os.Bundle']]"
	  [Register ("onCreate", "(Landroid/os/Bundle;)V", "GetOnCreate_Landroid_os_Bundle_Handler")]
	  protected virtual unsafe void OnCreate (Android.OS.Bundle? savedInstanceState) => …
	}

`Activity.n_OnCreate_Landroid_os_Bundle_()` is the JNI marshal method,
responsible for marshaling parameters from JNI values into C# types,
forwarding the method invocation to `Activity.OnCreate()`, and (if
necessary) marshaling the return value back to JNI.

`Activity.GetOnCreate_Landroid_os_Bundle_Handler()` is part of the
type registration infrastructure, providing a `Delegate` instance to
`RegisterNativeMembers .RegisterNativeMembers()`, which is eventually
passed to `JNIEnv::RegisterNatives()`.

While this works, it's not incredibly performant: unless using one of
the optimized delegate types (32cff43 et. al),
System.Reflection.Emit is used to create a wrapper around the marshal
method, which is something we've wanted to avoid doing for years.

Thus, the idea: since we're *already* bundling a native toolchain and
using LLVM-IR to produce `libxamarin-app.so` (b21cbf9, 5271f3e),
what if we emitted [Java Native Method Names][2] and *skipped* all
the done as part of `Runtime.register()` and
`JNIEnv.RegisterJniNatives()`?

Given:

	class MyActivity : Activity {
	    protected override void OnCreate(Bundle? state) => …
	}

During the build, `libxamarin-app.so` would contain the function:

	JNIEXPORT void JNICALL
	Java_crc…_MyActivity_n_1onCreate (JNIEnv *env, jobject self, jobject state);

During App runtime, the `Runtime.register()` invocation present in
[Java Callable Wrappers][3] would either be omitted or would be a
no-op, and Android/JNI would instead resolve `MyActivity.n_onCreate()`
as `Java_crc…_MyActivity_n_1onCreate()`.

Many of the specifics are still being investigated, and this feature
will be spread across various areas.

We call this effort "LLVM Marshal Methods".

First, prepare the way.  Update `Xamarin.Android.Build.Tasks.dll`
and `src/monodroid` to introduce support for generating JNI marshal
methods into `libxamarin-app.so`.  Most of the added code is
*disabled and hidden* behind `#if ENABLE_MARSHAL_METHODS`.

~~ TODO ~~

Other pieces, in no particular order:

  * Update [`Java.Interop.Tools.JavaCallableWrappers`][4] so that
    static constructors aren't needed when LLVM Marshal Methods
    are used.

  * Update [`generator`][5] so that *two* sets of marshal methods are
    emitted: the current set e.g.
    `Activity.n_OnCreate_Landroid_os_Bundle_()`, and an "overload"
    set which has [`UnmanagedCallersOnlyAttribute`][6].
    LLVM Marshal Methods will be able to directly call these
    "unmanaged marshal methods" without the overhead of
    `mono_runtime_invoke()`; see also f48b97c.

  * Finish the LLVM code generator so that LLVM Marshal Methods are
    emitted into `libxamarin-app.so`.

  * Update the linker so that much of the earlier marshal method
    infrastructure is removed in Release apps.  When
    LLVM Marshal Methods are used, there is no need for
    `Activity.cb_onCreate_Landroid_os_Bundle_`,
    `Actvitiy.GetOnCreate_Landroid_os_Bundle_Handler()`, or the
    `Activity.n_OnCreate_Landroid_os_Bundle_()` without
    `[UnmanagedCallersOnly]`.

Meanwhile, we cannot remove the existing infrastructure, as the
current System.Reflection.Emit-oriented code is needed for faster app
builds, a desirable feature of Debug configuration builds.

LLVM Marshal Methods will be a Release configuration-only feature.

[0]: https://docs.oracle.com/javase/8/docs/technotes/guides/jni/spec/design.html#native_method_arguments
[1]: https://docs.oracle.com/javase/8/docs/technotes/guides/jni/spec/functions.html#RegisterNatives
[2]: https://docs.oracle.com/javase/8/docs/technotes/guides/jni/spec/design.html#resolving_native_method_names
[3]: https://github.com/xamarin/xamarin-android/wiki/Blueprint#java-type-registration
[4]: https://github.com/xamarin/java.interop/tree/main/src/Java.Interop.Tools.JavaCallableWrappers
[5]: https://github.com/xamarin/xamarin-android/wiki/Blueprint#generator
[6]: https://docs.microsoft.com/dotnet/api/system.runtime.interopservices.unmanagedcallersonlyattribute?view=net-6.0
  • Loading branch information
grendello committed Jun 6, 2022
1 parent 9987069 commit e1af958
Show file tree
Hide file tree
Showing 28 changed files with 760 additions and 173 deletions.
4 changes: 4 additions & 0 deletions Directory.Build.props
Expand Up @@ -24,6 +24,10 @@
<RollForward>Major</RollForward>
</PropertyGroup>

<PropertyGroup Condition=" '$(MSBuildRuntimeType)' == 'Core' ">
<_EnableMarshalMethods>NoThanks</_EnableMarshalMethods> <!-- set to YesPlease to enable -->
</PropertyGroup>

<PropertyGroup>
<ProductVersion>12.3.99</ProductVersion>
<!-- NuGet package version numbers. See Documentation/guides/OneDotNet.md.
Expand Down
11 changes: 11 additions & 0 deletions src/Xamarin.Android.Build.Tasks/Tasks/GenerateJavaStubs.cs
Expand Up @@ -26,6 +26,8 @@ namespace Xamarin.Android.Tasks

public class GenerateJavaStubs : AndroidTask
{
public const string MarshalMethodsRegisterTaskKey = ".:!MarshalMethods!:.";

public override string TaskPrefix => "GJS";

[Required]
Expand Down Expand Up @@ -338,6 +340,9 @@ bool CreateJavaSources (IEnumerable<TypeDefinition> javaTypes, TypeDefinitionCac
string monoInit = GetMonoInitSource (AndroidSdkPlatform);
bool hasExportReference = ResolvedAssemblies.Any (assembly => Path.GetFileName (assembly.ItemSpec) == "Mono.Android.Export.dll");
bool generateOnCreateOverrides = int.Parse (AndroidSdkPlatform) <= 10;
#if ENABLE_MARSHAL_METHODS
var overriddenMethodDescriptors = new List<OverriddenMethodDescriptor> ();
#endif

bool ok = true;
foreach (var t in javaTypes) {
Expand All @@ -355,6 +360,9 @@ bool CreateJavaSources (IEnumerable<TypeDefinition> javaTypes, TypeDefinitionCac
};

jti.Generate (writer);
#if ENABLE_MARSHAL_METHODS
overriddenMethodDescriptors.AddRange (jti.OverriddenMethodDescriptors);
#endif
writer.Flush ();

var path = jti.GetDestinationPath (outputPath);
Expand Down Expand Up @@ -388,6 +396,9 @@ bool CreateJavaSources (IEnumerable<TypeDefinition> javaTypes, TypeDefinitionCac
}
}
}
#if ENABLE_MARSHAL_METHODS
BuildEngine4.RegisterTaskObjectAssemblyLocal (MarshalMethodsRegisterTaskKey, overriddenMethodDescriptors, RegisteredTaskObjectLifetime.Build);
#endif
return ok;
}

Expand Down
Expand Up @@ -299,8 +299,16 @@ void AddEnvironment ()

int assemblyCount = 0;
HashSet<string> archAssemblyNames = null;

#if ENABLE_MARSHAL_METHODS
var uniqueAssemblyNames = new HashSet<string> (StringComparer.OrdinalIgnoreCase);
#endif
Action<ITaskItem> updateAssemblyCount = (ITaskItem assembly) => {
string assemblyName = Path.GetFileName (assembly.ItemSpec);
#if ENABLE_MARSHAL_METHODS
if (!uniqueAssemblyNames.Contains (assemblyName)) {
uniqueAssemblyNames.Add (assemblyName);
}
#endif
if (!UseAssemblyStore) {
assemblyCount++;
return;
Expand All @@ -316,7 +324,6 @@ void AddEnvironment ()
} else {
archAssemblyNames ??= new HashSet<string> (StringComparer.OrdinalIgnoreCase);
string assemblyName = Path.GetFileName (assembly.ItemSpec);
if (!archAssemblyNames.Contains (assemblyName)) {
assemblyCount++;
archAssemblyNames.Add (assemblyName);
Expand Down Expand Up @@ -431,16 +438,34 @@ void AddEnvironment ()
JNIEnvRegisterJniNativesToken = jnienv_registerjninatives_method_token,
};
appConfigAsmGen.Init ();

#if ENABLE_MARSHAL_METHODS
var marshalMethodsAsmGen = new MarshalMethodsNativeAssemblyGenerator () {
NumberOfAssembliesInApk = assemblyCount,
UniqueAssemblyNames = uniqueAssemblyNames,
OverriddenMethodDescriptors = BuildEngine4.GetRegisteredTaskObjectAssemblyLocal<List<Java.Interop.Tools.JavaCallableWrappers.OverriddenMethodDescriptor>> (GenerateJavaStubs.MarshalMethodsRegisterTaskKey, RegisteredTaskObjectLifetime.Build)
};
marshalMethodsAsmGen.Init ();
#endif
foreach (string abi in SupportedAbis) {
string baseAsmFilePath = Path.Combine (EnvironmentOutputDirectory, $"environment.{abi.ToLowerInvariant ()}");
string llFilePath = $"{baseAsmFilePath}.ll";
string targetAbi = abi.ToLowerInvariant ();
string environmentBaseAsmFilePath = Path.Combine (EnvironmentOutputDirectory, $"environment.{targetAbi}");
string marshalMethodsBaseAsmFilePath = Path.Combine (EnvironmentOutputDirectory, $"marshal_methods.{targetAbi}");
string environmentLlFilePath = $"{environmentBaseAsmFilePath}.ll";
string marshalMethodsLlFilePath = $"{marshalMethodsBaseAsmFilePath}.ll";

AndroidTargetArch targetArch = GetAndroidTargetArchForAbi (abi);
using (var sw = MemoryStreamPool.Shared.CreateStreamWriter ()) {
appConfigAsmGen.Write (targetArch, sw, environmentLlFilePath);
sw.Flush ();
Files.CopyIfStreamChanged (sw.BaseStream, environmentLlFilePath);
}
#if ENABLE_MARSHAL_METHODS
using (var sw = MemoryStreamPool.Shared.CreateStreamWriter ()) {
appConfigAsmGen.Write (GetAndroidTargetArchForAbi (abi), sw, llFilePath);
marshalMethodsAsmGen.Write (targetArch, sw, marshalMethodsLlFilePath);
sw.Flush ();
Files.CopyIfStreamChanged (sw.BaseStream, llFilePath);
Files.CopyIfStreamChanged (sw.BaseStream, marshalMethodsLlFilePath);
}
#endif
}

void AddEnvironmentVariable (string name, string value)
Expand Down
8 changes: 7 additions & 1 deletion src/Xamarin.Android.Build.Tasks/Tasks/PrepareAbiItems.cs
Expand Up @@ -14,7 +14,9 @@ public class PrepareAbiItems : AndroidTask
const string TypeMapBase = "typemaps";
const string EnvBase = "environment";
const string CompressedAssembliesBase = "compressed_assemblies";

#if ENABLE_MARSHAL_METHODS
const string MarshalMethodsBase = "marshal_methods";
#endif
public override string TaskPrefix => "PAI";

[Required]
Expand Down Expand Up @@ -50,6 +52,10 @@ public override bool RunTask ()
baseName = EnvBase;
} else if (String.Compare ("compressed", Mode, StringComparison.OrdinalIgnoreCase) == 0) {
baseName = CompressedAssembliesBase;
#if ENABLE_MARSHAL_METHODS
} else if (String.Compare ("marshal_methods", Mode, StringComparison.OrdinalIgnoreCase) == 0) {
baseName = MarshalMethodsBase;
#endif
} else {
Log.LogError ($"Unknown mode: {Mode}");
return false;
Expand Down
Expand Up @@ -8,7 +8,7 @@
"Size": 60736
},
"assemblies/Mono.Android.dll": {
"Size": 152435
"Size": 152344
},
"assemblies/rc.bin": {
"Size": 1083
Expand Down Expand Up @@ -59,13 +59,13 @@
"Size": 9591
},
"assemblies/UnnamedProject.dll": {
"Size": 3560
"Size": 3555
},
"classes.dex": {
"Size": 348440
},
"lib/arm64-v8a/libmonodroid.so": {
"Size": 484512
"Size": 510920
},
"lib/arm64-v8a/libmonosgen-2.0.so": {
"Size": 4667768
Expand All @@ -80,7 +80,7 @@
"Size": 146816
},
"lib/arm64-v8a/libxamarin-app.so": {
"Size": 15944
"Size": 15904
},
"META-INF/BNDLTOOL.RSA": {
"Size": 1213
Expand All @@ -92,7 +92,7 @@
"Size": 3646
},
"res/drawable-hdpi-v4/icon.png": {
"Size": 4762
"Size": 4791
},
"res/drawable-mdpi-v4/icon.png": {
"Size": 2200
Expand All @@ -116,5 +116,5 @@
"Size": 1904
}
},
"PackageSize": 3492724
"PackageSize": 3500916
}
Expand Up @@ -5,10 +5,10 @@
"Size": 2604
},
"assemblies/Java.Interop.dll": {
"Size": 67948
"Size": 67950
},
"assemblies/Mono.Android.dll": {
"Size": 257194
"Size": 257187
},
"assemblies/mscorlib.dll": {
"Size": 769016
Expand All @@ -23,7 +23,7 @@
"Size": 2879
},
"classes.dex": {
"Size": 349820
"Size": 349736
},
"lib/arm64-v8a/libmono-btls-shared.so": {
"Size": 1613872
Expand All @@ -32,7 +32,7 @@
"Size": 750976
},
"lib/arm64-v8a/libmonodroid.so": {
"Size": 392576
"Size": 421872
},
"lib/arm64-v8a/libmonosgen-2.0.so": {
"Size": 4030448
Expand Down Expand Up @@ -74,5 +74,5 @@
"Size": 1724
}
},
"PackageSize": 4044500
"PackageSize": 4052692
}
Expand Up @@ -11,7 +11,7 @@
"Size": 66806
},
"assemblies/Mono.Android.dll": {
"Size": 447633
"Size": 447543
},
"assemblies/mscorlib.dll": {
"Size": 3888
Expand Down Expand Up @@ -119,7 +119,7 @@
"Size": 1906
},
"assemblies/UnnamedProject.dll": {
"Size": 117250
"Size": 117244
},
"assemblies/Xamarin.AndroidX.Activity.dll": {
"Size": 5941
Expand Down Expand Up @@ -191,7 +191,7 @@
"Size": 3460820
},
"lib/arm64-v8a/libmonodroid.so": {
"Size": 484512
"Size": 510920
},
"lib/arm64-v8a/libmonosgen-2.0.so": {
"Size": 4667768
Expand All @@ -206,7 +206,7 @@
"Size": 146816
},
"lib/arm64-v8a/libxamarin-app.so": {
"Size": 98480
"Size": 98440
},
"META-INF/android.support.design_material.version": {
"Size": 12
Expand Down Expand Up @@ -773,7 +773,7 @@
"Size": 470
},
"res/drawable-hdpi-v4/icon.png": {
"Size": 4762
"Size": 4791
},
"res/drawable-hdpi-v4/notification_bg_low_normal.9.png": {
"Size": 212
Expand Down Expand Up @@ -1961,5 +1961,5 @@
"Size": 341228
}
},
"PackageSize": 8347981
"PackageSize": 8356173
}
Expand Up @@ -8,10 +8,10 @@
"Size": 7215
},
"assemblies/Java.Interop.dll": {
"Size": 68913
"Size": 68916
},
"assemblies/Mono.Android.dll": {
"Size": 567718
"Size": 567711
},
"assemblies/Mono.Security.dll": {
"Size": 68432
Expand Down Expand Up @@ -65,22 +65,22 @@
"Size": 131930
},
"assemblies/Xamarin.AndroidX.DrawerLayout.dll": {
"Size": 15426
"Size": 15425
},
"assemblies/Xamarin.AndroidX.Fragment.dll": {
"Size": 43135
"Size": 43134
},
"assemblies/Xamarin.AndroidX.Legacy.Support.Core.UI.dll": {
"Size": 6715
"Size": 6714
},
"assemblies/Xamarin.AndroidX.Lifecycle.Common.dll": {
"Size": 7062
},
"assemblies/Xamarin.AndroidX.Lifecycle.LiveData.Core.dll": {
"Size": 7193
"Size": 7194
},
"assemblies/Xamarin.AndroidX.Lifecycle.ViewModel.dll": {
"Size": 4872
"Size": 4873
},
"assemblies/Xamarin.AndroidX.Loader.dll": {
"Size": 13585
Expand Down Expand Up @@ -113,7 +113,7 @@
"Size": 43497
},
"classes.dex": {
"Size": 3462252
"Size": 3462080
},
"lib/arm64-v8a/libmono-btls-shared.so": {
"Size": 1613872
Expand All @@ -122,7 +122,7 @@
"Size": 750976
},
"lib/arm64-v8a/libmonodroid.so": {
"Size": 392576
"Size": 421872
},
"lib/arm64-v8a/libmonosgen-2.0.so": {
"Size": 4030448
Expand Down Expand Up @@ -1883,5 +1883,5 @@
"Size": 341040
}
},
"PackageSize": 9562270
"PackageSize": 9570462
}
@@ -1,10 +1,8 @@
using System;
using System.Collections.Generic;
using System.IO;
using System.Text;

using Java.Interop.Tools.TypeNameMappings;
using K4os.Hash.xxHash;
using Microsoft.Build.Framework;
using Microsoft.Build.Utilities;

Expand Down Expand Up @@ -342,21 +340,11 @@ void WriteDSOCache (LlvmIrGenerator generator)

// We need to hash here, because the hash is architecture-specific
foreach (StructureInstance<DSOCacheEntry> entry in dsoCache) {
entry.Obj.hash = HashName (entry.Obj.HashedName);
entry.Obj.hash = HashName (entry.Obj.HashedName, is64Bit);
}
dsoCache.Sort ((StructureInstance<DSOCacheEntry> a, StructureInstance<DSOCacheEntry> b) => a.Obj.hash.CompareTo (b.Obj.hash));

generator.WriteStructureArray (dsoCacheEntryStructureInfo, dsoCache, "dso_cache");

ulong HashName (string name)
{
byte[] nameBytes = Encoding.UTF8.GetBytes (name);
if (is64Bit) {
return XXH64.DigestOf (nameBytes, 0, nameBytes.Length);
}

return (ulong)XXH32.DigestOf (nameBytes, 0, nameBytes.Length);
}
}
}
}

0 comments on commit e1af958

Please sign in to comment.