Support Type & member remapping

[jonpryor]

Scenarios

There are two scenarios for which a "generalized" type & member remapping solution would be useful:

1. Desugaring
2. Microsoft Intune MAM

Desugaring

See also: dotnet/android#4574 (comment)
See also: dotnet/android#6142 (comment)

The Android Runtime is responsible for running the Dalvik bytecode within e.g. classes.dex. The Android runtime and Dalvik bytecode formats have apparently changed over time in order to support newer Java language features, such as support for static interface methods:

package org.webrtc;

public /* partial */ interface EGLBase {
    public static EGLBase create() { /* … */}
}

Support for static interface methods was added in Java 8 and in Android v8.0 (API-26). If you want to use static interface methods on an Android device running API-25 or earlier, you must desugar the Java bytecode. The result of desugaring the above org.webrtc.EGLBase type is that the static methods are moved to a new type, with a $-CC suffix, "as if" the Java code were instead:

package org.webrtc;

public /* partial */ interface EGLBase {
    // …
}

public final /* partial */ class EGLBase$-CC {
    public static EGLBase create { /* … */ }
}

While this works for pure Java code, this does not work with Xamarin.Android, because our bindings currently expect (require) that the types & members our binding process detects don't "move":

// "Binding" for EGLBase
namespace Org.Webrtc {
    public partial interface IEGLBase {
        private static readonly JniPeerMembers _members = new JniPeerMembers ("org/webrtc/EGLBase", typeof (IEGLBase));
        public static IEGLBase Create()
        {
            const string __id = "create.()Lorg/webrtc/EglBase;"
            var __rm = _members.StaticMethods.InvokeObjectMethod (__id, null);
            return Java.Lang.Object.GetObject<IEGLBase>(__rm.Handle, JniHandleOwnership.TransferLocalRef);
        }
    }
}

The new JniPeerMembers(…) invocation will use JNIEnv::FindClass() to lookup the org/webrtc/EGLBase type, and IEGLBase.Create() will use JNIEnv::GetStaticMethodID() and JNIEnv::CallStaticObjectMethod() to lookup and invoke the EGLBase.create() method.

However, when desugaring is used, there is no EGLBase.create() method. Consequently, in a "desugared" environment, a Java.Lang.NoSuchMethodError will be thrown when IEGLBase.Create() is invoked, as EGLBase.create() doesn't exist; it "should" instead be looking for EGLBase$-CC.create()!

Microsoft Intune MAM

See also: https://www.nuget.org/packages/Microsoft.Intune.MAM.Remapper.Tasks/
See also: https://docs.microsoft.com/en-us/mem/intune/developer/app-sdk-xamarin#remapper

The Microsoft Intune App SDK Xamarin Bindings is in a similar-yet-different position: it involves Java & Dalvik bytecode manipulation for various security purposes, and in order to make that work reasonably from Xamarin.Android, they also rewrite Xamarin.Android IL so that it's consistent with the manipulated Dalvik bytecode.

See readme.md and content/MonoAndroid10/remapping-config.json within the Microsoft.Intune.MAM.Remapper.Tasks NuGet package for some additional details/tidbits such as:

This task operates on assemblies to replace the base type classes which Microsoft Intune requires to implement its SDK (for example, Intune requires using a MAMActivity in place of Activity and methods such as OnMAMCreate instead of OnCreate).

      "Class": {
        "From": "android.app.Activity",
        "To": "com.microsoft.intune.mam.client.app.MAMActivity"
      },
      "Methods": [
        {
          "MakeStatic": false,
          "OriginalName": "onCreate",
          "NewName": "onMAMCreate",
          "OriginalParams": [
            "android.os.Bundle"
          ]
        },

Discussion

Unfortunately use cases (1) and (2) differ in two significant ways:

1. Desugaring requires a "primary" type and a "fallback" type for member lookup purposes. MAM doesn't require more than one type be involved, once the correct type has been looked up.
2. Desugaring doesn't change method names or signatures, just the declaring type (as currently known…?). MAM does change method names and signatures.

Regardless, from the binding side, the primary API of interest is Java.Interop.JniPeerMembers: this is the type through which member bindings "happen" (see IEGLBase type, above, for an example, or see unit tests).

The problem with JniPeerMembers is that it isn't modifiable; assuming that IL rewriting isn't employed, there's no way to change the behavior of JniPeerMembers at runtime.

For class lookup, there is an "extensibility point", kinda: JniRuntime.ClassLoader_LoadClass: https://github.com/xamarin/java.interop/blob/ff2714200107fb616828b9d1013f69605791d2ba/src/Java.Interop/Java.Interop/JniRuntime.cs#L176 ; see also commit b4d44e4.

The execution path is:

1. JniPeerMembers.JniPeerType uses JniType.GetCachedJniType(): https://github.com/xamarin/java.interop/blob/ff2714200107fb616828b9d1013f69605791d2ba/src/Java.Interop/Java.Interop/JniPeerMembers.cs#L80-L86

2. JniType.GetCachedJniType() uses JniEnvironment.Types.FindClass():

https://github.com/xamarin/java.interop/blob/ff2714200107fb616828b9d1013f69605791d2ba/src/Java.Interop/Java.Interop/JniType.cs#L88-L97

https://github.com/xamarin/java.interop/blob/ff2714200107fb616828b9d1013f69605791d2ba/src/Java.Interop/Java.Interop/JniType.cs#L35-L39

3. JniEnvironmentt.Types.FindClass() can hit JniRuntime.ClassLoader_LoadClass: https://github.com/xamarin/java.interop/blob/ff2714200107fb616828b9d1013f69605791d2ba/src/Java.Interop/Java.Interop/JniEnvironment.Types.cs#L63-L86

However, this isn't a generalized lookup path; it exists because, on (older versions of?) Android, JNIEnv::FindClass() would only reliably lookup types from the main thread. Other threads wouldn't find types in classes.dex, and thus ClassLoader.loadClass() was needed instead.

Additionally, the use of ClassLoader.loadClass() is only done after trying JNIEnv::FindClass(). This isn't a (current) fix. However, it could be the "path" toward an actual fix.

Additionally, the above is only for Managed-to-Java invocation purposes. We also need to consider Java-to-Managed method registration support, as the Intune MAM tooling will e.g. rename the Java Activity.onCreate() method to Activity.onMAMCreate(). However (?), given that our normal Java Callable Wrapper generation output uses a private final method, this might not in fact matter, as we only register the native methods, not the public methods:

https://github.com/xamarin/java.interop/blob/ff2714200107fb616828b9d1013f69605791d2ba/tests/Java.Interop.Tools.JavaCallableWrappers-Tests/Java.Interop.Tools.JavaCallableWrappers/JavaCallableWrapperGeneratorTests.cs#L130-L169

i.e. Intune MAM would (presumably) only update IndirectApplication.onCreate(), but not IndirectApplication.onCreate.n_onCreate(). TODO: verify this.

So long as the native methods aren't changed and parameter types aren't changed, Java-to-Managed method registration shouldn't require any changes.

Binding Background

Given an API description of:

https://github.com/xamarin/java.interop/blob/d16b1e567e40762b49ccdd62772f2a083be334f2/tests/generator-Tests/expected/NormalMethods/NormalMethods.xml#L18-L65

The resulting "new style" binding is:

https://github.com/xamarin/java.interop/blob/d16b1e567e40762b49ccdd62772f2a083be334f2/tests/generator-Tests/expected.ji/NormalMethods/Xamarin.Test.SomeObject.cs

Note in particular the binding for SomeObject.VoidMethodWithParams():

https://github.com/xamarin/java.interop/blob/d16b1e567e40762b49ccdd62772f2a083be334f2/tests/generator-Tests/expected.ji/NormalMethods/Xamarin.Test.SomeObject.cs#L254-L268

[jonpryor]

Proposal

The Java.Interop.JniRuntime.JniTypeManager type should be the public API. Add the following members:

partial class JniRuntime.JniTypeManager {

    protected virtual JniType? GetFallbackType (JniType type);

    public readonly record struct ReplacementMethodInfo (
            string JniSourceType,
            string SourceJniMethodName,
            string SourceJniMethodSignature)
    {
        public  string? JniTargetType                   {get; init;}
        public  string? TargetJniMethodName             {get; init;}
        public  string? TargetJniMethodSignature        {get; init;}
        public  int?    TargetJniMethodParameterCount   {get; init;}
        public  bool    TargetJniMethodIsStatic         {get; init;}
    }

    protected virtual string? GetReplacementType (string jniTypeSimpleReference);
    protected virtual ReplacementMethodInfo? GetReplacementMethodInfo (string jniSourceType, string jniMethodName, string jniMethodSignature);
}

These virtual methods would then be overridden by AndroidTypeManager in xamarin-android, which allows us to delegate the different-yet-equally-complicated task of obtaining the required data to make this all work, as part of the xamarin-android build process. There, we could add a new @(AndroidRemapConfig) item group, and have it support the Intune MAM remapping-config.json input format.

Desugar

GetFallbackType() is for the Desugar case, and this assumes that there's only one "fallback" type to lookup. (Should it instead be an IEnumerable<JniType>??). An appropriate invocation could be added to e.g. JniPeerMembers.JniStaticMethods.GetMethodInfo() if the initial member lookup fails: https://github.com/xamarin/java.interop/blob/ff2714200107fb616828b9d1013f69605791d2ba/src/Java.Interop/Java.Interop/JniPeerMembers.JniStaticMethods.cs#L25-L36

namespace Java.Interop
{
	partial class JniEnvironment {
		partial class StaticMethods {
			internal static bool TryGetStaticMethodID (JniObjectReference type, string name, string signature, out JniMethodInfo method);
		}
	}

	partial class JniPeerMembers {
		public sealed partial class JniStaticMethods {
			public JniMethodInfo GetMethodInfo (string encodedMember) {
				lock (StaticMethods) {
					if (!StaticMethods.TryGetValue (encodedMember, out var m)) {
						string name, signature;
						JniPeerMembers.GetNameAndSignature (encodedMember, out name, out signature);
						if (!JniEnvironment.StaticMethods.TryGetStaticMethodID (Members.PeerType.PeerReference, name, signature, out m)) {
							var fallback = JniEnvironment.CurrentRuntime.TypeManager.GetFallbackSimpleReference (Members.PeerType);
							if (fallback != null &&
									!JniEnvironment.StaticMethods.TryGetStaticMethodID (fallback.PeerReference, name, signature, out m)) {
								m = null;
							}
						}
						if (m == null) throw …
						StaticMethods.Add (encodedMember, m);
					}
					return m;
				}
			}
		}
	}
}

InTune

GetReplacementType() and GetReplacementMethodInfo() are for the InTune MAM case.

ClassRewrites

GetReplacementType() is for when the class hierarchy needs to be changed. For example, given this remapping-config.json entry:

  "ClassRewrites": [
    {
      "Class": {
        "From": "android.app.Activity",
        "To": "com.microsoft.intune.mam.client.app.MAMActivity"
      },
      "Methods": [
        {
          "MakeStatic": false,
          "OriginalName": "onCreate",
          "NewName": "onMAMCreate",
          "OriginalParams": [
            "android.os.Bundle"
          ]
        },

and the managed binding (built in xamarin-android):

	partial class Activity {
		static readonly JniPeerMembers _members = new XAPeerMembers ("android/app/Activity", typeof (Activity));

		[Register ("onCreate", "(Landroid/os/Bundle;)V", "GetOnCreate_Landroid_os_Bundle_Handler")]
		protected virtual unsafe void OnCreate (Android.OS.Bundle? savedInstanceState)
		{
			const string __id = "onCreate.(Landroid/os/Bundle;)V";
			try {
				JniArgumentValue* __args = stackalloc JniArgumentValue [1];
				__args [0] = new JniArgumentValue ((savedInstanceState == null) ? IntPtr.Zero : ((global::Java.Lang.Object) savedInstanceState).Handle);
				_members.InstanceMethods.InvokeVirtualVoidMethod (__id, this, __args);
			} finally {
				global::System.GC.KeepAlive (savedInstanceState);
			}
		}
	}

We would update the JniPeerMembers constructor: https://github.com/xamarin/java.interop/blob/d16b1e567e40762b49ccdd62772f2a083be334f2/src/Java.Interop/Java.Interop/JniPeerMembers.cs#L37-L65

to include a call to GetReplacementType():

partial class JniMeerMembers {
	string _originalJniPeerType;
	JniPeerMembers (string jniPeerTypeName, Type managedPeerType, bool checkManagedPeerType, bool isInterface)
	{
		_originalJniPeerType = jniPeerTypeName;

		jniPeerTypeName = JniEnvironment.CurrentRuntime.TypeManager.GetReplacmeentType (jniPeerTypeName) ??
			jniPeerTypeName;
		// ...
	}
}

Note: doing this means we lose the original type, e.g. android/app/Activity, and thus all Activity method lookups would be performed against MAMActivity. Is this correct/desirable?

To replace the onCreate() invocation with onMAMCreate(), we'd update JniPeerMembers.JniInstanceMethods to call GetReplacementMethodInfo(): https://github.com/xamarin/java.interop/blob/d16b1e567e40762b49ccdd62772f2a083be334f2/src/Java.Interop/Java.Interop/JniPeerMembers.JniInstanceMethods.cs#L84-L95

partial class JniMeerMembers {
	partial class JniInstanceMethods {
		public JniMethodInfo GetMethodInfo (string encodedMember)
		{
			lock (InstanceMethods) {
				if (!InstanceMethods.TryGetValue (encodedMember, out var m)) {
					string method, signature;
					JniPeerMembers.GetNameAndSignature (encodedMember, out method, out signature);
					var replacement = JniEnvironment.CurrentRuntime.TypeManager.GetReplacementMethodInfo (_originalJniPeerType, method, signature);
					if (replacement == null) {
						m = JniPeerType.GetInstanceMethod (method, signature);
						InstanceMethods.Add (encodedMember, m);
					} else {
						// use info in `replacement` to lookup new method
					}
				}
				return m;
			}
		}
	}
}

For onCreate() to be remapped to onMAMCreate(), we'd want GetReplacementMethodInfo() to return:

new ReplacementMethodInfo ("android/app/Activity", "onCreate", "(Landroid/os/Bundle;)V",
	JniTargetType = "com/microsoft/intune/mam/client/app/MAMActivity",
	TargetJniMethodName = "onMAMCreate",
	TargetJniMethodSignature = "(Landroid/os/Bundle;)V",
	TargetJniMethodParameterCount = 1,
	TargetJniMethodIsStatic = false,
);

GlobalMethodCalls

GetReplacementMethodInfo() is also for the GlobalMethodCalls case. When "MakeStatic": false is specified, the method is expected to be a static method, and we change which type it's looked up on. This overlaps with the Desugar case, in that changes to JniPeerMembers.JniStaticMethods.GetMethodInfo() is implied, but it can't be identical (meaning we need to "unify" logic somewhat). Given:

  "GlobalMethodCalls": [
    {
      "Class": {
        "From": "android.app.PendingIntent",
        "To": "com.microsoft.intune.mam.client.app.MAMPendingIntent"
      },
      "Methods": [
        {
          "MakeStatic": false,
          "OriginalName": "getActivities"
        },

then when JniPeerMembers.JniStaticMethods.GetMethodInfo() calls GetReplacementMethodInfo(), we'd want GetReplacementMethodInfo() to return:

new ReplacementMethodInfo ("android/app/PendingIntent", "getActivities", "(...)...",
	JniTargetType = "com/microsoft/intune/mam/client/app/MAMPendingIntent",
	TargetJniMethodName = "getActivities",
	TargetJniMethodSignature = "(...)...", // same as SourceJniMethodSignature
	TargetJniMethodParameterCount = …,
	TargetJniMethodIsStatic = true,
);

This also needs to handle all method overloads.

When "MakeStatic": true is specified, we need to turn an instance method invocation into a static method invocation. This overlaps with the ClassRewrites changes to JniPeerMembers.JniInstanceMethods.GetMethodInfo(), with the addition that we need to change JniMethodInfo:

partial class JniMethodInfo {
    internal int ParameterCount;
}

JniMethodInfo.ParameterCount will be set to ReplacementMethodInfo.TargetJniMethodParameterCount.

Given the remapping-config.json entry:

  "GlobalMethodCalls": [
    {
      "Class": {
        "From": "android.app.backup.BackupDataInput",
        "To": "com.microsoft.intune.mam.client.app.backup.MAMBackupDataInputHelper"
      },
      "Methods": [
        {
          "MakeStatic": true,
          "OriginalName": "readNextHeader"
        }
      ]
    },

then we'd want GetReplacementMethodInfo() to return:

new ReplacementMethodInfo ("android/app/backup/BackupDataInput", "readNextHeader", "(...)...",
	JniTargetType = "com/microsoft/intune/mam/client/app/backup/MAMBackupDataInputHelper",
	TargetJniMethodName = "readNextHeader",
	TargetJniMethodSignature = "(Landroid/app/backup/BackupDataInput;...)...", // note new start parameter
	TargetJniMethodParameterCount = 0,
	TargetJniMethodIsStatic = true,
);

Here's where things get "squinty": the binding for BackupDataInput.readNextHeader() is:

	partial class BackupDataInput {
		public virtual unsafe bool ReadNextHeader ()
		{
			const string __id = "readNextHeader.()Z";
			try {
				var __rm = _members.InstanceMethods.InvokeVirtualBooleanMethod (__id, this, null);
				return __rm;
			} finally {
			}
		}
	}

_members.InstanceMethods.InvokeVirtualBooleanMethod() is: https://github.com/xamarin/java.interop/blob/d16b1e567e40762b49ccdd62772f2a083be334f2/src/Java.Interop/Java.Interop/JniPeerMembers.JniInstanceMethods_Invoke.cs#L62-L78

We'd have to update InvokeVirtual*Method() so that when m.IsStatic is true -- which is normally unexpected when we're in invoking an instance method -- then we need to re-package the parameters, using the parameter count in JniMethodInfo.ParameterCount:

if (m.IsStatic) {
    JniArgumentValue *newParameters = stackalloc JniArgumentValue [m.ParameterCount+1];
    newParameters [0] = new JniArgumentValue (self);
    for (int i = 0; i < m.ParameterCount; ++i) {
        newParameters [i+1] = parameters[i];
    }
    var r = JniEnvironment.StaticMethods.CallBooleanMethod (/* TODO: get class ref? */, m, newParameters);
    // …
    return r;
}

[jonpryor]

…though now that I peruse remapping-config.json a bit more, I'm not sure what to make of:

    {
      "Class": {
        "From": "android.content.pm.PackageManager",
        "To": "com.microsoft.intune.mam.client.content.pm.MAMPackageManagement"
      },
      "Methods": [
        {
          "MakeStatic": true,
          "OriginalName": "checkPermission"
        },

What does it mean when "MakeStatic": true is specified? How does that work?

[jonpryor]

In retrospect, I don't want JniEnvironment.Types.FindClass() to do the type name remapping, as that means there's no way to work around the remapping, should that ever be necessary.

Instead, type name remapping should be done within JniPeerMembers.

[jonpryor]

@jonpryor asked:

What does it mean when "MakeStatic": true is specified? How does that work?

The intention is two things: the instance method becomes a static method invocation on the specified type, and a new parameter is inserted at the start. For example, given a Java method invocation of:

Permission p = manager.checkPermission("permName", "packageName");

the MakeStatic logic would update the Java bytecode to become:

Permission p = MAMPackageManagement.CheckPermission(manager, "permName", "packageName");

Two items of note:

1. The method name is PascalCased, from checkPermission to CheckPermission.
2. manager changes from an instance to a parameter.

[4brunu]

Any news on this? 🙂