Auto-generate native Java-interfacing files

[dmitry-timofeev]

Overview

This is a feature request and an another take on #80 .

Consider adding a tool that takes a Java source file with native methods or a native C header of such file, produced by javac, and turns it into a Rust-template. Basically:

X.java --> (X.rs, X-human-readable.rs)

Such a tool may produce a single template file, similar to what one currently writes by hand, or a couple of files:

• An intermediate, always re-generated, low-level .rs file, that has the methods with signatures as required by C-JNI. Such file might have some boilerplate that is required to guarantee safe operations (catches panics, converts errors into Java exceptions, etc., see the primary issue Higher-level api for defining java functions in a more "rusty" manner #80)
• Optionally, a user-facing template file. This one is generated once, has human-readable identifiers, possibly, exactly matching the names of Java methods (e.g., nativeCreate instead of Java_com_example_company_NativePeerProxy_nativeCreate). Such methods might also enjoy some conversions that are performed by the code in intermediate file (e.g., jint -> i32, again, see Higher-level api for defining java functions in a more "rusty" manner #80).

If there is a user-facing file, intermediate one calls its functions.

Requirements

1. Standard-compliance or reasonable restrictions. I.e., such a tool must be able to handle whatever javac -h produces. Restrictions may apply on method names (e.g., method names that contain only ASCII symbols).
2. Ease of use: using such a tool must be simpler than copy-pasting javac -h output.
3. Good UX:
   • Reasonable defaults, works out-of-the box.
   • Descriptive error messages when things go wrong at FFI boundary, since FFI is hard.
   • User-facing template that is IDE-friendly (macros hidden in intermediate file).
   • Refactoring-friendly: must work reasonably when native methods are added/renamed/have their signatures modified.
   • Build integration: whatever build-system you are using, the tool must integrate nicely (e.g., Gradle/Maven/Ant/Bazel/Cargo).
4. Low or moderate overhead: ideally, you get extra safety for no extra performance or complexity cost.
5. Works on popular OSes.

Criteria

1. Low or moderate complexity: no knowledge of 🚀 science required to implement and maintain.
2. Maintainable: when this library updates, it's easy and well-known how to modify the tool.
3. Permissive license.

Possible approaches

1. GNU-fu: Awk & friends, taking a javac -h output.
   • Don't have to reinvent the wheel to process text files.
   • Need to un-mangle method names to produce good user-facing file.
   • Not cross-platform.
2. Rust-fu: a CLI-tool taking javac -h output.
   • Might plug better than ^ into Cargo.
   • Might reinvent the wheel.
3. Java-fu: Java compiler plug-ins (aka processor, annotation processors).
   • Plugs natively into the compiler, therefore, works with anything you use to invoke javac.
   • Have programmatic access to source-file structure. Don't have to parse things.
   • Have to generate .rs files by hand.

Non-goals

• Automatic generation of native Rust proxies of Java objects, see Support for generating native proxies of Java objects #82

See also

• JNR might do something similar under the hood: https://github.com/jnr/jnr-ffi
• A discussion regarding error-handling (both throwing exceptions from native code and catching panics at FFI boundary): How to throw exceptions from rust #76

[fpoli]

For a similar reason I ended up writing https://github.com/viperproject/jni-gen (undocumented...)

Our approach inspects the JVM by reflection at Rust's compile time and generates wrappers for methods listed in a whitelist in build.rs (see here for an example). We introduced the whitelist because we choose to generate a wrapper for every inherited method, thus increasing a lot the size of the generated code. Whenever possible (that is, unambiguous) the user can leave a field empty to let the generator choose the only possible name/signature/...

An example of the generated bindings: https://viperproject.github.io/prusti-dev/viper_sys/wrappers/index.html

Some problems that we found:

• Java methods can be overloaded. How to choose the corresponding name in Rust?
• Java method names allow more characters than function names in Rust. How to translate them?
• How to handle inherited Java methods? Should the user know what is the base class that implements it, or should a wrapper be generated in all inheriting classes?

The end result works fine, but there is a lot that can be improved. For example, the whitelist in build.rs may be intimidating at a first glance, the name of the method parameters is not preserved, and so on.

[dmitry-timofeev]

Federico, thank you for sharing your experience!

Just to avoid any confusion: are you talking about the opposite use-case -- creating native proxies of Java objects? I think that deserves a separate issue, it would be great if you submit one! There might be something the library can provide to ease that task.

We actually have both use-cases, yours on smaller scale, just a couple of hand-written proxies. They look like this:

// Pseudo-code
struct JavaFooProxy {
  // A global reference to the Java object.
  foo: GlobalRef;

  // An abstraction that allows you to execute some code with JNIEnv in a thread-safe manner.
  //
  // How it provides a context (JNIEnv) -- is an implementation detail.
  // For example, there might be implementations that:
  //   - Always attach/detach threads, which is VERY costly.
  //   - Keep a couple of attached threads and pass the lambda to them,
  //   - Rely on the fact that your app never creates more than N threads 
  //     and never `detach` them (a benign leak).
  //   - For local references: use JNIEnv directly.
  executor: Executor;
  
  // Calls a Java `Foo#bar` method.
  def bar() {
    executor.with_context((env: JNIEnv) -> {
        env.call_method(foo, "bar", BAR_SIGNATURE, …);
    }
  }
}

Regarding method name translation, you might find useful the JNI rules (that's the opposite use-case, but there might be some solutions that you may re-use): https://docs.oracle.com/javase/9/docs/specs/jni/design.html#resolving-native-method-names

[fpoli]

Oh, I got it the other way around. Yes, I meant creating native proxies of Java objects. I just opened an issue to track it (#82).
Thanks for the suggestions!