[Discussion] Roadmap

[mahesh-hegde]

This issue is a discussion on development roadmap.

The code generator is the most important part of this project, which comprises of two major parts - the API summarizer and the binding generator.

Current Progress (28 July 2022)

I have got the summarizer working to a satisfactory level, (except the bugs / insufficiencies we discover as write the binding generator). The default implementation is based on OpenJDK Doclet API, but there's place for including an ASM based summarizer as well.

The summarizer has to be vendored later, and a script should be provided to build jar using mvn.

For now, summarizer must output json or some other serialization format, since macos and windows JNI bindings are not quite ready. Also, hand-writing JNI bindings will be error prone. In later stages we might be able to inspect the possibility of using jni_gen itself to generate bindings to the summarizer. (It's currently outputting JSON).

For the binding generator part, I have a skeleton at codegen_temp branch of my fork. Still have to write the code that generates bindings.

Binding generator architecture (planned)

Divided into a summary provider, config, and output writer.

output writer calls C and Dart binding generators, passing parts of the config as required.

User can add a script depending on package:jni_gen, creating a JniGenTask(summarySource, config, writer) or something like that.

In future we can also add a YAML config for basic options. But I think some options are better served by ability to write arbitrary callbacks.

package:jni refactoring

package:jni works right now, but relies on some unreliable assumptions. It requires a refactor: (#12)

• Indirect access to JNIEnv through a global variable in C where each method
  • checks if a thread_local JNIEnv is init'd; if not, call AttachThread()
  • calls the actual function on JNIEnv
  • converts reference to global if it's local.

It can be done by patching some parts of ffigen to also emit some extra C code.

• Refactor JniObject API to use global references
  • convert reference types to global ref.
  • inspect the possibility to reduce generated boilerplate: convert callXYZMethod to a call from dart and pass extra parameters, delegating most of the dispatch stuff to an equivalent C function.
  • if the above task goes well, we can make JniObject the dart counterpart for java.lang.Object, also the way to access methods in API boundaries. [1]

Dependencies

• In my understanding, FFIGEN traverses through transitively included header files. However I don't think it's a good strategy for java, for 2 reasons:

  1. Java has very large packages sometimes. going through dependencies of a single class may cover large number of packages most of which may be unnecessary. We have no ways to automatically tell which is required and which is not.
  2. We might end up duplicating lot of symbols due to (1)

• Instead, one jni_gen translation should correspond to one java package (and its subpackages) in java. (example, say p.a.b)

• Any dependency packages should be specified either in list of packages to translate, or as [package_name : import_path] maps.

• If a type outside the depended packages is encountered, it should be mapped to JniObject / or whatever dart type representing basic java object;

  • More fancy strategy - most specific supertype included in one of the depended packages.

• From the base type, casting to other type should be easy, even if target type's package is not included as dependency.

Roadmap

Here's the roadmap

• Make static and object methods & fields work well. (Approx. time 2 weeks)

• Refactor package:jni. (Approx. time 2 weeks)

• Implement package dependency resoultion / import paths between different files of jni_gen generated code. (Approx. time 1 week)

Future goals:

• Subtyping: Being able to subclass a java class from dart, if possible, enables nice interop with many parts of Android API. It also enables passing functions to java (because java lambdas have single-method-interface types).

• Generics: The doclet summarizer outputs some generics information. We have to calculate type erasure properly in JNI calls and map java generics to dart generics.

[liamappelbe]

Is the API summarizer going to be part of package:jni_gen, or a separate package? I think outputting a JSON API summary is unnecessary, especially if it's within the same package.

You're basically doing this: [3rd party Java parser] -> [Dart representation of AST] --(serialize)-> [JSON] --(deserialize)-> [Dart representation of AST] -> [binding generator]

You could skip the JSON step, right? If you've already written it that way, that's fine, but if you find yourself spending a lot of time on the serialization and deserialization logic, just skip it.

Next step: Focus on the output writer/code generator. Ignore those bits of package:jni cleanup for now.

[mahesh-hegde]

Parsing java APIs is in java using doclet API, writing bindings (main tool) is in dart. We need a way to access at least until codegen is ready. Daco suggested writing some manual Jni bindings (like ffigen did), but desktop support on windows is fragile, it doesn't exist for macOS right now, and writing accessors manually using JNI is quite some repetitive work. Mahesh

…

On Fri, 29 Jul, 2022, 12:02 am Liam Appelbe, ***@***.***> wrote: Is the API summarizer going to be part of package:jni_gen, or a separate package? I think outputting a JSON API summary is unnecessary, especially if it's within the same package. You're basically doing this: [3rd party Java parser] -> [Dart representation of AST] --(serialize)-> [JSON] --(deserialize)-> [Dart representation of AST] -> [binding generator] You could skip the JSON step, right? If you've already written it that way, that's fine, but if you find yourself spending a lot of time on the serialization and deserialization logic, just skip it. *Next step:* Focus on the output writer/code generator. Ignore those bits of package:jni cleanup for now. — Reply to this email directly, view it on GitHub <https://github.com/dart-lang/jni_gen/issues/18#issuecomment-1198499054>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALAKLFVEMKOXRDNF2F2MZHDVWLG4HANCNFSM546DMTKA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[mahesh-hegde]

I forgot to add few things.

[3rd party Java parser] -> [Dart representation of AST] --(serialize)-> [JSON] --(deserialize)-> [Dart representation of AST]

No it's building a tree at once using the visitors provided by parser. I could not find a way to access a prebuilt tree representation. For example, the type mirrors are usages of the class, and we have to visit the members to get some required information about the class.

As for JSON, it's trivial on Java side because there's reflection, I am just passing the built objects to jackson API. On dart side also I am using code generation (json_serializable). Only extra thing is double definition of classes on both dart and java. So far it seems to be OK.

[dcharkes]

You're basically doing this: [3rd party Java parser] -> [Dart representation of AST] --(serialize)-> [JSON] --(deserialize)-> [Dart representation of AST] -> [binding generator]

No, the current approach is: [3rd party Java parser] -> [Java representation of AST] --(serialize)-> [JSON] --(deserialize)-> [Dart representation of AST] -> [binding generator]

When we can use jni_gen to bind the 3rd party Java parser, then we can get immediately go to a Dart representation of the AST.

The summarizer has to be vendored later, and a script should be provided to build jar using mvn.

We should maybe make that a sub-folder on this repository. Including a CI step that builds the JAR and some Java unit tests.

Next step: Focus on the output writer/code generator. Ignore those bits of package:jni cleanup for now.

Agreed. Let's try to focus on getting a code generator for the simple use case working as first prio.