Request to explore JavaCPP as an extension to model Python FFI.

[abhi18av]

Hi @Stewori

This is not an issue so please do feel free to close it down. This is more like a query/suggestion regarding the project. The basic question is -

Why is a seperate JyNI project needed to accomplish the linkage to python native extensions?

I'd like to suggest that perhaps, https://github.com/bytedeco/javacpp might be a worthy option to explore in this space as well. It's being used by ND4J and DeepLearning4J projects plus there are a number of bindings already generated located here https://github.com/bytedeco/javacpp-presets .

I hope that JavaCpp might take us closer to a proper package compatibility. I'm really looking forward to using proper python packages with Jython , Thanks again :)

[Stewori]

Hello @abhi18av,
note that JyNI is not yet another Java FFI, but actually emulates the existing CPython C-extension API.
This API is very involved and needs rather special treatment in terms of dynamic loading, memory management and various further aspects.

• it must bridge reference counting and mark-and-sweep gc
• it is especially challenging to get finalizers and weak references right in this context
• it must resolve GIL handling
• it needs to control RTLD flag, which isn't trivial via JNI

For details have a look at this paper.
E.g. with JavaCPP it is only possible to call into native code and to pass callbacks, while CPython extension API requires to export symbols (which is not supported by JavaCPP AFAICT).
Even if it would be feasible with JavaCPP, I would not see the benefit, because the JNI glue code is really the trivial part of JyNI and is already done.
Then, JavaCPP requires C/C++ compilation for each extension specifically, which is a major drawback compared to e.g. JNR ("interestingly" they ignore that aspect in their readme entirely and somehow describe JavaCPP as superior to e.g. JNR).
On the other hand, JavaCPP's main advantage - tight C++ integration - would not pay off for JyNI at all, because CPython is written in plain C.
Also, I doubt there is potential for performance gain, because JavaCPP boils down to JNI as well. In contrast to that, JNR is said to be slightly faster than plain JNI due to some smart use of assembler. Anyway, I would not explore any beyond-JNI optimization before project panama turns out, as that is by far the most promising advancement in terms of Java FFI performance (presumingly, JyNI 3 will make use of that one day).

[abhi18av]

After your wonderful explanation, I do have a better understanding of the scope of the project @Stewori . Thanks for taking out the time to be so thorough 👍

[abhi18av]

I do wish that Python3 transition would pick up speed in the Jython world :)

[saudet]

Hello, author of JavaCPP here. I agree, CPython doesn't need C++. :)
But I do hope that this is all going to get integrated one day...