Remove dependency on LMDBJava

[huahaiy]

LMDBJava implements LMDB comparators in Java, which creates a lot of overhead, as each comparison calls into Java from C. Since our native LMDB wrapper has already implemented the same logic, we can remove these overhead by switching to our own implementation that uses our own native C comparator. This also removes some duplicated code to save some maintenance burden. Another benefit is that we only need to implements the part that we need. Unfortunately, our native wrapper only works in GraalVM, so we need to write this wrapper again for regular JVM.

There's no point redoing the work that LMDBJava already does that uses Unsafe and reflections, as these are going to be removed from JDK in the future. If we don't want to do wasted work, we will have to wait for Project Panama. At least, we can try the new memory access API that seems to have already landed in JDK14, see design and a blog post.

So for JDK version prior to 14, we use LMDBJava, for versions after 14, we uses the new memory access API. Eventually, the dependency on LMDBJava can be removed.

[huahaiy]

Project Panama seems to be slow at this moment. Another option is JavaCPP, which is the fastest option that uses JNI, so we have time, we may write a LMDB wrapper using JavaCPP to replace LMDBJava for regular JVM.

[huahaiy]

Another option is to write a LMDB wrapper with https://github.com/cnuernber/dtype-next

[ieugen]

Would the new JDK 17 API's be of help with this?

https://openjdk.java.net/projects/jdk/17/
https://openjdk.java.net/jeps/412

[huahaiy]

Now, there's a Clojure wrapper for jdk 17 FFI. Seems to be promising. https://github.com/IGJoshua/coffi

[ieugen]

How would this work in the near future?
Depending on coffi will make minimum Java version be > 17 - once the API is out of incubator and stable.
So all apps that would like to use datalevin will have to upgrade.

[ieugen]

Once this is done, situations like this will be avoidable #73 (comment).

So the Foreign functions and memory API is out of incubator and in preview mode in JDK 19 https://openjdk.java.net/projects/jdk/19/ .
https://openjdk.java.net/jeps/424

If we are lucky it will be stable by this time next year (JDK 20 or JDK 21 in September 2023 ?! )

[huahaiy]

We would likely add a new LMDB binding for FFM, bringing the number to three: Java pre-FFM, Java post-FFM, and Graal. Datalevin library will choose the appropriate binding without user intervention, just like we do today between Graal and Java. So Java pre-FFM will still be using LMDBJava.

[huahaiy]

After weighting all the options, I am leaning towards writing a LMDB binding for JavaCPP, remove LMDBJava and my own GraalVM C API based binding.

JavaCPP is currently the fastest option that uses JNI, so it works fine in Graal native image. The only thing faster than it right now is the C API in GraalVM, which we currently use for native image.

Unfortunately, Oracle does not seem to consider the C API usage in native image something they are actively supporting. In fact, the latter versions of GraalVM effectively disabled this usage, see oracle/graal#5615

The new Java FFI API currently in early access mode is slower than JavaCPP and will not be able to support GraalVM for a long time.

The maintainer of JavaCPP is actively involved in exploring all the options, so I am sure JavaCPP will stay up to date no matter how things come about in the future.

By maintaining a single binding, the work load is reduced for us in the future as well.

[huahaiy]

Looks like Panama is getting faster than JNI. e.g. https://developer.okta.com/blog/2022/04/08/state-of-ffi-java

We will keep the current setup, and wait for Panama GA, and switch to it when that happens.