Build and publish JAR

[gavv]

Automatically build and publish our library.

Maybe use jitpack.io or Maven Central or JCenter.

[gavv]

I guess we'll have to publish both jar and aar.

[gavv]

One important question to think about.

When we're creating AAR, we include prebuilt libroc.so and libroc_jni.so for all 4 supported Android ABIs.

What should we do in case of JAR?

1. Don't include native libraries. Does it even make sense?
2. Include only libroc_jni.so.
3. Include both.

And if we're including prebuilt libraries, what architectures should we support?

On way to go is:

• Don't include libroc.so. Rationale: roc-java doesn't own it and doesn't control how it's built, so it should not ship it. Tell users that they should pre-install libroc system-wide to use the JAR.

• Include prebuilt libroc_jni.so for some limited set of operating systems and hardware architectures. Tell users that if their platform is not in the list, they should build bindings from source.

Another way:

• Ship both libraries. Instead of using system-wide libroc.so, roc-java should build it by itself, for all necessary platforms, like we do it for Android.

@MatteoArella what do you think?

[gavv]

BTW, in Go bindings we don't ship any native libraries and require system-wide installation of libroc.so. But in Go bindings we also don't need a native bridge like libroc_jni.so.

AFAIK, we could eliminate it in roc-java as well if we could use JNA or JNR-FFI instead of JNI. However, will we have performance and portability problems with them?

[MatteoArella]

I guess we'll have to publish both jar and aar.

Yes exactly

[MatteoArella]

One important question to think about.

When we're creating AAR, we include prebuilt libroc.so and libroc_jni.so for all 4 supported Android ABIs.

What should we do in case of JAR?

1. Don't include native libraries. Does it even make sense?

2. Include only libroc_jni.so.

3. Include both.

And if we're including prebuilt libraries, what architectures should we support?

On way to go is:

* Don't include libroc.so. Rationale: roc-java doesn't own it and doesn't control how it's built, so it should not ship it. Tell users that they should pre-install libroc system-wide to use the JAR.

* Include prebuilt libroc_jni.so for some limited set of operating systems and hardware architectures. Tell users that if their platform is not in the list, they should build bindings from source.

Another way:

* Ship both libraries. Instead of using system-wide libroc.so, roc-java should build it by itself, for all necessary platforms, like we do it for Android.

@MatteoArella what do you think?

In case of JAR in my opinion we should only ship libroc_jni.so library for differents os/architectures and let user pre-install libroc.

[MatteoArella]

BTW, in Go bindings we don't ship any native libraries and require system-wide installation of libroc.so. But in Go bindings we also don't need a native bridge like libroc_jni.so.

AFAIK, we could eliminate it in roc-java as well if we could use JNA or JNR-FFI instead of JNI. However, will we have performance and portability problems with them?

I've no experience with JNA and JNR-FFI but surely performances will be worse than using JNI because of runtime overhead due to java reflection operations.

[gavv]

In case of JAR in my opinion we should only ship libroc_jni.so library for differents os/architectures and let user pre-install libroc.

Another thought, what about libc? Should we also build libroc_jni for different glibc and musl versions, for example?

Different problems are possible here:

• We ship libroc_jni built with higher glibc version than installed on user system.

• We ship libroc_jni with an glibc version, and glibc on the user system has a higher major version (probably a rare case).

• We ship libroc_jni built with glibc, and the user uses musl (e.g. we're running on alpine, which is often used with docker nowadays).

How such problems are usually approached in Java world?

One other idea is to make libroc_jni.so part of roc repo instead of roc-java repo (e.g. enabled conditionally at build time). The user builds and installs roc and gets both libroc and libroc_jni compatible with current system. In this case there is no need to ship native libraries in JAR. But I'm not sure that would be good from architectural point of view, since libroc_jni and roc-java are highly tied.

[gavv]

BTW, are there any cases where a JAR without libroc_jni is useful (assuming that libroc doesn't provide libroc_jni too).

Also, a stupid question. Is there a way to create and publish a Java package that can be easily imported (without a need for manual clone-build-install) and that will build libroc_jni automatically during installation on the user system? Similar approach is used in Go.

[gavv]

I'm now thinking about the following solution:

• Select the list of hardware architectures we want to support.

• Select the list of operating systems and their versions we want to support.

• For Linux, select the list of popular distros and their versions we want to support explicitly. For each distro, find out what libc it uses and of what version.

• Given all the above, gather the list of toolchain triplets we will support, like: x86_64-pc-linux-gnu, aarch64-linux-gnu, aarch64-linux-musl. For each toolchain triplet, also select the list of libc versions. If a libc is always backwards compatible, this list may consist of a single oldest version we want to support. If the backwards compatibility is preserved only within major versions, this list may consists of a few major versions.

• Prepare an docker image or a set of images with all these toolchains and libc versions.

• Cross-compile libroc_jni.so using every toolchain and libc version. Pack all built libraries into the JAR.

• Write Java code that determines toolchain by the current environment and loads appropriate library from JAR.

[gavv]

It seems that rocksdb uses a similar approach: facebook/rocksdb#3143. It would be helpful to find out how other projects deal with this problem.

[gavv]

Regarding shipping libroc.so (in addition to libroc_jni.so).

We could build a self-sufficient version of libroc.so (using --build-3rdparty option) with no external dependencies except libc and ship it in JAR as well.

Probably, if a system-wide libroc.so is present, we can prefer using it, and otherwise fallback to the shipped version.

Pros:

• The user can just use the JAR and it will work out of the box. I guess this would be most convenient for Java world.

Cons:

• Shipping self-contained libroc.so means shipping statically linked libuv, sox, etc, which is convenient but has various disadvantages compared to using system libraries (not receiving security updates and distro-specific patches from distro maintainers; using only minimal versions of the libraries, e.g. not having most of the codecs available in sox).

• Being able to use both system-wide and shipped libroc.so may lead to some confusion. Maybe this should be controlled manually at build- or run-time.

It seems that rocksdb also ships the original native libraries in JAR: https://github.com/facebook/rocksdb/wiki/RocksJava-Basics.

I'm still not quite sure what approach is better.

[gavv]

Since the JAR publication wont be easy, I suggest to start with publishing AAR and online documentation and then continue working on JAR.

I've created a separate issue for AAR: #19, and renamed this issue to limit it to publishing JAR.

[MatteoArella]

Since the gradle cpp-library plugin doesn't cross-compile, in my opinion it's better to write a custom gradle plugin based on cmake (taking inspiration for example from gradle native cmake library and adapting it to our needs); then we can create a docker image for each variant we want to support and write a cmake toolchain file for cross-compiling the library. In addition with this approach we can share CMakeLists.txtbetween roc-java and roc-android builds.

[gavv]

@MatteoArella Makes sense.

[gavv]

After this task, we should not forget to update doc links in README.

[gavv]

We're thinking about gradually migrating from jni to jffi: #109. It seems that the hot path (writing and reading frames) won't have any performance downsides.

If we would migrate, creating portable JAR will become trivial because no native part would be needed.