creating static library requires ar

[johanvos]

When creating a static library, we typically use 2 steps:
clang++ -c foo.cpp -o foo.o
ar -rcs libfoo.a foo.o

Currently, Builder.compile() uses a single command to create a dynamic library.
Maybe an additional property platform.libtool could be used?

[saudet]

We could add a platform.linker, but that assumes we have more than one object file to link. We had problems with duplicate common functionality, right?

[johanvos]

There can be a single object file inside a library, but we still require an archiver to build the library.
About the duplicate common functionality: ultimately, all code and libraries are merged into a single executable, and external functions with the same name will cause issues indeed.
But that is fixed now that we have different OnLoad functions for different libraries.

[saudet]

But what about the JNI functions for the base Pointer classes? Those are exported and will cause conflicts...

As for the archiver, couldn't this be done anyway in the step where everything gets merged together?

[johanvos]

You're right about the common functions that are duplicated in every library (I forgot about those ones). I see 2 options here:

1. prefix those functions so that they have a unique name
2. make them static (internal, so they won't be exported)
3. provide them outside the generated .cpp, and reference them from inside the .cpp

Creating the archive: I don't think that can be done in a single step (as you currently do to create a shared library), the ar step is really needed, afaik.

[saudet]

Another option is to generate for all the dependencies in the same .cpp file. This is what I was assuming would be satisfactory for platforms like iOS, but is there any major downside to this approach? It all ends up in the same binary blob...

If we end up having to produce multiple compilation units though, couldn't we call the archiver after and do ar -rcs libfoo.a foo1.o foo2.o etc as part of the final build step for the blob?

[johanvos]

That is an option as well, but it requires more bookkeeping and changes in the dl4j build process. In order to run deeplearning4j, I noticed javacpp was used to create native libs at at least 2 places:

• the presets openblas jar
• the nd4j
  Those 2 jars each contain a native library generated by javacpp (hence duplicate symbols)

Ultimately, it would be the easiest if the native libraries are not contained in separate jars. A single library that is created based on all the object files would be sufficient.

I would be very ok with something like ar -rcs libeverything.a openblass.o nd4j.o as it makes it easy to include libeverything.a with the final executable on iOS. But we still have symbols in openblass.o that are also in nd4j.o, right?

[saudet]

Everything in one place, we can easily do something like that, that I've once tried with RoboVM:
https://github.com/bytedeco/javacpp#instructions-for-android-avian-and-robovm
Of course, that means we might have to build things like OpenBLAS and libnd4j from source, and the workflow would be something like:

1. Build openblas.a and libnd4j.a from source
2. Get all the Java classes into one place, say the classes subdirectory
3. Run java -jar javacpp.jar -cp classes/ -properties <ios-arm|ios-x86|etc> -o libjni on that
4. Link all the classes, openblas.a, libnd4j.a, and libjni.o together into the final binary

I don't think OpenBLAS and libnd4j have conflicting symbols, they shouldn't anyway. If they do, we will fix that in libnd4j.

[johanvos]

Actually, that is the flow I followed before. And it works.
But it would be nice to follow the same flow as on other platforms, so that building nd4j for ios is exactly the same flow (using mvn) is building nd4j for e.g. linux.

[saudet]

It would already work if Apple didn't remove the -multiply_defined functionality from their linker. That's how both the JDK and the GNU linker work...

[saudet]

I've added a commit to the "ci" branch only for now, until we can say for sure that we are committed to this approach, so please check it out and let me know what you think.

Basically, the idea is to generate all common functionality (that isn't inlined anyway) into jnijavacpp.cpp, sort of like a static library in itself, with it own "JNI_OnLoad_javacpp()" that gets called from the main jni<libname>.cpp file. On platforms that support dynamic libraries, we just use the compiler to link both files as usual into one library. On platforms like iOS, it ends up calling the compiler twice, and generates two .o files. In either case, they can be packaged into artifacts and deployed to Maven repositories.

I think it makes sense leaving them as .o files. It makes it easier since Maven, etc can consider them naturally as duplicate resources, which isn't the case if we archive them together in .a files, where we end up with linker errors if not further processed with native tools. Thoughts?

[johanvos]

Having the common functionality in a separate file is a good thing, I believe.
A question that remains is related to how nd4j uses javacpp (via the maven plugin): the output of building nd4j-native is a jar that includes a library, AND javacpp is also used for generating the jni code for libnd4j.

Running javacpp manually (without the maven plugin) and generating .o files should work fine. Since an app may contain a number of jars that are expected to include a javacpp generated library, this means we need to keep the .o files until all processing is done.
That is doable, but I think at this moment the nd4j build tools assume that separate libraries are generated.

A fix would be an option in the javacpp plugin that allows to disable the generation of a library, and an additional step in the nd4j build sequence, where at the very end all .o's are archived in a static library.

[saudet]

I thought you said you wanted it to work like other platforms where users just need to include the dependencies with Maven and not deal with JavaCPP, right? What changed your mind?

[johanvos]

Sorry, bad communication.
Dealing with JavaCPP manually works, and everything is working end-to-end. This is non-ideal of course, e.g. for build and release management. So ideally, we want to build nd4j using the javacpp maven plugin.

In my previous comment, I describe how the manual process works, where we have a manual step at the end where the .o files got archived into a static library.
As far as I know, the nd4j build process will have javacpp to build a shared library for every component that requires a jni bridge (e.g. libjnind4j.so). That shared library is packaged in a jar, and the end-developer application just depends on that jar.

If we want to use the maven build process (which we do) with the proposed patch in the ci branch, we somehow need to add a step to the nd4j build process where at the end, all .o files are archived in a static library, and that library can then be packaged with any of the generated jars.

If this is too much ios (or static library)-specific work, I would be ok doing this last step manually.

Actually (but it might be better to create a separate issue for this?) the situation is slightly more complicated, as we typically build fat archives on ios that hold the code for {armv7, arm64, x86, x86_64}. That library is easily created with lipo, but it requires 4 non-fat libraries to be build first.

[saudet]

What about packaging .o files in JAR files? I'm pretty sure it would make it easier vs having to bundle and then extract duplicate functionality from .a files, but am I missing something?

[johanvos]

The java code somewhere calls System.loadLibrary(String name) and that will cause the JVM to search for a native library -- it won't work for .o files.

Thinking about it, for iOS we do extract the libraries before runtime, and use them to link with the ultimate executable (the app). We do that by scanning jars for libraries, but since we control that process, we can also scan jars for object files.

It sounds work.

[saudet]

Why would System.loadLibrary(String name) work for .a files, but not .o files? What does the name of the file have to do with the name of the JNI_OnLoad_libname() function?

Yes, it is my understanding that on iOS we have to link all libraries to a binary blob before we can execute anything. So your framework scans for .a files in all resources? How does it work for multiple architectures? On Android, the way it works is that it basically copies all .so files from resources under /lib/<archname>/ and that's it. If you can make that process similar for iOS, it would probably help port apps to iOS. So when creating the final binary for armv7, it would add to the linker line all .o and .a files found in /lib/armv7, for x86, everything under /lib/x86/, etc, or something similar. In no way does that involve JavaCPP! Unless there's a way to create multiarch .o files, that's probably the best option IMO.

[johanvos]

The VM and JNI specs talk about libraries only (dynamic and static), not about object files. System.loadLibrary does much more than simply invoking the onLoad (see http://hg.openjdk.java.net/jdk10/jdk10/jdk/file/777356696811/src/java.base/share/classes/java/lang/ClassLoader.java#l2545)

I already modified part of the implementation of the native implementation for loadLibrary, so we can make it work on iOS. It will fail on e.g. Linux, but that's ok as there we still provide real libraries.
Let me try that...

As for fat libraries: the libraries are built with lipo. After that, the linker will only include the symbols for the required architecture(s) into the executable.

Also note: our jfxmobile plugin creates the app for iOS and Android, so the end user doesn't have to change a single line of code. We bundle the libraries and resources on Android as well, but since we can deal with shared libraries there, I didn't have to change anything.

[saudet]

Ok, so I'm not following. If we can have "builtin libs", why can't we have multiple JNI_OnLoad() functions? And how do we define builtin libraries?

I see, "universal static libraries", so that's a thing, hum...

[johanvos]

The ci branch works perfect. Creation of object files works as expected, and I link them in a later phase into a static library.
I think you can close this issue?

[saudet]

Modifications included in newly released version 1.4. Thanks for the contribution!