-
Notifications
You must be signed in to change notification settings - Fork 582
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
creating static library requires ar #213
Comments
We could add a platform.linker, but that assumes we have more than one
object file to link. We had problems with duplicate common functionality,
right?
|
There can be a single object file inside a library, but we still require an archiver to build the library. |
But what about the JNI functions for the base Pointer classes? Those are exported and will cause conflicts... As for the archiver, couldn't this be done anyway in the step where everything gets merged together? |
You're right about the common functions that are duplicated in every library (I forgot about those ones). I see 2 options here:
Creating the archive: I don't think that can be done in a single step (as you currently do to create a shared library), the ar step is really needed, afaik. |
Another option is to generate for all the dependencies in the same .cpp file. This is what I was assuming would be satisfactory for platforms like iOS, but is there any major downside to this approach? It all ends up in the same binary blob... If we end up having to produce multiple compilation units though, couldn't we call the archiver after and do |
That is an option as well, but it requires more bookkeeping and changes in the dl4j build process. In order to run deeplearning4j, I noticed javacpp was used to create native libs at at least 2 places:
Ultimately, it would be the easiest if the native libraries are not contained in separate jars. A single library that is created based on all the object files would be sufficient. I would be very ok with something like ar -rcs libeverything.a openblass.o nd4j.o as it makes it easy to include libeverything.a with the final executable on iOS. But we still have symbols in openblass.o that are also in nd4j.o, right? |
Everything in one place, we can easily do something like that, that I've once tried with RoboVM:
I don't think OpenBLAS and libnd4j have conflicting symbols, they shouldn't anyway. If they do, we will fix that in libnd4j. |
Actually, that is the flow I followed before. And it works. |
It would already work if Apple didn't remove the |
I've added a commit to the "ci" branch only for now, until we can say for sure that we are committed to this approach, so please check it out and let me know what you think. Basically, the idea is to generate all common functionality (that isn't inlined anyway) into I think it makes sense leaving them as .o files. It makes it easier since Maven, etc can consider them naturally as duplicate resources, which isn't the case if we archive them together in .a files, where we end up with linker errors if not further processed with native tools. Thoughts? |
Having the common functionality in a separate file is a good thing, I believe. Running javacpp manually (without the maven plugin) and generating .o files should work fine. Since an app may contain a number of jars that are expected to include a javacpp generated library, this means we need to keep the .o files until all processing is done. A fix would be an option in the javacpp plugin that allows to disable the generation of a library, and an additional step in the nd4j build sequence, where at the very end all .o's are archived in a static library. |
I thought you said you wanted it to work like other platforms where users
just need to include the dependencies with Maven and not deal with JavaCPP,
right? What changed your mind?
|
Sorry, bad communication. In my previous comment, I describe how the manual process works, where we have a manual step at the end where the .o files got archived into a static library. If we want to use the maven build process (which we do) with the proposed patch in the ci branch, we somehow need to add a step to the nd4j build process where at the end, all .o files are archived in a static library, and that library can then be packaged with any of the generated jars. If this is too much ios (or static library)-specific work, I would be ok doing this last step manually. Actually (but it might be better to create a separate issue for this?) the situation is slightly more complicated, as we typically build fat archives on ios that hold the code for {armv7, arm64, x86, x86_64}. That library is easily created with lipo, but it requires 4 non-fat libraries to be build first. |
What about packaging .o files in JAR files? I'm pretty sure it would make
it easier vs having to bundle and then extract duplicate functionality from
.a files, but am I missing something?
|
The java code somewhere calls System.loadLibrary(String name) and that will cause the JVM to search for a native library -- it won't work for .o files. Thinking about it, for iOS we do extract the libraries before runtime, and use them to link with the ultimate executable (the app). We do that by scanning jars for libraries, but since we control that process, we can also scan jars for object files. It sounds work. |
Why would Yes, it is my understanding that on iOS we have to link all libraries to a binary blob before we can execute anything. So your framework scans for .a files in all resources? How does it work for multiple architectures? On Android, the way it works is that it basically copies all .so files from resources under |
The VM and JNI specs talk about libraries only (dynamic and static), not about object files. System.loadLibrary does much more than simply invoking the onLoad (see http://hg.openjdk.java.net/jdk10/jdk10/jdk/file/777356696811/src/java.base/share/classes/java/lang/ClassLoader.java#l2545) I already modified part of the implementation of the native implementation for loadLibrary, so we can make it work on iOS. It will fail on e.g. Linux, but that's ok as there we still provide real libraries. As for fat libraries: the libraries are built with lipo. After that, the linker will only include the symbols for the required architecture(s) into the executable. Also note: our jfxmobile plugin creates the app for iOS and Android, so the end user doesn't have to change a single line of code. We bundle the libraries and resources on Android as well, but since we can deal with shared libraries there, I didn't have to change anything. |
Ok, so I'm not following. If we can have "builtin libs", why can't we have multiple I see, "universal static libraries", so that's a thing, hum... |
The ci branch works perfect. Creation of object files works as expected, and I link them in a later phase into a static library. |
Modifications included in newly released version 1.4. Thanks for the contribution! |
When creating a static library, we typically use 2 steps:
clang++ -c foo.cpp -o foo.o
ar -rcs libfoo.a foo.o
Currently, Builder.compile() uses a single command to create a dynamic library.
Maybe an additional property platform.libtool could be used?
The text was updated successfully, but these errors were encountered: