Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

native-image: support statically linking JNI libraries #3359

Open
smasher164 opened this issue Apr 18, 2021 · 16 comments
Open

native-image: support statically linking JNI libraries #3359

smasher164 opened this issue Apr 18, 2021 · 16 comments

Comments

@smasher164
Copy link

native-image can currently produce static or mostly static executables, except for object files loaded with System/loadLibrary when using JNI. This makes it inconvenient to distribute binaries for applications where the user doesn't have the JNI library present on their system.

The only alternative appears to be to build GraalVM from source with library linked, but I would prefer a solution where I could tell native-image a list of object files to statically link into the resulting binary.

@kristofdho
Copy link

You can do this using -H:AdditionalLinkerOptions=<path-to-your-lib>, not sure if that's strictly the right way to do it.
from the output of native-image --expert-options-all:
-H:AdditionalLinkerOptions="" String which would be appended to the linker call.

Alternatively you can write your own feature for your lib and tell native-image exactly what to do with it:

private static void linkSunEC(DuringAnalysisAccess a) {
NativeLibraries nativeLibraries = ((DuringAnalysisAccessImpl) a).getNativeLibraries();
/* We statically link sunec thus we classify it as builtIn library */
PlatformNativeLibrarySupport.singleton();
NativeLibrarySupport.singleton().preregisterUninitializedBuiltinLibrary("sunec");
/* and ensure native calls to sun_security_ec* will be resolved as builtIn. */
PlatformNativeLibrarySupport.singleton().addBuiltinPkgNativePrefix("sun_security_ec");
nativeLibraries.addStaticJniLibrary("sunec");
if (isPosix()) {
/* Library sunec depends on stdc++ */
nativeLibraries.addDynamicNonJniLibrary("stdc++");
}
}

@smasher164
Copy link
Author

Thanks for following up @kristofdho!
I made a repo for a minimal proof-of-concept here: https://github.com/smasher164/staticjni.
I passed in AdditionalLinkerOptions with the absolute path to the library, and I no longer compile the library with -shared, however I still see the following error:

root@8b3118f6d2a6:~/staticjni# ./helloworld
Exception in thread "main" java.lang.UnsatisfiedLinkError: no HelloWorld in java.library.path
	at com.oracle.svm.core.jdk.NativeLibrarySupport.loadLibraryRelative(NativeLibrarySupport.java:132)
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:275)
	at java.lang.Runtime.loadLibrary0(Runtime.java:830)
	at java.lang.System.loadLibrary(System.java:1871)
	at HelloWorld.main(HelloWorld.java:5)

(Note, I ran this example inside an ubuntu docker container)

Should I not be using loadLibrary? Or is there something I'm missing here?

@kristofdho
Copy link

kristofdho commented Apr 18, 2021

I'm having troubles getting it to work with -H:AdditionalLinkerOptions to the point that I'm doubting if I ever got it working that way..

The second example does work however. I am on windows btw, so I had to figure out some stuff before I could run your example.

In the end, to get it statically linked into the image, I had to add this custom feature to my classpath:

import com.oracle.svm.core.annotate.AutomaticFeature;
import com.oracle.svm.core.jdk.NativeLibrarySupport;
import com.oracle.svm.core.jdk.PlatformNativeLibrarySupport;
import com.oracle.svm.hosted.FeatureImpl;
import com.oracle.svm.hosted.c.NativeLibraries;
import org.graalvm.nativeimage.hosted.Feature;

@AutomaticFeature
public class HelloWorldFeature implements Feature {

    @Override
    public void beforeAnalysis(BeforeAnalysisAccess access) {
        NativeLibrarySupport.singleton().preregisterUninitializedBuiltinLibrary("HelloWorld");
        PlatformNativeLibrarySupport.singleton().addBuiltinPkgNativePrefix("HelloWorld");
        NativeLibraries nativeLibraries = ((FeatureImpl.BeforeAnalysisAccessImpl) access).getNativeLibraries();
        nativeLibraries.addStaticJniLibrary("HelloWorld");
    }
}

For this to compile, you need this dependency:

<dependency>
    <groupId>org.graalvm.nativeimage</groupId>
    <artifactId>svm</artifactId>
    <version>21.0.0</version>
    <scope>provided</scope>
</dependency>

After this, native-image will itself put HelloWorld.lib (since I'm on windows) in the linker command, so all we have to do is make sure the linker can find it, so I had to add -H:CLibraryPath="<path to folder containing library>" to the native-image command.

In the end, my native-image command looked like this:

native-image --no-fallback --no-server --initialize-at-build-time -H:Name=hello -cp HelloWorld.jar -H:CLibraryPath="<path to folder containing library>"

The part that is missing when using AdditionalLinkerOptions is probably the part where you tell native-image that the Java_HelloWorld* symbols are to be resolved internally, which the addBuiltinPkgNativePrefix call does. But don't quote me on that, it's merely a slightly educated guess.

@smasher164
Copy link
Author

Thanks @kristofdho, that works! Did you mean -jar HelloWorld.jar instead of -cp HelloWorld.jar in the native-image command? I've updated my repo accordingly.

@kristofdho
Copy link

Ah yes, I didn't have a manifest, so I actually used -cp HelloWorld.jar HelloWorld, but I only partially changed it back to your use case. Should indeed be -jar HelloWorld.jar. Glad I could help you!

@smasher164
Copy link
Author

Great. I'll go ahead and close this issue.

@smasher164
Copy link
Author

Actually, is it possible to put this information in the GraalVM docs somewhere? I understand the user must implement a Feature, but I feel that this is a common enough use-case to warrant its own place in the docs.

@smasher164 smasher164 changed the title native-image: support statically linking libraries with JNI native-image: document how to statically link libraries JNI libraries Apr 19, 2021
@kristofdho
Copy link

Unfortunately that's not something I can help you with. And strictly speaking, I think those calls are implementation details, and not part of the API. So for an actual API conform solution some code changes and indeed, definitely documentation, would be required.

@smasher164
Copy link
Author

Gotcha. In that case, would the original purpose of this issue still be valid? Since there is no stable (or at least a way we would want to document) way to currently do this.
I could imagine a flag like -H:RegisterBuiltin=<comma-delimited list of libraries to resolve internally>

@smasher164 smasher164 changed the title native-image: document how to statically link libraries JNI libraries native-image: support statically linking JNI libraries Apr 19, 2021
@munishchouhan
Copy link
Contributor

@olyagpl please check it out

@smasher164
Copy link
Author

I published a blog post about this method today, and mentioned the caveat that it's undocumented and unstable.
See https://www.blog.akhil.cc/static-jni.

@pquiring
Copy link

pquiring commented May 12, 2021

An alternative is to build a --shared library and then build your own loader with JNI libraries baked in. That's what I do now. The loader just has to registerNatives()

@chanseokoh
Copy link

chanseokoh commented May 12, 2021

@pquiring very interesting. Could you elaborate on how exactly you made it work in your way, step by step? Actually I don't have much experience with JNI, so I don't know what's a loader in this context, how you can build their own loader, how you make .so files baked in where (e.g., as "resources" for native-image like -H:IncludeResources=...)? And how did you confirm that the System.loadLibrary() loads JNI libraries from the baked-in .so files and not from the runtime env?

EDIT: and does it require having source code for .so files so that you can modify and build from source?

@pquiring
Copy link

@Noisyfox
Copy link
Contributor

Noisyfox commented Feb 7, 2022

Should I not be using loadLibrary? Or is there something I'm missing here?

For statically linked lib to make loadLibrary work there have to be an exported function called JNI_OnLoad_<library name>() according to JEP 178, or you can simply don't call loadLibrary at all, and the native method should simply work as long as the native function name matches.

This is what I've done in one of my project that using native-image:
https://github.com/multi-os-engine/moe-core/blob/7199da6723cc42c0f6ce4a9afc1fb91f03c1f6d1/moe.apple/moe.core.native/moe.sdk/src/MOE.mm#L313
https://github.com/multi-os-engine/moe-core/blob/7199da6723cc42c0f6ce4a9afc1fb91f03c1f6d1/moe.apple/moe.core.java/src/main/java/org/moe/MOE.java#L27

@iseki0
Copy link

iseki0 commented May 14, 2024

Does it easier to be used in Panama?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

8 participants