Support for graal native-image

[heuermh]

Hello,

I am tracking a runtime error with zstd-jni when attempting to build a GraalVM native-image that includes com.github.luben:zstd-jni:jar:1.4.4-7:runtime as a dependency.

heuermh/dishevelled-bio#24

I am still rather new to native-image, and may be missing some configuration or other obvious fix
https://github.com/oracle/graal/blob/master/substratevm/JNI.md

[luben]

I have not tried it with graal native images, I don't understand what the error in the linked issue means. Any help with deciphering it will be of great help.

[heuermh]

Thanks for the comment, @luben!

I am waiting for advice on the native-image Slack channel, will keep this updated if I find anything that works.

[luben]

One thing you can try is extract the libzstd-jni for your architecutre from the jar and put it on your LD_LIBRARY_PATH, e.g. in /usr/local/lib - the java loader always try to fall-back to locally installed library. (your paths on mac can be different)

[danielkec]

Would be great if Native.load() was not invoked in static block, it's quite ugly to substitute that for Graal's native image oracle/graal#439 (comment)

zstd-jni/src/main/java/com/github/luben/zstd/ZstdOutputStream.java

Lines 15 to 17 in a4926bb

	static {
	Native.load();
	}

[heuermh]

Hello @danielkec, would such a change allow for a workaround?

I've since updated to version 1.4.5-12 and this is still an open issue.

[luben]

It's may be easy to replace the static Native.load with call from the constructor but it may be harder to do it in the Zstd that is all static methods. I will try.

[danielkec]

I did that lazy loading when ZstdOutputStream is constructed and it seems to be working well, but my use case is limited to Kafka client needs. https://github.com/oracle/helidon/pull/2555/files#diff-e753f841f896d9d0153523f5980d3bac292135793c9eacfab8fe2a082c41665bR76-R92

[luben]

Then I will make it lazy for all the non-static classes. This should cover most of the use-cases, though the Zstd class (collection of static methods) will not be covered - it will still load the native part on in static section. I think that should be fine - if you don't use the class it will not run the static section.

[danielkec]

@luben That would be super cool 👍

[luben]

@danielkec , I have just released v1.4.7-1 and it moves the loading of the native library from the static sections to the constructors. Though it does not cover the Zstd class that just holds a bunch of static methods.

[luben]

Hmm, this breaks apache commons/compress. I will revert and release v1.4.7-2 and will explore independently.

[siwee]

Hi, @luben is there a new update for this issue?

Thank you.

[luben]

So the lazy loading didn't work. Another option that I can thing of is to let you load manually the native library and tell zstd-jni.Native to assume it's loaded. Is this going to work for you?

Edit: added it - a3b5c4c , can you test and see if it lets you workaround the loading issue?

[linghengqian]

• I noticed this issue while tracking Make ShardingSphere Proxy in GraalVM Native Image form available apache/shardingsphere#21347 (comment) . However, https://github.com/apache/shardingsphere/actions/runs/3303870293/jobs/5452309095 builds GraalVM Native Image normally.

• There are just some warnings related to zstd-jni's classes. I'm assuming that if we don't need to maintain GraalVM reachability metadata in zstd-jni, to completely solve the warning just need to submit some necessary GraalVM reachability metadata to https://github.com/oracle/graalvm-reachability-metadata ?

Warning: Could not resolve class com.github.luben.zstd.ZstdCompressCtx for reflection configuration. Reason: java.lang.ClassNotFoundException: com.github.luben.zstd.ZstdCompressCtx.
Warning: Could not resolve class com.github.luben.zstd.Zstd for reflection configuration. Reason: java.lang.ClassNotFoundException: com.github.luben.zstd.Zstd.

[luben]

I don't have any experience with GraalVM. I don't mind maintaining our own reachability data in the zstd-jni package, but I will need some help with the initial setup/version.

[linghengqian]

I don't have any experience with GraalVM. I don't mind maintaining our own reachability data in the zstd-jni package, but I will need some help with the initial setup/version.

• One of the main challenges I faced with submitting PRs was that I was not familiar with Gradle (especially when I saw zstd-jni's build.gradle), of course due to the inclination of the GraalVM community to gradually start using it actively, I'm not sure I am familiar with how long it takes. . . . 😂

• Gradle Pugin of GraalVM Native Tools allows to generate graalvm reachability metadata by configuring GraalVM Trace Agent when running unit tests, so that the native-image component of GraalVM can discover those reflection information. The maven equivalent of this process should be something like Add GraalVM native-image configuration brettwooldridge/HikariCP#1959 or Automatically generate native-image conditional metadata for ChannelHandler implementations. netty/netty#12794. One of the reasons their configuration is so cumbersome is that Add agent modes to Maven plugin graalvm/native-build-tools#260 is not closed, porting the Agent Mode of the Gradle plugin to the Maven plugin seems to be difficult.

• If it were me, I would like to rewrite some unit tests for zstd-jni at https://github.com/oracle/graalvm-reachability-metadata , and then use some template code to generate GraalVM reachability metadata json file for these unit tests.🤣

[linghengqian]

• Maybe I'm understanding something wrong, maybe there's nothing that needs to be handled in zstd-jni when building a GraalVM Native Image. Netty itself uses the zstd-jni library, and the real reason for the warning may be that it generates the wrong GraalVM reachability metadata by itself. Refer to Maybe need to add more detailed metadata for io.netty:netty-* oracle/graalvm-reachability-metadata#89 (comment) . In fact, all applications based on Netty 4.1.84.Final are normal, such as building a GraalVM Native Image corresponding to a Webflux application based on Springboot OSS 3.0.0-RC1 and netty 4.1.84.Final.

[luben]

BTW, the gradle build is just for the Android aars, the JAR build is handled by the sbt.

If I understand correctly, the metadata is needed if somebody uses runtime reflection. In the zstd-jni we don't use any reflection, so may be you are right that netty (if it reflects over zstd-jni classes) should declare its reflection targets.

[linghengqian]

• Yes, GraalVM reachability metadata only expects to record runtime reflections, as https://www.graalvm.org/22.2/reference-manual/native-image/metadata/ says.

• Of course, problems like log4j2 have had little to do with GraalVM reachability metadata, since it uses java.lang.invoke.LambdaMetafactory which GraalVM does not yet support.😆

• From my point of view, there is nothing that can be done in zstd-jni library because GraalVM Native Build Tools does not provide sbt plugin. It looks like this issue can be closed as it seems that all the issues involved can be resolved outside of zstd-jni. All the problems I see come from the configuration of the GraalVM Native Build Tools in Netty itself. 😄

[Ampflower]

For runtime initialised, simply including the binary for the platform it's running on and declaring in jni-config.json: srcPos and dstPos for the I/O streams without finalizers works.

If you're curious what that'd look like:

[
  {
    "name": "com.github.luben.zstd.ZstdOutputStreamNoFinalizer",
    "fields": [
      {
        "name": "dstPos"
      },
      {
        "name": "srcPos"
      }
    ]
  },
  {
    "name": "com.github.luben.zstd.ZstdInputStreamNoFinalizer",
    "fields": [
      {
        "name": "dstPos"
      },
      {
        "name": "srcPos"
      }
    ]
  }
]

Although, for build-time initialized, I've been working on reimplementing the JNI methods in Java for native-image use that only requires the system binary. Would there be any interest in upstreaming it and hooking it up with testing?

[linghengqian]

@KJP12 I can proactively submit some necessary GraalVM metadata to https://github.com/oracle/graalvm-reachability-metadata , for Gradle there is an Agent integration of Native Build Tools to handle unit tests. I've never used sbt, could you provide a unit test for this part of the metadata? Or do you intend to submit a PR?

[Ampflower]

I was thinking of maybe pushing the metadata here with a bit extra for not retaining the fields if the classes aren't reachable, since it'd probably be a decent place for that? Tho the central reachability repository should be fine too.

I'm not really experienced with getting testing done with native image, so I am not sure if I can PR that in immediately.

[luben]

Hi,

Sorry for the long delays on my side. I am open to PRs to add the metadata here. Having it in the https://github.com/oracle/graalvm-reachability-metadata will also work, please tag me if you send a PR there.

Regarding bindings directly to libzstd - I think this will be a fine approach but we have to drop support for old JVMs as far as I can tell. If we do that we may refactor and cleanup the whole library as it has accumulated a lot of cruft due to mistakes, changes in libzstd, and the goal to keep backward compatibility.

[heuermh]

Same here! I'm willing to test whatever approaches might work.

In the original reported issue I am using zstd through commons-compress, which adds an additional layer to consider.

[Ampflower]

Regarding bindings directly to libzstd - I think this will be a fine approach but we have to drop support for old JVMs as far as I can tell. If we do that we may refactor and cleanup the whole library as it has accumulated a lot of cruft due to mistakes, changes in libzstd, and the goal to keep backward compatibility.

The method for Graal native-image using its C API wouldn't require any level of rewriting, beyond translating the JNI code to Java, or dropping support, just substitutions and calling APIs from there, which would not be active outside of native-image. If you feel it'd be best to rewrite it so that mistakes can be ironed out first, I could maintain it independently in the meantime.

Panama is a different thing that would require more intervention tho, and given that there is no LTS JDK with it included, that would have to hold off for a bit longer.

[luben]

@KJP12 , thanks for clarifying, yes, I was thinking about Panama. I am open to get make it work seamlessly on Graal native.

There are some mistakes, e.g. ZstdOutputStream vs ZstdOutputStreamNoFinalizer, but there is nothing major. I think the risk of rewrite from scratch would be greater than continue with the current structure. So if we can keep the backward compatibility, we don't need to rewrite.

[linghengqian]

• For someone like me who has never used scala in gradle, it seems difficult to see a small unit test, for me https://github.com/luben/zstd-jni/tree/master/src/test/resources is surprisingly large. I even learned for the first time because of doing this test that Windows downloaded files cannot be dragged directly into IntelliJ IDEA via WSLg 🫠

• Anyway, I don't really see any GraalVM reachability metadata being generated, it really exists need unsolicited GraalVM reachability metadata?

• You can enter https://github.com/linghengqian/graalvm-trace-metadata-smoketest , and then execute the following command to verify.

sdk use java 22.3.r17-grl
cd ./zstd-jni/
sdk install scala 2.13.10
./gradlew -Pagent clean test
./gradlew metadataCopy --task test --dir src/main/resources/META-INF/native-image/com.github.luben/zstd-jni/1.5.2-5

• 

[luben]

@linghengqian
I am not sure what you are talking about:

• you should be able to use zstd-jni without bringing the testing code and dependencies - testing artifacts are large because we keep compatibility with Zstd-0.4 and we bring files compressed with these versions in order to be able to test against them.
• yes, there is no graal reachability data generate yet - we are discussing what's the best home for it in this thread.
• I don't understand the suggestion - I have zero interest in supporting graal, but I am open to accept contributions to help people that have different needs. If you are willing to contribute, please send a PR.

[linghengqian]

@luben

• The reason I use tests is because the GraalVM reachability metadata is generated with proper unit tests. Neither my personal test nor the test of the main branch of zstd-jni can find the part of the metadata mentioned by @KJP12 .

[
  {
    "name": "com.github.luben.zstd.ZstdOutputStreamNoFinalizer",
    "fields": [
      {
        "name": "dstPos"
      },
      {
        "name": "srcPos"
      }
    ]
  },
  {
    "name": "com.github.luben.zstd.ZstdInputStreamNoFinalizer",
    "fields": [
      {
        "name": "dstPos"
      },
      {
        "name": "srcPos"
      }
    ]
  }
]

• What I'm referring to is that this piece of GraalVM reachability metadata is not known to be required in unit tests. If we can run through the oracle tck plugin task without adding GraalVM reachability metadata, then obviously we don't need to submit it to https://github.com/oracle/graalvm-reachability-metadata .

[luben]

Sorry for the delay. I think it would be easier to write a few simple happy-path unit tests in Java and try to validate the GraalVM metadata with them. I am not sure what is the support for GraalVM in Scala.

[linghengqian]

@luben I'm just curious, besides https://github.com/luben/ZstdAndroidExample , does zstd-jni have a document or something for my reference? I don't seem to see it in the README.

[luben]

There are javadocs: https://www.javadoc.io/doc/com.github.luben/zstd-jni/latest/com/github/luben/zstd/package-summary.html

And if it's about writing tests, I think it would be easy to rewrite some of the Scala tests in Java

[linghengqian]

• I'm sorry for what I said above, I absolutely believe it's because I have absolutely no idea how to use GraalVM Native Build Tools with Scala under Gradle, it looks like the Scala plugin for IntelliJ IDEA has no idea what's going on.
• In fact, I have neither used Gradle nor Scala before contacting the current issue. So instead of trying to translate Scala to Java I wrote my own unit tests under Java.
• The current issue is meaningless, let's go back to Add support for com.github.luben:zstd-jni:1.5.2-5 oracle/graalvm-reachability-metadata#121.

[luben]

Thanks a lot for your work! I put some comments on the PR - mostly how we can make the refection configs more generic so they could apply to other versions/OS-es

[linghengqian]

Thanks a lot for your work! I put some comments on the PR - mostly how we can make the refection configs more generic so they could apply to other versions/OS-es

• On Add support for com.github.luben:zstd-jni:1.5.2-5 oracle/graalvm-reachability-metadata#121 (comment) , I leave a message for the treatment of other platforms.

• My development environment is only WSL under Windows 11, so I only have GraalVM reachability metadata for linux/amd64 at the end of my unit tests. Due to my personal habit, I did not install the tool chain required by native-image such as msvc in windows, and I did not have a Macbook.

• Other friends can submit a PR after the metadata be generated directly , like Add metadata for Netty MacOS specific nameserver resolution oracle/graalvm-reachability-metadata#103 did.

[luben]

@linghengqian, thanks a lot for your work! Let hope oracle/graalvm-reachability-metadata#121 is merged soon.

[linghengqian]

• Oracle employees are used to reviewing PR batch by batch and then releasing it directly. Now can only wait.

[luben]

it just got merged: oracle/graalvm-reachability-metadata#121

[luben]

Closing as it is solved by oracle/graalvm-reachability-metadata#121

[heuermh]

Thank you, @luben! I'll try to follow the advice above to solve my initial issue.