Package native library files in Maven release

[andrewmwells-amazon]

Category

Other

Describe the feature you'd like to request

It would be nice if we packaged native library files in the Maven release so people don't have to download them from github.

Describe the solution you'd like

It looks like we could follow https://stackoverflow.com/a/10071519 to include the native libraries in the Maven release.

Describe alternatives you've considered

Current system.

Additional context

No response

Is this something that you'd be interested in working on?

• 👋 I may be able to implement this feature request
• ⚠️ This feature might incur a breaking change

[exceptionfactory]

@andrewmwells-amazon I am interested in adding native library bundling to cedar-java, and I would be glad to put together a pull request if that would be helpful.

In terms of supported operating systems and architectures, what should be prioritized for initial implementation? Starting with Linux and x86_64/amd64 seems like a minimum requirement. The popularity of macOS and aarch64/arm64 is also worth considering. Should supporting Windows be included in the initial scope?

As far as runtime implementation, the Stack Overflow discussion has a helpful example of direct code that reads JVM System properties and loads the library from a preset path in the JAR. The Fizzed Java Native Environment library also seems worth considering, as it is licensed under Apache-2.0, only depends on SLF4J API, and provides a simple API with convention-based storage of libraries to be loaded. Is there a preferred approach in terms of direct implementation versus such a library?

The last major question relates to the build approach itself. Regardless of the build strategy, the end result requires bundling the platform-specific binaries from a common location. GitHub Actions provides runners for Linux, macOS, and Windows, and just recently introduced macOS 14 with M1 processors, which would allow building for aarch64. It seems like it would be helpful to have a build process that also works a local machine, provided certain required components are installed. This could be implemented through a shell script, or integrated more directly as a task in the Gradle build. Part of the challenge includes how to orchestrate the multi-platform builds for the native Rust library.

To help move this forward, I created a branch with that uses the Fizzed JNE library and builds a JAR that contains both Linux x86_64 and macOS aarch64 binaries using Maven to execute Cargo build commands, running on a GitHub workflow with macOS 14. The following commit contains the initial set of changes:

exceptionfactory@be785ca

I used Maven to avoid changes to the Gradle build configuration, but the Cargo build execution could be handled just as easily in Gradle, or perhaps a preparatory shell script. I replaced the separate System.load() calls with a shared NativeLibraryLoader utility that calls the JNE.loadLibrary() method. For compatibility, it also checks for the currently-defined environment variable, which would provide a workaround for platforms that are not bundled in the JAR itself.

I am not very familiar with all of the build options for Rust, but after evaluating cross and its Docker-based approach, I went with cargo-zigbuild to build the Linux x86_64 binary on the macOS 14 runner.

With this background, I would be glad to know what approach would be preferred so that this could be turned into a pull request for review.

[exceptionfactory]

@khieta Thanks for the recent reviews of several pull requests.

Do you have any initial feedback on a preferred way forward for native library bundling in the cedar-java JAR?

If it would be helpful to consider this in the context of a pull request, I can look at putting something together that integrates with the current Gradle build.

[khieta]

Thanks for reaching out @exceptionfactory! I've pinged some of our collaborators who have more Java expertise than myself. I'll ask them to take a look 🙂

[lucajone]

Hi @exceptionfactory, and thanks a lot for building this out! I agree if we can package the native dependencies conveniently then it massively reduces the friction for folks to use and then deploy the library.

Packaging all the native libraries in the JAR is definitely the easiest option for users. I've not used the JNE library you mention before, but it does look like it handles most of the legwork for us – if it ends up not being suitable later on, I think our use case is probably on the simpler end. I agree that we should keep the option around to manually specify a library path with the existing environment variable, thanks for keeping that in.

One concern I have going forward though is about the size of the resulting JAR. If we anticipate supporting in the medium term the following platforms:

• Windows x86-64,
• Linux x86-64 and ARM (for Graviton),
• macOS ARM and possibly x86-64

and we include all of these in a single JAR, then it could get quite large. On my laptop, a release build of CedarJavaFFI results in a .dylib that's 8.6MB. Rounding up a bit to account for future growth, including all of the above platforms brings us into 45-50MB territory, which is quite chunky. Do you know if there's any way we can separate these out into different JARs in Maven, and still have dependencies resolve sensibly? (It looks like there's some support).

Cross-compiling the Linux artifacts is neat! That might make things a bit easier at build time. However, I would suggest that as part of any release process that we have to run integration tests using the JAR and the native libraries in the JAR on each platform, just to make sure they all work. If we have sufficient testing, I wonder whether one runner could just compile all the artifacts – I don't think there's anything stopping us from cross-compiling ARM targets from an x86 host in principle?

As a minor point, I think it would be nicer to keep as much of the build logic in the one place as possible. If we can get a Gradle task to build the JARs that might be more consistent (it also means less XML!).

All in all, thanks again for putting this together! I'll leave it with the Cedar folks, but I think a pull request for a build target to produce a combined JAR with native artifacts would be a useful addition. We might need more iteration around testing before we move to publishing them automatically to a Maven repository or similar, but this is a great step on that road which reduces the barrier to entry quite a bit.

[exceptionfactory]

Thanks for the helpful reply @lucajone!

I agree with you about the size concern for bundling platform-specific libraries. Use of the classifier approach is a good solution that would allow integrating services to decide on which architectures are required, while keeping the bundle small. The conscrypt library is one example that produces both platform-specific JAR files and an uber JAR that bundles all supported platforms. Adoption of the uber JAR seems more common as it just works in most common environments.

Although the uncompressed size of each native library is 9 or 10 MB, the compression ratio for those files in pretty good. For example, the cedar-java package I published for testing includes an 8 MB native library for macOS ARM and a 13 MB native library for Linux x86-64, but resulting JAR file is only 6.6 MB. That is not small, but it means a JAR for the most commons platforms should be closer to 20 MB than 50 MB.

I agree with you that running integration tests on the native platforms would be ideal. GitHub Action workflows should be able to support this approach, so it is really just an orchestration question. One approach could be building the cross-compiled JAR on one runner, and then having runners with each OS (Linux, macOS, and Windows) pull the JAR and run tests. The cross-compilation options for Rust open up various possibilities, so it really just seems to be a better of deciding on the build host environment, and then structuring the rest of the build accordingly. If the goal is to provide the greatest amount of flexibility, Gradle tasks could be implemented to call the appropriate commands based on the OS running the build.

I will take a closer look at putting together a pull request soon to keep the discussion moving forward.

[lucajone]

The conscrypt approach looks like exactly what I was hoping for, and from what you're saying it sounds like it is fairly straightforward for us to vend platform-specific JARs and a bundled JAR. Excellent! I had completely forgotten about compression; 20MB is much more reasonable, and seems pretty feasible to offer as a convenience if we've also got the option to publish per-platform artifacts as well.

Presuming the cross-compilation is stable enough, I think the suggestion to build in one place and just test in others makes sense since it saves us a step at the end to build the final JAR. But it's much of a muchness – I think to begin with, having some Gradle targets to build JARs with the libraries included (cross- or native-compiled) would be a good place to start and we can iterate on the different combinations as they become useful.

[john-h-kastner-aws]

Is this issue resolved? #105 is merged and it looks like the 3.1.2 maven release includes cedar-java-3.1.2-uber.jar

[cdisselkoen]

I think this is resolved, primarily by #104. I'm going to go ahead and close this; if there are any remaining tasks or things not yet done, let's open separate issues for those.