feat(loader): bundle native library inside the JAR#77
Conversation
Rework NativeLibraryLoader so it can load datafusion_jni from a
JAR-bundled native library instead of requiring java.library.path:
1. Try System.loadLibrary so operators can still override with a
system-installed build via java.library.path / LD_LIBRARY_PATH.
2. On UnsatisfiedLinkError, detect the host OS/arch, look up the
bundled resource at
org/apache/datafusion/<os>/<arch>/lib<name>.<ext>, extract it to
$TMPDIR/datafusion-java/<sha256>/ and load via System.load.
Concurrent JVMs sharing the temp directory converge on the same SHA-256
hash directory; the extraction uses ATOMIC_MOVE so racing writers don't
clobber each other.
Wire core/pom.xml to copy the host's locally built native lib from
native/target/<profile>/lib<name>.<ext> into target/classes at the
matching resource path, so the produced JAR works out of the box on the
build host. Per-platform Maven profiles set the OS/arch directory
segments and library filename; -Ddatafusion.native.profile=release
switches the copy source from debug to release.
Drop -Djava.library.path from the surefire and exec-maven-plugin
argLines so the test path now exercises the same resource-extraction
code path users will hit.
Refs apache#33 (work items 1 and 3); cross-build CI, Sonatype publishing, and
release docs remain as follow-ups.
|
@pgwhalen fyi |
| <property name="datafusion.native.lib.source" | ||
| value="${maven.multiModuleProjectDirectory}/native/target/${datafusion.native.profile}/${datafusion.lib.filename}"/> | ||
| <fail message="Native library not found at ${datafusion.native.lib.source}. Run 'cd native && cargo build' (or 'make') before building the JAR."> | ||
| <condition><not><available file="${datafusion.native.lib.source}"/></not></condition> | ||
| </fail> |
There was a problem hiding this comment.
On Windows, none of the new native profiles defines datafusion.lib.filename, datafusion.lib.os, or datafusion.lib.arch, so this path remains unresolved and the new process-classes fail check aborts mvn test/package even after cargo build creates native/target/debug/datafusion_jni.dll. This regresses the previous java.library.path flow before NativeLibraryLoader can fall back to System.loadLibrary. One fix is adding a Windows profile mirroring the Linux/Mac ones: <datafusion.lib.os>windows</datafusion.lib.os>, <datafusion.lib.arch>amd64</datafusion.lib.arch>, <datafusion.lib.filename>datafusion_jni.dll</datafusion.lib.filename>. Even though no Windows binary will be bundled today, the build at least stops failing, or skipping the copy step on unsupported platforms.
2 participants
Which issue does this PR close?
Rationale for this change
NativeLibraryLoadercurrently callsSystem.loadLibrary("datafusion_jni"), so consumers need to setjava.library.pathor install the.so/.dylibsystem-wide before importing the library. That's a barrier to publishing a Maven-Central artifact that just works.This PR adds the runtime piece of the fat-JAR design from #33: the loader can pull a platform-specific native library out of the JAR's resource tree, and the Maven build copies the locally-built
datafusion_jniinto that tree before packaging.What changes are included in this PR?
NativeLibraryLoader:System.loadLibraryfirst so users can still override the bundled library viajava.library.path/LD_LIBRARY_PATH.UnsatisfiedLinkError, detects the host OS/arch, looks uporg/apache/datafusion/<os>/<arch>/lib<name>.<ext>on the classpath, extracts it to$TMPDIR/datafusion-java/<sha256>/, and loads viaSystem.load. Hash-based directory naming lets concurrent JVMs sharing a temp directory converge on the same file; extraction usesATOMIC_MOVEto avoid racing writers clobbering each other.Platformclass that maps the OS/arch combinations from the design in Publish fat JAR with platform-specific native libraries to Maven Central #33:linux/amd64,linux/aarch64,darwin/x86_64,darwin/aarch64(with a stub forwindows/.dllto make the future Windows port a build-matrix change).Maven (
core/pom.xml):datafusion.lib.os/datafusion.lib.arch/datafusion.lib.filename.maven-antrun-pluginexecution atprocess-classescopiesnative/target/<profile>/lib<name>.<ext>intotarget/classes/org/apache/datafusion/<os>/<arch>/lib<name>.<ext>, failing fast with a clear message if the native crate hasn't been built.-Ddatafusion.native.profile=releaseswitches the source fromdebugtorelease.-Djava.library.path=...from the surefire argline (and the equivalent argument from the examples module'sexec-maven-plugin) so the test path now exercises the same resource-extraction code path users will hit.--add-opens=java.base/java.nio=ALL-UNNAMEDstays.Are these changes tested?
PlatformTestcovers OS detection (linux/darwin/windows), arch detection acrossamd64/x86_64/x64/aarch64/arm64aliases, unsupported-platform error handling, library filename mapping per OS, and resource paths matching the spec in Publish fat JAR with platform-specific native libraries to Maven Central #33 exactly.-Djava.library.pathis gone). Full localmvn testis green on macOS aarch64 (229 tests, 13 skipped, 0 failures). Verified that the JAR containsorg/apache/datafusion/darwin/aarch64/libdatafusion_jni.dyliband that the extracted copy lands under$TMPDIR/datafusion-java/<sha256>/.Other host platforms (linux/amd64, linux/aarch64, darwin/x86_64) will be exercised once the cross-build CI matrix lands.
Are there any user-facing changes?
The
NativeLibraryLoader.loadLibrary()public API is unchanged. Behavior is strictly broader — code that previously neededjava.library.pathset still works, and code that depended only on the bundled lib now works too.