Native library loading fails on openjdk7u4 for mac #6

Closed
xerial opened this Issue Sep 6, 2012 · 32 comments

Comments

Projects
None yet
9 participants
Owner

xerial commented Sep 6, 2012

What steps will reproduce the problem?

  1. Install openjdk7u4 preview for mac
  2. Invoke Snappy#compress

What is the expected output? What do you see instead?
I expect Snappy to work. But it fails to find the native library.

What version of the product are you using? On what operating system?
Snappy 1.0.4.1, OS X 10.7, OpenJDK7u4

Please provide any additional information below.
The openjdk guys have changed System.mapLibraryName to return an extension of "dylib" instead of "jnilib". It looks like this will be the standard for openjdk7 on mac. They have put some backwards compatibility hooks in System.loadLibrary, but this does not help the code in SnappyLoader.

The related code is in SnappyLoader:

    // Resolve the library file name with a suffix (e.g., dll, .so, etc.) 
    if (snappyNativeLibraryName == null)
        snappyNativeLibraryName = System.mapLibraryName("snappyjava");

    if (snappyNativeLibraryPath != null) {
        File nativeLib = new File(snappyNativeLibraryPath, snappyNativeLibraryName);
        if (nativeLib.exists())
            return nativeLib;
    }

    {
        // Load an OS-dependent native library inside a jar file
        snappyNativeLibraryPath = "/org/xerial/snappy/native/" + OSInfo.getNativeLibFolderPathForCurrentOS();

        if (SnappyLoader.class.getResource(snappyNativeLibraryPath + "/" + snappyNativeLibraryName) != null) {

On OpenJDK7 on Mac, it attempts to look for libsnappyjava.dylib. In the jar, it is packaged as libsnappyjava.jnilib, so it fails.

It seems the only reasonable fix is to check for both the dylib and jnilib extensions when on a Mac.

Migrated from http://code.google.com/p/snappy-java/issues/detail?id=39


earlier comments

james@tyrrells.com said, at 2012-06-26T04:06:14.000Z:

Duplicating "libsnappyjava.jnilib" with a .dylib extension fixed the issue for me too.

taroleo said, at 2012-06-30T12:04:12.000Z:

Issue 42 has been merged into this issue.

taroleo said, at 2012-06-30T12:04:26.000Z:

Issue 42 has been merged into this issue.

Any news on this issue? Among other things, it makes it impossible to run Cassandra on JDK 7 on OS X without workarounds. If @xerial has a good strategy for dealing with the root cause in mind, I would be interested in submitting a pull request with a fix.

Owner

xerial commented Sep 7, 2012

Having both .jnilib and .dylib in src/main/resources/org/xerial/snappy/native/Mac/x86_64 would be reasonable, since
the default JDK in Mac OS X Lion still uses .jnilib extension.

And building a native library for Mac with jdk7 would be possible by supplying an option:
$ make native LIBNAME=libsnappyjava.dylib

Fow now I simply duplicate the native library.

@xerial xerial added a commit that referenced this issue Sep 7, 2012

@xerial xerial Fix on issue #6 a1c36bb

@xerial fantastic. Would it be difficult to publish a new point release with this in? Thank you very much!

Owner

xerial commented Sep 7, 2012

I deployed snapshot release with this fix.

https://oss.sonatype.org/content/repositories/snapshots/org/xerial/snappy/snappy-java/1.0.5-SNAPSHOT/

Try snappy-java-1.0.5-20120907.023008-2.jar

I can confirm that Cassandra 1.1 starts fine when the Snappy jar is replaced with this snapshot.

Owner

xerial commented Sep 7, 2012

Thanks for the report.

Now I close this issue.

xerial closed this Sep 7, 2012

When you issue a new release, please leave a comment here. I will file an issue for Cassandra to update the dependency.

Owner

xerial commented Sep 11, 2012

A milestone release is here, which will be synchronized to Maven central repo in a few hours:
https://oss.sonatype.org/content/repositories/releases/org/xerial/snappy/snappy-java/1.0.5-M1/

I updated my app to 1.0.5-M1 but I am still hitting this issue. I am using embedded cassandra. I can try to create a repro

This fails for me: https://github.com/carllerche/snappy-repro


java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:317)
    at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:219)
    at org.xerial.snappy.Snappy.<clinit>(Snappy.java:48)
    at com.carllerche.snappyrepro.Main.main(Main.java:9)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:297)
    at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.UnsatisfiedLinkError: no snappyjava in java.library.path
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1860)
    at java.lang.Runtime.loadLibrary0(Runtime.java:845)
    at java.lang.System.loadLibrary(System.java:1084)
    at org.xerial.snappy.SnappyNativeLoader.loadLibrary(SnappyNativeLoader.java:52)
    ... 14 more
[WARNING] 
java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:297)
    at java.lang.Thread.run(Thread.java:722)
Caused by: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] null
    at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:229)
    at org.xerial.snappy.Snappy.<clinit>(Snappy.java:48)
    at com.carllerche.snappyrepro.Main.main(Main.java:9)
    ... 6 more

Here is the contents of the jar:

META-INF/MANIFEST.MF
META-INF/
META-INF/maven/
META-INF/maven/org.xerial.snappy/
META-INF/maven/org.xerial.snappy/snappy-java/
META-INF/maven/org.xerial.snappy/snappy-java/LICENSE
META-INF/maven/org.xerial.snappy/snappy-java/pom.properties
META-INF/maven/org.xerial.snappy/snappy-java/pom.xml
org/
org/xerial/
org/xerial/snappy/
org/xerial/snappy/OSInfo.class
org/xerial/snappy/Snappy.class
org/xerial/snappy/SnappyBundleActivator.class
org/xerial/snappy/SnappyCodec.class
org/xerial/snappy/SnappyError.class
org/xerial/snappy/SnappyErrorCode.class
org/xerial/snappy/SnappyException.class
org/xerial/snappy/SnappyInputStream.class
org/xerial/snappy/SnappyLoader.class
org/xerial/snappy/SnappyNative.class
org/xerial/snappy/SnappyNativeAPI.class
org/xerial/snappy/SnappyNativeLoader.bytecode
org/xerial/snappy/SnappyOutputStream.class
org/xerial/snappy/VERSION
org/xerial/snappy/native/
org/xerial/snappy/native/Linux/
org/xerial/snappy/native/Linux/amd64/
org/xerial/snappy/native/Linux/amd64/libsnappyjava.so
org/xerial/snappy/native/Linux/arm/
org/xerial/snappy/native/Linux/arm/libsnappyjava.so
org/xerial/snappy/native/Linux/armhf/
org/xerial/snappy/native/Linux/armhf/libsnappyjava.so
org/xerial/snappy/native/Linux/i386/
org/xerial/snappy/native/Linux/i386/libsnappyjava.so
org/xerial/snappy/native/Mac/
org/xerial/snappy/native/Mac/i386/
org/xerial/snappy/native/Mac/i386/libsnappyjava.jnilib
org/xerial/snappy/native/Mac/x86_64/
org/xerial/snappy/native/Mac/x86_64/libsnappyjava.dylib
org/xerial/snappy/native/Mac/x86_64/libsnappyjava.jnilib
org/xerial/snappy/native/README
org/xerial/snappy/native/Windows/
org/xerial/snappy/native/Windows/amd64/
org/xerial/snappy/native/Windows/amd64/snappyjava.dll
org/xerial/snappy/native/Windows/x86/
org/xerial/snappy/native/Windows/x86/snappyjava.dll
Owner

xerial commented Sep 27, 2012

Thanks for the report.

It looks like 1.0.5-M1 properly contains
org/xerial/snappy/native/Mac/x86_64/libsnappyjava.dylib
for open jdk for Mac but failed to load.

Does it work if you replace 1.0.5-M1 to a snapshot version 1.0.5-SNAPSHOT mentioned above?

Owner

xerial commented Sep 27, 2012

I confirmed the reported problem is reproduced in 1.7.0u4.jdk both in M1 and the snapshot version.

Owner

xerial commented Sep 27, 2012

It looks like SnappyLoader.class.getResource(...) cannot find .dylib file.
https://github.com/xerial/snappy-java/blob/develop/src/main/java/org/xerial/snappy/SnappyLoader.java#L440

Owner

xerial commented Sep 27, 2012

The cause of this problem is that openjdk7u4 returns "universal" as os.arch value, so
it tries to find
/org/xerial/snappy/native/Mac/universal/libsnappyjava.dylib
instead of
/org/xerial/snappy/native/Mac/x86_64/libsnappyjava.dylib

@xerial xerial added a commit that referenced this issue Sep 27, 2012

@xerial xerial Fixes issue #6 5a60c13
Owner

xerial commented Sep 27, 2012

Deployed snappy-java-1.0.5-M2 that fixes this issue. This release will be available in a few hours from Maven central.

I updated my simple repro case to M2, however that does not fix the issue for me.

xerial was assigned Sep 28, 2012

Owner

xerial commented Sep 28, 2012

I created a new snapshot release that at least works in my Mac 10.6 with openjdk7.
https://oss.sonatype.org/content/repositories/snapshots/org/xerial/snappy/snappy-java/1.0.5-M3-SNAPSHOT/

xerial reopened this Sep 28, 2012

@xerial xerial added a commit that referenced this issue Sep 28, 2012

@xerial xerial Fixes for issue #6 ed37dfa

That seems to work for me. Thanks

xerial closed this Sep 28, 2012

coltnz commented Dec 14, 2012

Hi is there a release planned that incorporates this fix anytime soon? The Cassandra project won't take a beta version -
https://issues.apache.org/jira/browse/CASSANDRA-4958

cheers
Colin

stfp commented Jan 2, 2013

Hey - Same question here; eagerly waiting for the next release 🎉

I have to upgrade to a -M release for this to work. When will an actual release (not -rc or -M) include this? 1.0.5 seems abandoned for 1.1.0, but neither is released, only in pre-release.

Owner

xerial commented May 1, 2013

@scottcarey If 1.0.5-M4 works well for your usage, I will create a new release version for 1.0.5 based on -M4 this week. I simply have been waiting bug reports.

Release of 1.1.0 needs some work to do, listed in milestone page: https://github.com/xerial/snappy-java/issues/milestones

For reference, I'm working dependency details for Apache Avro's next release: https://issues.apache.org/jira/browse/AVRO-1311 I'll see about changing to 1.0.5-M4 temporarily so others can test, then using a 1.0.5 final if that works out.

Its working for me on mac, and others on Avro in Ubuntu. I'd be interested in a release of 1.0.5. Are there changes in this version that are risky?

Owner

xerial commented May 17, 2013

Hi,

Thanks for testing the 1.0.5-M4 release.

I just released stable 1.0.5 version, which is available in Maven central
and is exactly the same with 1.0.5-M4 except the version number.

In version 1.1.0, we will introduce several changes in native library
loading mechanism and framed-format support, which is
added in the original snappy 1.1.0.

Regards,

Hi,

I have just checked the binary for released version 1.0.5 from google code as well as the tar.gz archive on github and it seems that it still does not include the Mac library with the .dylib extension. Could you please confirm the fix is in 1.0.5?

Cheers,
Vassilis

Owner

xerial commented Jul 24, 2013

Please use 1.0.5-M4.
https://github.com/xerial/snappy-java/tree/1.0.5-M4/src/main/resources/org/xerial/snappy/native/Mac/i386

This version has a fix that even if .dylib file is not found, trying to load .jnilib instead.

mdaxini commented Sep 10, 2013

It boggles me why the fix is in M4 but removed from the final release.

Here is a simple workaround in Mac with JDK 7 for anyone looking for answers in this thread. Add a system property:

-Dorg.xerial.snappy.lib.name=libsnappyjava.jnilib

Owner

xerial commented Sep 11, 2013

@mdaxini
The latest version is 1.1.0-M4 and it is quite close to the release version, since it only lacks FreeBSD native libs.
Please use this version.

Thanks.

mdaxini commented Sep 11, 2013

@xerial

Cannot use 1.1.0-M4 due to native libraries not being compatible with the c libraries on the available OS. No control over changing the OS at this time.

Thanks for the quick response.

Owner

xerial commented Sep 11, 2013

@mdaxini
Could you tell me your OS version (Mac OS X?)

I tested 1.1.0-M4 in Mac OS X 10.7 (Intell 64bit) and it works well.

dgerges commented Feb 5, 2014

An easy fix if you want to continue to use 1.4.0.1 on MacOSX with JDK 1.7

unzip snappy-java-1.0.4.1.jar
cd org/xerial/snappy/native/Mac/x86_64/
copy libsnappyjava.jnilib libsnappyjava.dylib
cd ../../../../../..
cp snappy-java-1.0.4.1.jar snappy-java-1.0.4.1.jar.old
jar cf snappy-java-1.0.4.1.jar org

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment