Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Native library loading fails on openjdk7u4 for mac #6

Closed
xerial opened this issue Sep 6, 2012 · 32 comments

Comments

@xerial
Copy link
Owner

commented Sep 6, 2012

What steps will reproduce the problem?

  1. Install openjdk7u4 preview for mac
  2. Invoke Snappy#compress

What is the expected output? What do you see instead?
I expect Snappy to work. But it fails to find the native library.

What version of the product are you using? On what operating system?
Snappy 1.0.4.1, OS X 10.7, OpenJDK7u4

Please provide any additional information below.
The openjdk guys have changed System.mapLibraryName to return an extension of "dylib" instead of "jnilib". It looks like this will be the standard for openjdk7 on mac. They have put some backwards compatibility hooks in System.loadLibrary, but this does not help the code in SnappyLoader.

The related code is in SnappyLoader:

    // Resolve the library file name with a suffix (e.g., dll, .so, etc.) 
    if (snappyNativeLibraryName == null)
        snappyNativeLibraryName = System.mapLibraryName("snappyjava");

    if (snappyNativeLibraryPath != null) {
        File nativeLib = new File(snappyNativeLibraryPath, snappyNativeLibraryName);
        if (nativeLib.exists())
            return nativeLib;
    }

    {
        // Load an OS-dependent native library inside a jar file
        snappyNativeLibraryPath = "/org/xerial/snappy/native/" + OSInfo.getNativeLibFolderPathForCurrentOS();

        if (SnappyLoader.class.getResource(snappyNativeLibraryPath + "/" + snappyNativeLibraryName) != null) {

On OpenJDK7 on Mac, it attempts to look for libsnappyjava.dylib. In the jar, it is packaged as libsnappyjava.jnilib, so it fails.

It seems the only reasonable fix is to check for both the dylib and jnilib extensions when on a Mac.

Migrated from http://code.google.com/p/snappy-java/issues/detail?id=39


earlier comments

james@tyrrells.com said, at 2012-06-26T04:06:14.000Z:

Duplicating "libsnappyjava.jnilib" with a .dylib extension fixed the issue for me too.

taroleo said, at 2012-06-30T12:04:12.000Z:

Issue 42 has been merged into this issue.

taroleo said, at 2012-06-30T12:04:26.000Z:

Issue 42 has been merged into this issue.

@michaelklishin

This comment has been minimized.

Copy link

commented Sep 7, 2012

Any news on this issue? Among other things, it makes it impossible to run Cassandra on JDK 7 on OS X without workarounds. If @xerial has a good strategy for dealing with the root cause in mind, I would be interested in submitting a pull request with a fix.

@xerial

This comment has been minimized.

Copy link
Owner Author

commented Sep 7, 2012

Having both .jnilib and .dylib in src/main/resources/org/xerial/snappy/native/Mac/x86_64 would be reasonable, since
the default JDK in Mac OS X Lion still uses .jnilib extension.

And building a native library for Mac with jdk7 would be possible by supplying an option:
$ make native LIBNAME=libsnappyjava.dylib

Fow now I simply duplicate the native library.

xerial added a commit that referenced this issue Sep 7, 2012

@michaelklishin

This comment has been minimized.

Copy link

commented Sep 7, 2012

@xerial fantastic. Would it be difficult to publish a new point release with this in? Thank you very much!

@xerial

This comment has been minimized.

Copy link
Owner Author

commented Sep 7, 2012

I deployed snapshot release with this fix.

https://oss.sonatype.org/content/repositories/snapshots/org/xerial/snappy/snappy-java/1.0.5-SNAPSHOT/

Try snappy-java-1.0.5-20120907.023008-2.jar

@michaelklishin

This comment has been minimized.

Copy link

commented Sep 7, 2012

I can confirm that Cassandra 1.1 starts fine when the Snappy jar is replaced with this snapshot.

@xerial

This comment has been minimized.

Copy link
Owner Author

commented Sep 7, 2012

Thanks for the report.

Now I close this issue.

@xerial xerial closed this Sep 7, 2012

@michaelklishin

This comment has been minimized.

Copy link

commented Sep 7, 2012

When you issue a new release, please leave a comment here. I will file an issue for Cassandra to update the dependency.

@xerial

This comment has been minimized.

Copy link
Owner Author

commented Sep 11, 2012

A milestone release is here, which will be synchronized to Maven central repo in a few hours:
https://oss.sonatype.org/content/repositories/releases/org/xerial/snappy/snappy-java/1.0.5-M1/

@carllerche

This comment has been minimized.

Copy link

commented Sep 26, 2012

I updated my app to 1.0.5-M1 but I am still hitting this issue. I am using embedded cassandra. I can try to create a repro

@carllerche

This comment has been minimized.

Copy link

commented Sep 26, 2012

This fails for me: https://github.com/carllerche/snappy-repro


java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:317)
    at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:219)
    at org.xerial.snappy.Snappy.<clinit>(Snappy.java:48)
    at com.carllerche.snappyrepro.Main.main(Main.java:9)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:297)
    at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.UnsatisfiedLinkError: no snappyjava in java.library.path
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1860)
    at java.lang.Runtime.loadLibrary0(Runtime.java:845)
    at java.lang.System.loadLibrary(System.java:1084)
    at org.xerial.snappy.SnappyNativeLoader.loadLibrary(SnappyNativeLoader.java:52)
    ... 14 more
[WARNING] 
java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:297)
    at java.lang.Thread.run(Thread.java:722)
Caused by: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] null
    at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:229)
    at org.xerial.snappy.Snappy.<clinit>(Snappy.java:48)
    at com.carllerche.snappyrepro.Main.main(Main.java:9)
    ... 6 more

Here is the contents of the jar:

META-INF/MANIFEST.MF
META-INF/
META-INF/maven/
META-INF/maven/org.xerial.snappy/
META-INF/maven/org.xerial.snappy/snappy-java/
META-INF/maven/org.xerial.snappy/snappy-java/LICENSE
META-INF/maven/org.xerial.snappy/snappy-java/pom.properties
META-INF/maven/org.xerial.snappy/snappy-java/pom.xml
org/
org/xerial/
org/xerial/snappy/
org/xerial/snappy/OSInfo.class
org/xerial/snappy/Snappy.class
org/xerial/snappy/SnappyBundleActivator.class
org/xerial/snappy/SnappyCodec.class
org/xerial/snappy/SnappyError.class
org/xerial/snappy/SnappyErrorCode.class
org/xerial/snappy/SnappyException.class
org/xerial/snappy/SnappyInputStream.class
org/xerial/snappy/SnappyLoader.class
org/xerial/snappy/SnappyNative.class
org/xerial/snappy/SnappyNativeAPI.class
org/xerial/snappy/SnappyNativeLoader.bytecode
org/xerial/snappy/SnappyOutputStream.class
org/xerial/snappy/VERSION
org/xerial/snappy/native/
org/xerial/snappy/native/Linux/
org/xerial/snappy/native/Linux/amd64/
org/xerial/snappy/native/Linux/amd64/libsnappyjava.so
org/xerial/snappy/native/Linux/arm/
org/xerial/snappy/native/Linux/arm/libsnappyjava.so
org/xerial/snappy/native/Linux/armhf/
org/xerial/snappy/native/Linux/armhf/libsnappyjava.so
org/xerial/snappy/native/Linux/i386/
org/xerial/snappy/native/Linux/i386/libsnappyjava.so
org/xerial/snappy/native/Mac/
org/xerial/snappy/native/Mac/i386/
org/xerial/snappy/native/Mac/i386/libsnappyjava.jnilib
org/xerial/snappy/native/Mac/x86_64/
org/xerial/snappy/native/Mac/x86_64/libsnappyjava.dylib
org/xerial/snappy/native/Mac/x86_64/libsnappyjava.jnilib
org/xerial/snappy/native/README
org/xerial/snappy/native/Windows/
org/xerial/snappy/native/Windows/amd64/
org/xerial/snappy/native/Windows/amd64/snappyjava.dll
org/xerial/snappy/native/Windows/x86/
org/xerial/snappy/native/Windows/x86/snappyjava.dll
@xerial

This comment has been minimized.

Copy link
Owner Author

commented Sep 27, 2012

Thanks for the report.

It looks like 1.0.5-M1 properly contains
org/xerial/snappy/native/Mac/x86_64/libsnappyjava.dylib
for open jdk for Mac but failed to load.

Does it work if you replace 1.0.5-M1 to a snapshot version 1.0.5-SNAPSHOT mentioned above?

@xerial

This comment has been minimized.

Copy link
Owner Author

commented Sep 27, 2012

I confirmed the reported problem is reproduced in 1.7.0u4.jdk both in M1 and the snapshot version.

@xerial

This comment has been minimized.

Copy link
Owner Author

commented Sep 27, 2012

It looks like SnappyLoader.class.getResource(...) cannot find .dylib file.
https://github.com/xerial/snappy-java/blob/develop/src/main/java/org/xerial/snappy/SnappyLoader.java#L440

@xerial

This comment has been minimized.

Copy link
Owner Author

commented Sep 27, 2012

The cause of this problem is that openjdk7u4 returns "universal" as os.arch value, so
it tries to find
/org/xerial/snappy/native/Mac/universal/libsnappyjava.dylib
instead of
/org/xerial/snappy/native/Mac/x86_64/libsnappyjava.dylib

xerial added a commit that referenced this issue Sep 27, 2012

@xerial

This comment has been minimized.

Copy link
Owner Author

commented Sep 27, 2012

Deployed snappy-java-1.0.5-M2 that fixes this issue. This release will be available in a few hours from Maven central.

@carllerche

This comment has been minimized.

Copy link

commented Sep 28, 2012

I updated my simple repro case to M2, however that does not fix the issue for me.

@ghost ghost assigned xerial Sep 28, 2012

@xerial

This comment has been minimized.

Copy link
Owner Author

commented Sep 28, 2012

I created a new snapshot release that at least works in my Mac 10.6 with openjdk7.
https://oss.sonatype.org/content/repositories/snapshots/org/xerial/snappy/snappy-java/1.0.5-M3-SNAPSHOT/

@xerial xerial reopened this Sep 28, 2012

xerial added a commit that referenced this issue Sep 28, 2012

@carllerche

This comment has been minimized.

Copy link

commented Sep 28, 2012

That seems to work for me. Thanks

@coltnz

This comment has been minimized.

Copy link

commented Dec 14, 2012

Hi is there a release planned that incorporates this fix anytime soon? The Cassandra project won't take a beta version -
https://issues.apache.org/jira/browse/CASSANDRA-4958

cheers
Colin

@stfp

This comment has been minimized.

Copy link

commented Jan 2, 2013

Hey - Same question here; eagerly waiting for the next release 🎉

@scottcarey

This comment has been minimized.

Copy link

commented May 1, 2013

I have to upgrade to a -M release for this to work. When will an actual release (not -rc or -M) include this? 1.0.5 seems abandoned for 1.1.0, but neither is released, only in pre-release.

@xerial

This comment has been minimized.

Copy link
Owner Author

commented May 1, 2013

@scottcarey If 1.0.5-M4 works well for your usage, I will create a new release version for 1.0.5 based on -M4 this week. I simply have been waiting bug reports.

Release of 1.1.0 needs some work to do, listed in milestone page: https://github.com/xerial/snappy-java/issues/milestones

@scottcarey

This comment has been minimized.

Copy link

commented May 1, 2013

For reference, I'm working dependency details for Apache Avro's next release: https://issues.apache.org/jira/browse/AVRO-1311 I'll see about changing to 1.0.5-M4 temporarily so others can test, then using a 1.0.5 final if that works out.

@scottcarey

This comment has been minimized.

Copy link

commented May 16, 2013

Its working for me on mac, and others on Avro in Ubuntu. I'd be interested in a release of 1.0.5. Are there changes in this version that are risky?

@xerial

This comment has been minimized.

Copy link
Owner Author

commented May 17, 2013

Hi,

Thanks for testing the 1.0.5-M4 release.

I just released stable 1.0.5 version, which is available in Maven central
and is exactly the same with 1.0.5-M4 except the version number.

In version 1.1.0, we will introduce several changes in native library
loading mechanism and framed-format support, which is
added in the original snappy 1.1.0.

Regards,

@vbekiaris

This comment has been minimized.

Copy link

commented Jul 23, 2013

Hi,

I have just checked the binary for released version 1.0.5 from google code as well as the tar.gz archive on github and it seems that it still does not include the Mac library with the .dylib extension. Could you please confirm the fix is in 1.0.5?

Cheers,
Vassilis

@xerial

This comment has been minimized.

Copy link
Owner Author

commented Jul 24, 2013

Please use 1.0.5-M4.
https://github.com/xerial/snappy-java/tree/1.0.5-M4/src/main/resources/org/xerial/snappy/native/Mac/i386

This version has a fix that even if .dylib file is not found, trying to load .jnilib instead.

@mdaxini

This comment has been minimized.

Copy link

commented Sep 10, 2013

It boggles me why the fix is in M4 but removed from the final release.

Here is a simple workaround in Mac with JDK 7 for anyone looking for answers in this thread. Add a system property:

-Dorg.xerial.snappy.lib.name=libsnappyjava.jnilib

@xerial

This comment has been minimized.

Copy link
Owner Author

commented Sep 11, 2013

@mdaxini
The latest version is 1.1.0-M4 and it is quite close to the release version, since it only lacks FreeBSD native libs.
Please use this version.

Thanks.

@mdaxini

This comment has been minimized.

Copy link

commented Sep 11, 2013

@xerial

Cannot use 1.1.0-M4 due to native libraries not being compatible with the c libraries on the available OS. No control over changing the OS at this time.

Thanks for the quick response.

@xerial

This comment has been minimized.

Copy link
Owner Author

commented Sep 11, 2013

@mdaxini
Could you tell me your OS version (Mac OS X?)

I tested 1.1.0-M4 in Mac OS X 10.7 (Intell 64bit) and it works well.

@dgerges

This comment has been minimized.

Copy link

commented Feb 5, 2014

An easy fix if you want to continue to use 1.4.0.1 on MacOSX with JDK 1.7

unzip snappy-java-1.0.4.1.jar
cd org/xerial/snappy/native/Mac/x86_64/
copy libsnappyjava.jnilib libsnappyjava.dylib
cd ../../../../../..
cp snappy-java-1.0.4.1.jar snappy-java-1.0.4.1.jar.old
jar cf snappy-java-1.0.4.1.jar org

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
9 participants
You can’t perform that action at this time.