Native library loading fails on openjdk7u4 for mac #6

xerial · 2012-09-06T02:34:45Z

What steps will reproduce the problem?

Install openjdk7u4 preview for mac
Invoke Snappy#compress

What is the expected output? What do you see instead?
I expect Snappy to work. But it fails to find the native library.

What version of the product are you using? On what operating system?
Snappy 1.0.4.1, OS X 10.7, OpenJDK7u4

Please provide any additional information below.
The openjdk guys have changed System.mapLibraryName to return an extension of "dylib" instead of "jnilib". It looks like this will be the standard for openjdk7 on mac. They have put some backwards compatibility hooks in System.loadLibrary, but this does not help the code in SnappyLoader.

The related code is in SnappyLoader:

    // Resolve the library file name with a suffix (e.g., dll, .so, etc.) 
    if (snappyNativeLibraryName == null)
        snappyNativeLibraryName = System.mapLibraryName(&quot;snappyjava&quot;);

    if (snappyNativeLibraryPath != null) {
        File nativeLib = new File(snappyNativeLibraryPath, snappyNativeLibraryName);
        if (nativeLib.exists())
            return nativeLib;
    }

    {
        // Load an OS-dependent native library inside a jar file
        snappyNativeLibraryPath = &quot;/org/xerial/snappy/native/&quot; + OSInfo.getNativeLibFolderPathForCurrentOS();

        if (SnappyLoader.class.getResource(snappyNativeLibraryPath + &quot;/&quot; + snappyNativeLibraryName) != null) {

On OpenJDK7 on Mac, it attempts to look for libsnappyjava.dylib. In the jar, it is packaged as libsnappyjava.jnilib, so it fails.

It seems the only reasonable fix is to check for both the dylib and jnilib extensions when on a Mac.

Migrated from http://code.google.com/p/snappy-java/issues/detail?id=39

earlier comments

james@tyrrells.com said, at 2012-06-26T04:06:14.000Z:

Duplicating "libsnappyjava.jnilib" with a .dylib extension fixed the issue for me too.

taroleo said, at 2012-06-30T12:04:12.000Z:

Issue 42 has been merged into this issue.

taroleo said, at 2012-06-30T12:04:26.000Z:

Issue 42 has been merged into this issue.

The text was updated successfully, but these errors were encountered:

michaelklishin · 2012-09-07T02:09:51Z

Any news on this issue? Among other things, it makes it impossible to run Cassandra on JDK 7 on OS X without workarounds. If @xerial has a good strategy for dealing with the root cause in mind, I would be interested in submitting a pull request with a fix.

xerial · 2012-09-07T02:21:38Z

Having both .jnilib and .dylib in src/main/resources/org/xerial/snappy/native/Mac/x86_64 would be reasonable, since
the default JDK in Mac OS X Lion still uses .jnilib extension.

And building a native library for Mac with jdk7 would be possible by supplying an option:
$ make native LIBNAME=libsnappyjava.dylib

Fow now I simply duplicate the native library.

michaelklishin · 2012-09-07T02:26:06Z

@xerial fantastic. Would it be difficult to publish a new point release with this in? Thank you very much!

xerial · 2012-09-07T02:31:27Z

I deployed snapshot release with this fix.

https://oss.sonatype.org/content/repositories/snapshots/org/xerial/snappy/snappy-java/1.0.5-SNAPSHOT/

Try snappy-java-1.0.5-20120907.023008-2.jar

michaelklishin · 2012-09-07T02:35:07Z

I can confirm that Cassandra 1.1 starts fine when the Snappy jar is replaced with this snapshot.

xerial · 2012-09-07T02:38:39Z

Thanks for the report.

Now I close this issue.

michaelklishin · 2012-09-07T02:43:27Z

When you issue a new release, please leave a comment here. I will file an issue for Cassandra to update the dependency.

xerial · 2012-09-11T05:16:38Z

A milestone release is here, which will be synchronized to Maven central repo in a few hours:
https://oss.sonatype.org/content/repositories/releases/org/xerial/snappy/snappy-java/1.0.5-M1/

carllerche · 2012-09-26T17:36:51Z

I updated my app to 1.0.5-M1 but I am still hitting this issue. I am using embedded cassandra. I can try to create a repro

carllerche · 2012-09-26T17:52:45Z

This fails for me: https://github.com/carllerche/snappy-repro


java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:317)
    at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:219)
    at org.xerial.snappy.Snappy.<clinit>(Snappy.java:48)
    at com.carllerche.snappyrepro.Main.main(Main.java:9)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:297)
    at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.UnsatisfiedLinkError: no snappyjava in java.library.path
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1860)
    at java.lang.Runtime.loadLibrary0(Runtime.java:845)
    at java.lang.System.loadLibrary(System.java:1084)
    at org.xerial.snappy.SnappyNativeLoader.loadLibrary(SnappyNativeLoader.java:52)
    ... 14 more
[WARNING] 
java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:297)
    at java.lang.Thread.run(Thread.java:722)
Caused by: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] null
    at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:229)
    at org.xerial.snappy.Snappy.<clinit>(Snappy.java:48)
    at com.carllerche.snappyrepro.Main.main(Main.java:9)
    ... 6 more

Here is the contents of the jar:

META-INF/MANIFEST.MF
META-INF/
META-INF/maven/
META-INF/maven/org.xerial.snappy/
META-INF/maven/org.xerial.snappy/snappy-java/
META-INF/maven/org.xerial.snappy/snappy-java/LICENSE
META-INF/maven/org.xerial.snappy/snappy-java/pom.properties
META-INF/maven/org.xerial.snappy/snappy-java/pom.xml
org/
org/xerial/
org/xerial/snappy/
org/xerial/snappy/OSInfo.class
org/xerial/snappy/Snappy.class
org/xerial/snappy/SnappyBundleActivator.class
org/xerial/snappy/SnappyCodec.class
org/xerial/snappy/SnappyError.class
org/xerial/snappy/SnappyErrorCode.class
org/xerial/snappy/SnappyException.class
org/xerial/snappy/SnappyInputStream.class
org/xerial/snappy/SnappyLoader.class
org/xerial/snappy/SnappyNative.class
org/xerial/snappy/SnappyNativeAPI.class
org/xerial/snappy/SnappyNativeLoader.bytecode
org/xerial/snappy/SnappyOutputStream.class
org/xerial/snappy/VERSION
org/xerial/snappy/native/
org/xerial/snappy/native/Linux/
org/xerial/snappy/native/Linux/amd64/
org/xerial/snappy/native/Linux/amd64/libsnappyjava.so
org/xerial/snappy/native/Linux/arm/
org/xerial/snappy/native/Linux/arm/libsnappyjava.so
org/xerial/snappy/native/Linux/armhf/
org/xerial/snappy/native/Linux/armhf/libsnappyjava.so
org/xerial/snappy/native/Linux/i386/
org/xerial/snappy/native/Linux/i386/libsnappyjava.so
org/xerial/snappy/native/Mac/
org/xerial/snappy/native/Mac/i386/
org/xerial/snappy/native/Mac/i386/libsnappyjava.jnilib
org/xerial/snappy/native/Mac/x86_64/
org/xerial/snappy/native/Mac/x86_64/libsnappyjava.dylib
org/xerial/snappy/native/Mac/x86_64/libsnappyjava.jnilib
org/xerial/snappy/native/README
org/xerial/snappy/native/Windows/
org/xerial/snappy/native/Windows/amd64/
org/xerial/snappy/native/Windows/amd64/snappyjava.dll
org/xerial/snappy/native/Windows/x86/
org/xerial/snappy/native/Windows/x86/snappyjava.dll

xerial · 2012-09-27T01:03:33Z

Thanks for the report.

It looks like 1.0.5-M1 properly contains
org/xerial/snappy/native/Mac/x86_64/libsnappyjava.dylib
for open jdk for Mac but failed to load.

Does it work if you replace 1.0.5-M1 to a snapshot version 1.0.5-SNAPSHOT mentioned above?

xerial · 2012-09-27T01:26:41Z

I confirmed the reported problem is reproduced in 1.7.0u4.jdk both in M1 and the snapshot version.

xerial · 2012-09-27T01:57:06Z

It looks like SnappyLoader.class.getResource(...) cannot find .dylib file.
https://github.com/xerial/snappy-java/blob/develop/src/main/java/org/xerial/snappy/SnappyLoader.java#L440

xerial · 2012-09-27T02:04:10Z

The cause of this problem is that openjdk7u4 returns "universal" as os.arch value, so
it tries to find
/org/xerial/snappy/native/Mac/universal/libsnappyjava.dylib
instead of
/org/xerial/snappy/native/Mac/x86_64/libsnappyjava.dylib

xerial · 2012-09-27T03:23:55Z

Deployed snappy-java-1.0.5-M2 that fixes this issue. This release will be available in a few hours from Maven central.

carllerche · 2012-09-28T00:18:32Z

I updated my simple repro case to M2, however that does not fix the issue for me.

xerial · 2012-09-28T02:37:49Z

I created a new snapshot release that at least works in my Mac 10.6 with openjdk7.
https://oss.sonatype.org/content/repositories/snapshots/org/xerial/snappy/snappy-java/1.0.5-M3-SNAPSHOT/

carllerche · 2012-09-28T02:52:38Z

That seems to work for me. Thanks

coltnz · 2012-12-14T11:07:57Z

Hi is there a release planned that incorporates this fix anytime soon? The Cassandra project won't take a beta version -
https://issues.apache.org/jira/browse/CASSANDRA-4958

cheers
Colin

stfp · 2013-01-02T20:22:22Z

Hey - Same question here; eagerly waiting for the next release 🎉

scottcarey · 2013-05-01T00:26:38Z

I have to upgrade to a -M release for this to work. When will an actual release (not -rc or -M) include this? 1.0.5 seems abandoned for 1.1.0, but neither is released, only in pre-release.

xerial · 2013-05-01T01:18:30Z

@scottcarey If 1.0.5-M4 works well for your usage, I will create a new release version for 1.0.5 based on -M4 this week. I simply have been waiting bug reports.

Release of 1.1.0 needs some work to do, listed in milestone page: https://github.com/xerial/snappy-java/issues/milestones

scottcarey · 2013-05-01T17:00:01Z

For reference, I'm working dependency details for Apache Avro's next release: https://issues.apache.org/jira/browse/AVRO-1311 I'll see about changing to 1.0.5-M4 temporarily so others can test, then using a 1.0.5 final if that works out.

scottcarey · 2013-05-16T21:39:35Z

Its working for me on mac, and others on Avro in Ubuntu. I'd be interested in a release of 1.0.5. Are there changes in this version that are risky?

xerial · 2013-05-17T01:50:31Z

Hi,

Thanks for testing the 1.0.5-M4 release.

I just released stable 1.0.5 version, which is available in Maven central
and is exactly the same with 1.0.5-M4 except the version number.

In version 1.1.0, we will introduce several changes in native library
loading mechanism and framed-format support, which is
added in the original snappy 1.1.0.

Regards,

vbekiaris · 2013-07-23T18:45:11Z

Hi,

I have just checked the binary for released version 1.0.5 from google code as well as the tar.gz archive on github and it seems that it still does not include the Mac library with the .dylib extension. Could you please confirm the fix is in 1.0.5?

Cheers,
Vassilis

xerial · 2013-07-24T08:11:03Z

Please use 1.0.5-M4.
https://github.com/xerial/snappy-java/tree/1.0.5-M4/src/main/resources/org/xerial/snappy/native/Mac/i386

This version has a fix that even if .dylib file is not found, trying to load .jnilib instead.

mdaxini · 2013-09-10T20:14:40Z

It boggles me why the fix is in M4 but removed from the final release.

Here is a simple workaround in Mac with JDK 7 for anyone looking for answers in this thread. Add a system property:

-Dorg.xerial.snappy.lib.name=libsnappyjava.jnilib

xerial · 2013-09-11T06:34:41Z

@mdaxini
The latest version is 1.1.0-M4 and it is quite close to the release version, since it only lacks FreeBSD native libs.
Please use this version.

Thanks.

mdaxini · 2013-09-11T07:33:57Z

@xerial

Cannot use 1.1.0-M4 due to native libraries not being compatible with the c libraries on the available OS. No control over changing the OS at this time.

Thanks for the quick response.

xerial · 2013-09-11T08:00:22Z

@mdaxini
Could you tell me your OS version (Mac OS X?)

I tested 1.1.0-M4 in Mac OS X 10.7 (Intell 64bit) and it works well.

dgerges · 2014-02-05T13:31:09Z

An easy fix if you want to continue to use 1.4.0.1 on MacOSX with JDK 1.7

unzip snappy-java-1.0.4.1.jar
cd org/xerial/snappy/native/Mac/x86_64/
copy libsnappyjava.jnilib libsnappyjava.dylib
cd ../../../../../..
cp snappy-java-1.0.4.1.jar snappy-java-1.0.4.1.jar.old
jar cf snappy-java-1.0.4.1.jar org

See: jblas-project/jblas#19 and xerial/snappy-java#6

xerial added a commit that referenced this issue Sep 7, 2012

Fix on issue #6

a1c36bb

xerial closed this as completed Sep 7, 2012

ifesdjeen mentioned this issue Sep 23, 2012

Fixing build / tests on Mac Os X 64 bit with JDK 7 that was failing due to Snappy compressor NL. mpenet/casyn#2

Merged

xerial added a commit that referenced this issue Sep 27, 2012

Fixes issue #6

5a60c13

ghost assigned xerial Sep 28, 2012

xerial reopened this Sep 28, 2012

xerial added a commit that referenced this issue Sep 28, 2012

Fixes for issue #6

ed37dfa

xerial closed this as completed Sep 28, 2012

mlehman mentioned this issue Dec 7, 2012

System.mapLibraryName returns suffix 'dylib' and not 'jnilib' using JDK 7 on OS X jblas-project/jblas#19

Closed

dehora mentioned this issue Jan 11, 2013

Upgraded to snappy-1.0.5-M3. Allows build from mac Netflix/Priam#130

Merged

michaelklishin mentioned this issue Feb 25, 2013

System.err.println causes NullPointerException when using ElasticSearch as a java library elastic/elasticsearch#2321

Closed

virtuoushub pushed a commit to virtuoushub/lwjgl3 that referenced this issue Nov 27, 2014

Added workaround for Apple's JDK 6 System#mapLibraryName(String).

4f5a9f7

See: jblas-project/jblas#19 and xerial/snappy-java#6

virtuoushub mentioned this issue Nov 28, 2014

OS X java 6 support LWJGL/lwjgl#100

Closed

remi-parain mentioned this issue Mar 4, 2015

SnappyJava error (Mac OSX ) wisdom-framework/wisdom-orientdb#22

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Native library loading fails on openjdk7u4 for mac #6

Native library loading fails on openjdk7u4 for mac #6

xerial commented Sep 6, 2012

michaelklishin commented Sep 7, 2012

xerial commented Sep 7, 2012

michaelklishin commented Sep 7, 2012

xerial commented Sep 7, 2012

michaelklishin commented Sep 7, 2012

xerial commented Sep 7, 2012

michaelklishin commented Sep 7, 2012

xerial commented Sep 11, 2012

carllerche commented Sep 26, 2012

carllerche commented Sep 26, 2012

xerial commented Sep 27, 2012

xerial commented Sep 27, 2012

xerial commented Sep 27, 2012

xerial commented Sep 27, 2012

xerial commented Sep 27, 2012

carllerche commented Sep 28, 2012

xerial commented Sep 28, 2012

carllerche commented Sep 28, 2012

coltnz commented Dec 14, 2012

stfp commented Jan 2, 2013

scottcarey commented May 1, 2013

xerial commented May 1, 2013

scottcarey commented May 1, 2013

scottcarey commented May 16, 2013

xerial commented May 17, 2013

vbekiaris commented Jul 23, 2013

xerial commented Jul 24, 2013

mdaxini commented Sep 10, 2013

xerial commented Sep 11, 2013

mdaxini commented Sep 11, 2013

xerial commented Sep 11, 2013

dgerges commented Feb 5, 2014

Native library loading fails on openjdk7u4 for mac #6

Native library loading fails on openjdk7u4 for mac #6

Comments

xerial commented Sep 6, 2012

earlier comments

michaelklishin commented Sep 7, 2012

xerial commented Sep 7, 2012

michaelklishin commented Sep 7, 2012

xerial commented Sep 7, 2012

michaelklishin commented Sep 7, 2012

xerial commented Sep 7, 2012

michaelklishin commented Sep 7, 2012

xerial commented Sep 11, 2012

carllerche commented Sep 26, 2012

carllerche commented Sep 26, 2012

xerial commented Sep 27, 2012

xerial commented Sep 27, 2012

xerial commented Sep 27, 2012

xerial commented Sep 27, 2012

xerial commented Sep 27, 2012

carllerche commented Sep 28, 2012

xerial commented Sep 28, 2012

carllerche commented Sep 28, 2012

coltnz commented Dec 14, 2012

stfp commented Jan 2, 2013

scottcarey commented May 1, 2013

xerial commented May 1, 2013

scottcarey commented May 1, 2013

scottcarey commented May 16, 2013

xerial commented May 17, 2013

vbekiaris commented Jul 23, 2013

xerial commented Jul 24, 2013

mdaxini commented Sep 10, 2013

xerial commented Sep 11, 2013

mdaxini commented Sep 11, 2013

xerial commented Sep 11, 2013

dgerges commented Feb 5, 2014