JRuby can Not load JFFI library under IBM Power 64 with PLinux platform

[xialixl]

Environments:
IBM Power ppc 64 bit , POWER7 architecture,
Red Hat Enterprise Linux Server release 6.3 (Santiago)
JRuby ver 1.6.8， jar file: jruby-complete-1.6.8.jar
Apache Hbase 0.95.2

In case of running Command from Hbase :
.../hbase-0.95.2/bin/hbase shell

We got Errors like following:

INFO  [main] Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
java.lang.RuntimeException: java.lang.UnsatisfiedLinkError: Could not locate stub library in jar file.  Tried [jni/ppc64-Linux/libjffi-1.0.so, /jni/ppc64-Linux/libjffi-1.0.so]
        at com.kenai.jffi.Foreign$InValidInstanceHolder.getForeign(Foreign.java:90)

My Analysis:
It looks like JRuby was going to load Java JFFI lib: libjffi-1.0.so which is supposed to exist under location jni/ppc64-Linux/libjffi-1.0.so inside the Jar 'ruby-complete-1.6.8.jar'.
for 1.6.8 version currently Jruby only has following kinds of JFFI lib for choices :

jni/Darwin/
jni/Darwin/libjffi-1.0.jnilib
jni/i386-FreeBSD/
jni/i386-FreeBSD/libjffi-1.0.so
jni/i386-Linux/
jni/i386-Linux/libjffi-1.0.so
jni/i386-OpenBSD/
jni/i386-OpenBSD/libjffi-1.0.so
jni/i386-SunOS/
jni/i386-SunOS/libjffi-1.0.so
jni/i386-Windows/
jni/i386-Windows/jffi-1.0.dll
jni/ppc-AIX/
jni/ppc-AIX/libjffi-1.0.a
jni/ppc-Linux/
jni/ppc-Linux/libjffi-1.0.so
jni/sparc-SunOS/
jni/sparc-SunOS/libjffi-1.0.so
jni/sparcv9-SunOS/
jni/sparcv9-SunOS/libjffi-1.0.so
jni/x86_64-FreeBSD/
jni/x86_64-FreeBSD/libjffi-1.0.so
jni/x86_64-Linux/
jni/x86_64-Linux/libjffi-1.0.so
jni/x86_64-OpenBSD/
jni/x86_64-OpenBSD/libjffi-1.0.so
jni/x86_64-SunOS/
jni/x86_64-SunOS/libjffi-1.0.so
jni/x86_64-Windows/
jni/x86_64-Windows/jffi-1.0.dll

Fix:
Since '/jni/ppc64-Linux/libjffi-1.0.so' is missed inside jar , I downloaded the right version of JFFI sources (https://github.com/jnr/jffi) and built them by IBM XL C Compiler instead of default GCC at IBM ppc64 platform , Added the output libjffi-1.0.so into the right place of the jruby-complete-1.6.8.jar. Now it works well at my ppc64 env.

Advice:
To support IBM ppc64 with Linux env , Could Jruby add JFFI library for ppc64 version into released Jruby jar files.

Thanks

[BanzaiMan]

We generally don't have access to these hardware, so we ask users to provide us with binary files such as these.

Thanks.

[xialixl]

Ok, I got the lib file and jar file that could provide you with jruby, but how I could share with you and the project , Fork the project ?
sorry I am quite new for using github .

[faisalmoeen]

I am facing the same issue. Xialixl, can you please send the files to fmorakzai@gmail.com ?

[mkristian]

@xialixl sorry for the late reply: fork https://github.com/jnr/jffi and send a pull request

[ayappanec]

A pull request has been sent to jffi repo with both ppc64-Linux and ppc64le-Linux binaries. Now the question here is whether the binaries 'ppc64 (ppc64le) libjffi-1.0.so' files will be added to the already published jruby-complete-1.6.8.jar file ?

[mkristian]

we can not republish released jar (at least on maven central this is not
possible), but if you have a local maven repo you could add the missing
libjffi.so in the jruby-complete file

[ayappanec]

Ok. We have this problem of locally building jruby-complete-1.6.8.jar file in ppc64 (Big Endian) systems
A issue has been opened ( https://jira.codehaus.org/browse/JRUBY-7201 )

[ayappanec]

Can any maintainers look into this issue ( jnr/jffi#12 )

[headius]

Wow, apologies for missing this one for so long. The PR has been merged and we'll try to get releases of the whole JNR chain out for JRuby 1.7.x and 9k.pre2.

[headius]

Oops...don't close until we have updated JNR-* in a JRuby release.

[ayappanec]

Can this be added to upcoming JRuby 1.7.x release also ?

[ayappanec]

Any update for the above comment?

[gustavotemple]

@ayappanec, did you see this issue?

jnr/jffi#15

[headius]

I don't see why we couldn't update this in 1.7.x as well. @enebo and I know we need to update everything for next releases, which will involve fixing some regressions in the jnr-* chain of projects. Marking for 1.7.20.

[headius]

jnr/jffi#16 details the now-mitigated showstopper preventing us from releaseing jffi 1.2.8. Should be able to get the family released this week.

[ayappanec]

I came to know about the problem you guys facing for releasing jffi 1.2.8. I hope this one will get into Jruby 1.7.20 & 9000. Thank you.

[enebo]

@headius was able to release jffi and dependent artifacts yesterday so the plan is to update master and jruby-1_7 before our next releases.

[headius]

1.7 and master have both been updated for latest jnr-* projects.

If it's possible for someone on PLinux/ppc64 to do a build of jruby-1_7 branch and confirm it works properly, we'd appreciate it :-)

[ayappanec]

Building jruby-1_7 branch fails in PLinux/ppc64 & PLinux/ppc64le due to unknown compilation issue.
"Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project jruby-core: Compilation failure"

But the main problem here is we are not able to build jruby-complete jar individually due to missing artfifacts.
"Failed to execute goal on project jruby-complete: Could not resolve dependencies for project org.jruby:jruby-complete:bundle:1.7.20-SNAPSHOT: The following artifacts could not be resolved: org.jruby:jruby-core:jar:noasm:1.7.20-SNAPSHOT, org.jruby:jruby-stdlib:jar:1.7.20-SNAPSHOT: Could not find artifact org.jruby:jruby-core:jar:noasm:1.7.20-SNAPSHOT in sonatype (https://oss.sonatype.org/content/repositories/snapshots/)"

It seems like the artifacts are not yet available in sonatype repository.

[mkristian]

to build jruby-complete you need to run from the basedir of jruby
mvn -Pcomplete
or mvn clean install -Pcomplete
if this is not passing beyond jruby-core then nothing furtehr down the road
will work since all those missing artifacts will be build on the way

On Wed, Mar 4, 2015 at 8:25 AM, ayappanec notifications@github.com wrote:

Building jruby-1_7 branch fails in PLinux/ppc64 & PLinux/ppc64le due to
unknown compilation issue.
"Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile
(default-compile) on project jruby-core: Compilation failure"

But the main problem here is we are not able to build jruby-complete jar
individually due to missing artfifacts.
"Failed to execute goal on project jruby-complete: Could not resolve
dependencies for project org.jruby:jruby-complete:bundle:1.7.20-SNAPSHOT:
The following artifacts could not be resolved:
org.jruby:jruby-core:jar:noasm:1.7.20-SNAPSHOT,
org.jruby:jruby-stdlib:jar:1.7.20-SNAPSHOT: Could not find artifact
org.jruby:jruby-core:jar:noasm:1.7.20-SNAPSHOT in sonatype (
https://oss.sonatype.org/content/repositories/snapshots/)"

It seems like the artifacts are not yet available in sonatype repository.

—
Reply to this email directly or view it on GitHub
#1389 (comment).

[ayappanec]

I tried both the options but they failed to get beyond jruby-core. Finally i went into the maven/jruby-complete directory and tried to build jruby-complete jar by issuing "mvn clean install" and there i faced this missing artifacts issue..

[mkristian]

I built the snapshot for you:
https://oss.sonatype.org/content/repositories/snapshots/org/jruby/jruby-complete/1.7.20-SNAPSHOT/

[headius]

@ayappanec Please open a separate issue for your problems building JRuby on PLinux and we'll work through them separately. I suspect it's some oddity in how our build interacts with the JVM on that platform.

[ayappanec]

@mkristian It requires jruby-core,jruby-stdlib and other such jars also.Can you please place those jars also in sonatype?
@headius Sure. I will open a separate issue for building JRuby on PLinux

[mkristian]

https://oss.sonatype.org/content/repositories/snapshots/org/jruby/jruby-core/ https://oss.sonatype.org/content/repositories/snapshots/org/jruby/jruby-stdlib/ BUT I do not understand since jruby-complete contains both the other artifacts. and you do not need to build anything anymore, just download what you need. if you need them as maven artifacts somehow you should be able to build everything after the core with mvn clean install -Pcomplete -rf :jruby-lib let me know if you need something else.

[ayappanec]

I tried with the command mentioned above. its trying to download the jruby-core jar from sonatype but failed since the jar uploaded there has some number associated with the filename "jruby-stdlib-1.7.20-20150304.132127-1.jar"

[headius]

@ayappanec Thank you...please let us know when you've opened the other issue. For confirming this issue right now, just download the 1.7.20 build that @mkristian provided above.

[ayappanec]

@headius I could see the jffi binaries for ppc64 & ppc64le in the jruby-complete jar. Thank you

[ayappanec]

@headius I opened a new issue for Jruby build failure (#2648)

[ayappanec]

It seems like we need ppc64le jnr-ffi TypeAliases and ruby ffi ppc64le platform specific types.conf as well to get the latest jruby working ppc64le. I could see the changes already went into the master branch of jnr-ffi & jruby. I don't know when the jnr-ffi next release is going to happen. If it happens before jruby release, then we need to change the jnr-ffi version to 2.0.3 in jruby to make use of it. Otherwise we have wait till further releases of jruby.

[headius]

@ayappanec I'm largely paralyzed from improving ppc64le support right now without your help, so if there are additional issues we need to fix, I'm hoping to see PRs from you :-)

[ayappanec]

@headius Sure.. Right now #2790 & #1897 seems to be the major hurdle. #2647 is yet another one but we have openjdk as an option here.