JNI: unable to compile self-contained library, or a wrapper that works in tandem with the C library

[themarvelousbreadfish]

I'm trying to build the Java port of cld on Ubuntu 12.04 64-bit, using OpenJDK 6. The shared object file built by Maven contains the JNI bindings, but the CompactLangDet::DetectLanguage symbol is undefined. (Bit of a C++ newbie) I can't tell if the intent is for the Maven-built wrapper to be used alongside the make-built libcld, or whether Maven is supposed to be generating an "all-in-one" library.

I am encountering a symbol lookup error when running CompactLanguageDetector's main method from the command line, with the Maven-generated shared object being loaded as libcld, like so:

ln -s libcld/0.0.1-SNAPSHOT/libcld-0.0.1-SNAPSHOT.so libcld/0.0.1-SNAPSHOT/libcld.so 
java -Djava.library.path=./libcld/0.0.1-SNAPSHOT/ -cp cld/1.0-SNAPSHOT/cld-1.0-SNAPSHOT.jar com.mzsanford.cld.CompactLanguageDetector "octocat"
java: symbol lookup error: ~/.m2/repository/com/mzsanford/cld/libcld/0.0.1-SNAPSHOT/libcld-0.0.1-SNAPSHOT.so: undefined symbol: _ZN14CompactLangDet14DetectLanguageEPKNS_15DetectionTablesEPKcibbbbS4_i8LanguagePS5_PiPdS7_Pb

I'm setting the JAVA_HOME and LD_LIBRARY_PATH variables via an export statement:

export LD_LIBRARY_PATH="/usr/local/lib/cld:/usr/lib" && export JAVA_JOME="/usr/lib/jvm/java-6-openjdk-amd64" && mvn install

Examining the shared object produced by Maven (libcld-0.0.1-SNAPSHOT.so) using nm -gC shows that the CompactLangDet:DetectLanguage symbol is undefined.

U CompactLangDet::DetectLanguage(CompactLangDet::DetectionTables const*, char const*, int, bool, bool, bool, bool, char const*, int, Language, Language*, int*, double*, int*, bool*)

I've also tried writing code that loads the Maven-generated wrapper then libcldfrom /usr/local/lib/cld/ and vice-versa, but it hasn't helped.

Could someone please tell me:
a) is the Maven-generated artifact supposed to be self-contained? (i.e. provides libcld and the JNI wrappers in the same file)
and
b) am I building incorrectly?

Thanks!

Here's the mvn install output for native:compile and native:link for reference:

[INFO] ------------------------------------------------------------------------
[INFO] Building Compact Language Detector - native unix
[INFO]    task-segment: [install]
[INFO] ------------------------------------------------------------------------
[INFO] [native:initialize {execution: default-initialize}]
[INFO] [native:unzipinc {execution: default-unzipinc}]
[INFO] [native:javah {execution: default-javah}]
[INFO] [native:compile {execution: default-compile}]
[INFO] /bin/sh -c cd ~/workspace/tools/investigations/compact-language-detector/ports/java/native/unix && gcc -c -fPIC -DCLD_WINDOWS -Inull/include/linux -I/usr/local/include/cld -I~/workspace/tools/investigations/compact-language-detector/ports/java/native/src/main/native -I/usr/lib/jvm/java-6-openjdk-amd64/jre/../include -I/usr/lib/jvm/java-6-openjdk-amd64/jre/../include/unix -o ~/workspace/tools/investigations/compact-language-detector/ports/java/native/unix/target/objs/com_mzsanford_cld_CompactLanguageDetector.o -c ~/workspace/tools/investigations/compact-language-detector/ports/java/native/src/main/native/com_mzsanford_cld_CompactLanguageDetector.cpp
[INFO] [native:link {execution: default-link}]
[INFO] /bin/sh -c cd ~/workspace/tools/investigations/compact-language-detector/ports/java/native/unix && gcc -shared -lcld -L/usr/local/lib/cld -o ~/workspace/tools/investigations/compact-language-detector/ports/java/native/unix/target/libcld.so target/objs/com_mzsanford_cld_CompactLanguageDetector.o

[mzsanford]

Hi @themarvelousbreadfish,

The Maven artifact should still link to the underlying C++ library, it is not self-contained. Can you do me a favor and run ldd libcld/0.0.1-SNAPSHOT/libcld.so? That should show what it linked to. The fix may be as simple as adding /usr/local/lib/cld to your java.library.path like so:

java -Djava.library.path=./libcld/0.0.1-SNAPSHOT:/usr/local/lib/cld -cp cld/1.0-SNAPSHOT/cld-1.0-SNAPSHOT.jar com.mzsanford.cld.CompactLanguageDetector "octocat"

[themarvelousbreadfish]

ldd libcld/0.0.1-SNAPSHOT/libcld-0.0.1-SNAPSHOT.so 
    linux-vdso.so.1 =>  (0x00007fff79872000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f43b7f10000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f43b8501000)

And:

java -Djava.library.path=./libcld/0.0.1-SNAPSHOT:/usr/local/lib/cld -cp cld/1.0-SNAPSHOT/cld-1.0-SNAPSHOT.jar com.mzsanford.cld.CompactLanguageDetector "octocat"
Exception in thread "main" java.lang.UnsatisfiedLinkError:  com.mzsanford.cld.CompactLanguageDetector.detectLanguageDetails(Ljava/lang/String;ZZZZLjava/lang/String;)Lcom/mzsanford/cld/LanguageDetectionResult;
    at com.mzsanford.cld.CompactLanguageDetector.detectLanguageDetails(Native Method)
    at com.mzsanford.cld.CompactLanguageDetector.detect(CompactLanguageDetector.java:23)
    at com.mzsanford.cld.CompactLanguageDetector.main(CompactLanguageDetector.java:41)

My thinking is that System.loadLibrary("libcld") is only loading one of the libcld files. I've tried modifying the code to load the C library and the wrapper individually using two different System.loadLibrary() calls but it had no effect.

[mzsanford]

It does not look like the JNI library (libcld-0.0.0.1-SNAPSHOT.so) is linked to the main libcld.so like I would expect. My Ubuntu setup shows the following:

matt@ubuntu:~/src/cld/ports/java$ ldd native/unix/target/libcld.so 
    linux-gate.so.1 =>  (0x0044b000)
    libcld.so.0 => /usr/local/lib/cld/libcld.so.0 (0x00df6000)
    libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0x00501000)
    libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00b7c000)
    libm.so.6 => /lib/tls/i686/cmov/libm.so.6 (0x006ae000)
    libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x0088f000)
    /lib/ld-linux.so.2 (0x003a8000)

My guess is that my Maven config has goofed up the compilation path in some way or is otherwise not playing nice with OpenJDK. I'll keep digging.

[themarvelousbreadfish]

Would you please detail your testing setup so that I can duplicate it on a VM? My best guess based on your blog posts was a 32-bit Ubuntu 11.10 VM.

[mzsanford]

You are correct. My Ubuntu setup is a VMWare image started fresh for this with no history.

matt@ubuntu:~/src/cld/ports/java$ uname -a
Linux ubuntu 2.6.32-38-generic #83-Ubuntu SMP Wed Jan 4 11:13:04 UTC 2012 i686 GNU/Linux

In my case I installed GCC and the Oracle JDK. I then installed the base library:

$ git clone http://github.com/mzsanford/cld.git
$ cd cld
$ ./configure
$ make install

Note that there have been some autoconf issues. Check the closed issues for resolutions

I then set JAVA_HOME to /usr/lib/jvm/java-7-oracle and was able to compile:

$ git clone http://github.com/mzsanford/cld.git
$ cd cld/ports/java
$ mvn compile
$ mvn test
$ mvn install

[themarvelousbreadfish]

2.6.32-38

Okay, so you're using 11.04. (For anyone reading's benefit.)

[themarvelousbreadfish]

I was able to compile a working version of the library using the following steps. I will follow up later as to whether I can reproduce on Ubuntu 12.04 64-bit. There is one modification necessary for running on amd64 - adding the -fPIC flag to compilerStartOptions in ports/java/native/unix/pom.xml. I haven't tested this enough yet to open an issue.

1. Start with a fresh 32-bit Ubuntu 11.04 image.
2. Install autoconf, libtool, make and g++.
3. Fix the ./configure script using autoreconf -vi as described here.
4. Build and install the C library as described above.
5. Install the Oracle Java 7 JDK (again, different versions and sources - e.g. OpenJDK - may work but this is just what I've tested). One way to do this:
   • sudo apt-get install python-software-properties to provide the add-apt-repository command.
   • sudo add-apt-repository ppa:webupd8team/java
   • sudo apt-get update
   • sudo apt-get install oracle-java7-installer
   • Source: http://www.webupd8.org/2012/01/install-oracle-java-jdk-7-in-ubuntu-via.html
6. Install Apache Maven 2. sudo apt-get install maven2 --no-install-recommends
7. In ~/.bashrc, using the export command, set JAVA_HOME to point to /usr/lib/jvm/java-7-oracle/. Set LD_LIBRARY_PATH to include /usr/local/lib/cld/
8. Run the Java build steps as described above.

To test the final result from the command line, note that the libcld/0.0.1-SNAPSHOT/libcld-0.0.1-SNAPSHOT.so artifact must be named libcld.so. You can create a symlink to it, e.g. libcld.so -> libcld-0.0.1-SNAPSHOT.so.

The test string, to be executed from ~/.m2/repository/com/mzsanford/cld:
java -Djava.library.path=./libcld/0.0.1-SNAPSHOT/ -cp cld/1.0-SNAPSHOT/cld-1.0-SNAPSHOT.jar com.mzsanford.cld.CompactLanguageDetector "Je mange un orange ici en France. Je veut un pomme."

You should get the following output:
DETECTED: LanguageDetectionResult [probableLocale=fr, reliable=true, candidates=[LanguageDetectionCandidate [locale=fr, score=100.0, normalizedScore=26.72147995889003]]]

[themarvelousbreadfish]

I am unable to build a properly linked .so by following the same steps on Ubuntu 12.04 64-bit, with the -fPIC flag added to the Maven native:compile step (as described above). I am using OpenJDK 6, however.

The generated library is linked like so:

cld/ports/java$ ldd native/unix/target/libcld.so 
    linux-vdso.so.1 =>  (0x00007fff061ff000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f20e1400000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f20e19f1000)

[themarvelousbreadfish]

An additional detail: building with gcc 4.4 on Ubuntu 12.04 64-bit will generate properly linked JNI bindings. I can't build a properly linked JNI binding using gcc 4.6 on Ubuntu 12.04 64-bit.

[josephxsxn]

Second this, cannot build a properly linked JNI when using gcc-4.8 on Ubuntu 17, If i try the same with Docker on Centos7 I am able to get a properly linked so file.