Building ppc64le natives for use in Spark MLlib

Build instructions

git clone https://github.com/ibmsoe/netlib-java.git
cd netlib-java

cd generator
# edit pom.xml to comment out maven-gpg-plugin
mvn clean compile
mvn install

cd ..
# edit native_ref/pom.xml to comment out osx-* and win-* modules
# edit native_system/pom.xml to comment out osx-* and win-* modules
mvn clean compile
mvn install

cd native_system/ppc
# ensure libgfortran, libblas and liblapack are installed.
# See below for alternative implementations.
mvn clean compile
ldd linux-ppc64le/target/netlib-native_system-linux-ppc64le.so
    linux-vdso64.so.1 =>  (0x00003fffa0cb0000)
    libgfortran.so.3 => /usr/lib/powerpc64le-linux-gnu/libgfortran.so.3 (0x00003fffa0950000)
    libblas.so.3 => /usr/lib/libblas.so.3 (0x00003fffa08b0000)
    liblapack.so.3 => /usr/lib/liblapack.so.3 (0x00003fffa0330000)
    libc.so.6 => /lib/powerpc64le-linux-gnu/libc.so.6 (0x00003fffa0170000)
    libm.so.6 => /lib/powerpc64le-linux-gnu/libm.so.6 (0x00003fffa0090000)
    libgcc_s.so.1 => /lib/powerpc64le-linux-gnu/libgcc_s.so.1 (0x00003fffa0060000)
    /lib64/ld64.so.2 (0x0000000049ca0000)
mvn install

cd ../../all
# edit pom.xml to comment out non-ppc native dependencies
mvn clean
mvn install

Bundle the following in a new jar

• Contents of native_system/java/target/classes
• Native library at native_system/ppc/linux-ppc64le/target/netlib-native_system-linux-ppc64le.so
• Contents of jniloader jar with ppc64le support (https://github.com/fommil/jniloader)

jar -tf native-netlib.jar

META-INF/
META-INF/MANIFEST.MF
com/
com/github/
com/github/fommil/
com/github/fommil/netlib/
com/github/fommil/netlib/NativeSystemBLAS.class
com/github/fommil/netlib/NativeSystemLAPACK.class
com/github/fommil/netlib/NativeSystemARPACK.class
com/github/fommil/jni/
com/github/fommil/jni/JniNamer.class
com/github/fommil/jni/JniLoader.class
netlib-native_system-linux-ppc64le.so

Ensure the jar is in the classpath of your spark executor(s).

Alternative BLAS implementations

OpenBLAS

Clone and build OpenBLAS library from https://github.com/xianyi/OpenBLAS

cd native_system/ppc
# edit linux-ppc64le/pom.xml to specify -lopenblas in place of -lblas and -llapack
# also add a "-L<library_path_to_libopenblas>" if necessary.
mvn clean compile
ldd linux-ppc64le/target/netlib-native_system-linux-ppc64le.so
    linux-vdso64.so.1 =>  (0x00003fff894a0000)
    libgfortran.so.3 => /usr/lib/powerpc64le-linux-gnu/libgfortran.so.3 (0x00003fff890f0000)
    libopenblas.so.0 => <library_path>/libopenblas.so.0 (0x00003fff885d0000)
    libc.so.6 => /lib/powerpc64le-linux-gnu/libc.so.6 (0x00003fff88410000)
    libm.so.6 => /lib/powerpc64le-linux-gnu/libm.so.6 (0x00003fff88330000)
    libgcc_s.so.1 => /lib/powerpc64le-linux-gnu/libgcc_s.so.1 (0x00003fff88300000)
    libpthread.so.0 => /lib/powerpc64le-linux-gnu/libpthread.so.0 (0x00003fff882d0000)
    /lib64/ld64.so.2 (0x000000004e090000)

ESSL

Work in progress. Still working to resolve all symbols needed by libessl.

cd native_system/ppc
# edit linux-ppc64le/pom.xml to specify -lessl in place of -lblas and -llapack
mvn clean compile
ldd linux-ppc64le/target/netlib-native_system-linux-ppc64le.so
    linux-vdso64.so.1 =>  (0x00003fffb4c20000)
    libgfortran.so.3 => /usr/lib/powerpc64le-linux-gnu/libgfortran.so.3 (0x00003fffb48c0000)
    libessl.so.1 => /usr/lib/libessl.so.1 (0x00003fffb2680000)
    libc.so.6 => /lib/powerpc64le-linux-gnu/libc.so.6 (0x00003fffb24c0000)
    libm.so.6 => /lib/powerpc64le-linux-gnu/libm.so.6 (0x00003fffb23e0000)
    libgcc_s.so.1 => /lib/powerpc64le-linux-gnu/libgcc_s.so.1 (0x00003fffb23b0000)
    /lib64/ld64.so.2 (0x00000000557f0000)