[jvm-packages] xgboost4j-0.7.jar missing libxgboost4j.so in Mac

[jq]

Environment info

Operating System:
Mac, Sierra
Compiler:

Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 8.0.0 (clang-800.0.42.1)
Target: x86_64-apple-darwin16.1.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

Package used (python/R/jvm/C++):

xgboost version used:
b9a9d2b

If installing from source, please provide

1. The commit hash (git rev-parse HEAD)
   b9a9d2b

2. Logs will be helpful (If logs are large, please upload as attachment).
   build jar ok, and compiled no problem,

but the jar has no libxgboost4j.so

Steps to reproduce

1. make libxgboost.so, no error
2. mvn -DskipTests=true package as https://xgboost.readthedocs.io/en/latest/jvm/index.html no problem, saw jar under target/xgboost4j-0.7.jar
3. jar tvf xgboost4j/target/xgboost4j-0.7.jar | grep libxgboost4j.so

nothing found

What have you tried?

1. check out 0.6 tag, same result.
2. 0.5 build under linux, able to include .so in jar.
3. run create_jni.sh under jvm-packages, and then rebuild jar, still no .so file inside.

[CodingCat]

jar tvf xgboost4j/target/xgboost4j-0.7.jar | grep libxgboost4j.dylib

[jq]

@CodingCat Thanks for quick answer, I do find libxgboost4j.dylib, but after I run the jar, it report missing libxgboost4j.so File /lib/libxgboost4j.so was not found inside JAR I use gradle compile files('local-lib/xgboost4j-0.7.jar') to use it, am I done something wrong here?

[jq]

ok, I guess that's because the .so I build on mac, can't run on linux

[CodingCat]

ah. Geneally you cannot build in mac and run in linux, because xgboost4j will adjust the suffix of the native lib file based on your os

[CodingCat]

You can modify the build script to work around it

[CodingCat]

clicked close and comment mistakenly....

@jq You are more than welcome to share Airbnb's story about XGBoost/XGBoost4J, if possible

We have an incomplete list here: https://github.com/dmlc/xgboost/blob/master/demo/README.md#usecases , other names include EBay, Linkedin, etc. we are trying to complete this list....

[jq]

@CodingCat thanks for help, just started, will share story :)

[CodingCat]

thanks

[hy-2013]

How can I modify the build script, so I can build libxgboost4j.so rather than libxgboost.dylib on mac? @CodingCat
thanks.

[CodingCat]

it is building dylib automatically in the current version of script...has been a long time since last check, @superbobry ?

[superbobry]

@hy-2013 why do you want the so? dylib is the native extension for a shared library on Mac, that's what the linker expects when looking for libraries.

[hy-2013]

First, thanks for your reply. Because I met the follow Exception (/lib/libxgboost4j.so was not found inside JAR), when I was running xgboost4j on spark cluster installed on CentOS 6.6. And the xgboost4j-0.7.jar and xgboost4j-spark-0.7.jar is packaged on mac. So I guess I should build libxgboost4j.so rather than libxgboost.dylib on mac.

The whole Exception stack is:
`
ERROR NativeLibLoader [Executor task launch worker-2]: failed to load library from both native path and jar
ERROR DMatrix [Executor task launch worker-2]: Failed to load native library

java.io.FileNotFoundException: File /lib/libxgboost4j.so was not found inside JAR.
at ml.dmlc.xgboost4j.java.NativeLibLoader.createTempFileFromResource(NativeLibLoader.java:121)
at ml.dmlc.xgboost4j.java.NativeLibLoader.loadLibraryFromJar(NativeLibLoader.java:64)
at ml.dmlc.xgboost4j.java.NativeLibLoader.smartLoad(NativeLibLoader.java:152)
at ml.dmlc.xgboost4j.java.NativeLibLoader.initXGBoost(NativeLibLoader.java:40)
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1124)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1124)
at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:251)
at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:61)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:357)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:356)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:60)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
`

thanks again.

[superbobry]

Are you using the latest master version? How did you build it?

[hy-2013]

My building process as follows:

git clone --recursive https://github.com/dmlc/xgboost
git pull && git submodule init && git submodule update && git submodule status
cp make/minimum.mk ./config.mk
comment line 20 of CMakeLists.txt as #2596
make -j4
cd jvm-packages
mvn package

It is success.
I scp xgboost4j-0.7.jar and xgboost4j-spark-0.7.jar to spark cluster, and add them to spark by "--jars", then the FileNotFoundException appears.

[superbobry]

Could you upload the resulting JAR for xgboost4j here? Also, are you building on Linux?

[hy-2013]

I am building on mac (version 10.12.6) not on Linux.

My resulting JAR for xgboost4j as follow:
xgboost4j_jar.zip

thanks.

[superbobry]

Judging by the error message, you're then running on Linux, right?

The build does not currently do cross compilation, so if you need to compile on the same platform you intend to run on. Or, alternatively, use the unofficial precompiled binaries.

[hy-2013]

I download the latest version of unofficial precompiled binaries and package them all in my jar.
but when running on spark cluster, it occurs ERROR (java.lang.NoClassDefFoundError: scala/runtime/AbstractPartialFunction$mcVL$sp).

I google the ERROR, someone say it is caused by version mismatch. Does the unofficial precompiled binaries correspond with the certain version of scala or spark?
My scala and spark version as follows:
<scala.version>2.11.8</scala.version>
<scala.binary.version>2.11</scala.binary.version>
<spark.version>2.0.2</spark.version>

The whole EROOR stack is:
Uncaught error from thread [RabitTracker-akka.actor.default-dispatcher-2] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[RabitTracker] java.lang.NoClassDefFoundError: scala/runtime/AbstractPartialFunction$mcVL$sp at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:800) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) at java.net.URLClassLoader.access$100(URLClassLoader.java:71) at java.net.URLClassLoader$1.run(URLClassLoader.java:361) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at ml.dmlc.xgboost4j.scala.rabit.handler.WorkerDependencyResolver.receive(RabitTrackerHandler.scala:284) at akka.actor.ActorCell.newActor(ActorCell.scala:558) at akka.actor.ActorCell.create(ActorCell.scala:578) at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456) at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478) at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263) at akka.dispatch.Mailbox.run(Mailbox.scala:219) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Caused by: java.lang.ClassNotFoundException: scala.runtime.AbstractPartialFunction$mcVL$sp at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

thanks.

[superbobry]

As the README states, the binaries are Scala 2.10 only for now.

[hy-2013]

Thanks for your reply.

For the version ERROR, I changed scala and spark verson as follows:
<scala.version>2.10.4</scala.version>
<scala.binary.version>2.10</scala.binary.version>
<spark.version>1.6.3</spark.version>

then I re-package jar. The bug above disappeared, but I meet an another ERROR:

ERROR ApplicationMaster [Driver]: User class threw exception: ml.dmlc.xgboost4j.java.XGBoostError: XGBoostModel training failed ml.dmlc.xgboost4j.java.XGBoostError: XGBoostModel training failed at ml.dmlc.xgboost4j.scala.spark.XGBoost$.postTrackerReturnProcessing(XGBoost.scala:358) at ml.dmlc.xgboost4j.scala.spark.XGBoost$.trainDistributed(XGBoost.scala:334) at ml.dmlc.xgboost4j.scala.spark.XGBoost$.trainWithRDD(XGBoost.scala:288) at com.bj.rec.rank.alg.xgboost.SparkWithRDD$.main(SparkWithRDD.scala:68) at com.bj.rec.rank.alg.xgboost.SparkWithRDD.main(SparkWithRDD.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:558) 17/08/29 20:22:45 INFO ApplicationMaster [Driver]: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: ml.dmlc.xgboost4j.java.XGBoostError: XGBoostModel training failed) 17/08/29 20:22:45 INFO SparkContext [Thread-3]: Invoking stop() from shutdown hook 17/08/29 20:22:45 ERROR RabitTracker [Thread-60]: Uncaught exception thrown by worker: java.lang.IllegalStateException: SparkContext has been shutdown at org.apache.spark.SparkContext.runJob(SparkContext.scala:1824) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1845) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1858) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1929) at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:920) at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:918) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111) at org.apache.spark.rdd.RDD.withScope(RDD.scala:316) at org.apache.spark.rdd.RDD.foreachPartition(RDD.scala:918) at ml.dmlc.xgboost4j.scala.spark.XGBoost$$anon$1.run(XGBoost.scala:326) 17/08/29 20:22:45 ERROR RabitTracker [Thread-60]: java.lang.InterruptedException: sleep interrupted

I have noticed #1911 and #1682 etc. Unfortunately, none of them can resolve the bug. Wish all your reply @CodingCat @superbobry .

thanks again.

[superbobry]

This error could pretty much mean anything. Could you upload the full executor log?

[hy-2013]

Thanks for your reply @superbobry .

My full executor log:
executor_application_1502943842013_1249774.txt

thanks.

[superbobry]

Here's the relevant bit

17/08/29 20:22:45 INFO RabitTracker$TrackerProcessLogger [Thread-59]:     import argparse
17/08/29 20:22:45 INFO RabitTracker$TrackerProcessLogger [Thread-59]: ImportError: No module named argparse

It seems you're using Python 2.6. In order to fix this either update to 2.7 or later, or install argparse.

[hy-2013]

Thanks for your reply again @superbobry .

Is the argparse package indispensable? I mean whether exists a early version of xgboost4j which does not need argparse or not? Since python using in spark cluster in my company may be upgraded difficultly and I am out of control.

thanks.

[superbobry]

You don't have to upgrade Python, just pip install argparse.

Otherwise, no it's not indispensable, it can be swapped to optparse.

[hy-2013]

Thanks for your reply @superbobry.
How can I swap to optparse when I use unofficial precompiled binaries and the binaries are still Scala 2.10 only for now, i.e. 0.7-criteo-20171220?