JVM Package New RoadMap

[tqchen]

Now that #884 is finished, we will be proposing a new roadmap, where more contributors can get involved. Please reply to this issue to add things if you have more thoughts.

• More robust maven packaging, use maven packaging scripts to copy resources, and create native lib, remove create_jni script
  • Maybe @Javelinjs have some experience on this in mxnet-scala project
• Integration with spark dataframe and pipeline api
• Integration with FlinkML API
• Support of google's dataflow as a runtime
• Tutorial on using the spark/flink API from scratch.
• Instruction on best performance settings of executors, and cores
• Documentations on API
• Developer documentations and a blog on how it works
• across application integration (via something like Tachyon)
• external memory support
• upload to maven

[yzhliu]

Awesome. Seems that mxnet can learn from the design and implementation here.

[tqchen]

@Javelinjs Currently we solve the problem of embedding rabit allreduce jobs. We need to think a bit about how to handle server jobs, which involves start container that do not take data. Maybe @CodingCat will also have some thoughts on this

[yzhliu]

About the maven package thing, you could refer to https://github.com/dmlc/mxnet/tree/master/scala-package. I suggest to make native lib an independent module under jvm project. And here's a great reference for how to setup the jni compiling procedure in maven: http://www.tricoder.net/blog/?p=197

I put native lib into assembly jar and load it using https://github.com/dmlc/mxnet/blob/master/scala-package/core/src/main/scala/ml/dmlc/mxnet/util/NativeLibraryLoader.scala
It was mainly inspired by https://github.com/mikiobraun/jblas/blob/master/src/main/java/org/jblas/util/LibraryLoader.java , though I'm not sure whether there may exist any license problem.

[rotationsymmetry]

I can work on the Integration with spark dataframe and pipeline api

[CodingCat]

@rotationsymmetry welcome

[CodingCat]

OK, with the post here http://dmlc.ml/2016/03/14/xgboost4j-portable-distributed-xgboost-in-spark-flink-and-dataflow.html we have finished the first stage of the roadmap,

I will start working on dataframe and pipeline integration tmr

[yzhliu]

Great work!
and FYI
https://issues.apache.org/jira/browse/SPARK-8547
https://issues.apache.org/jira/browse/SPARK-13868

[CodingCat]

hmmm....I glanced it this afternoon...I'm sending the email about our xgboost4j to spark user list tmr morning

[futurely]

"Apache Beam is a unified model for defining both batch and streaming data-parallel processing pipelines, as well as a set of language-specific SDKs for constructing pipelines and Runners for executing them on distributed processing backends like Apache Spark, Apache Flink, and Google Cloud Dataflow."

[CodingCat]

@futurely thanks for information, yeah Beam is of courses on our radar when we considering the future version of XGBoost4J

[maropu]

Any plan to upload xgboost4j into the maven repository? We are planning to integrate xgboost in our product, hivemall, though, this becomes a barrier for the integration. Thanks in advance!