xgboost

[TarasDerevianko]

Do you plan realize something like ml_xgboost?

(based on https://libraries.io/github/dmlc/xgboost)

[javierluraschi]

@TarasDerevianko No plans for this, so this could be a great extension package.

[TarasDerevianko]

I have builded version of this library for my server. How import this library when I start spark connection in sparklyr package?

I can use sparklyr::invoke for xgboost-library functions?

[javierluraschi]

@TarasDerevianko if this is already correctly configured in the cluster, then you might still need to spark_connect with a config that contains the right sparklyr.jars.default or if there is a Spark package wrapping xgboost then a config with the right sparklyr.defaultPackages. Then you should be able to use invoke, invoke_static, etc. to use the package.

However, it is common to see in packages the use of functional parameters, as in map(x => x.filter()), if that's the case for xgboost, then you would have to write scala code that wraps the mapping and exposes simple interfaces to sparklyr. You can see an example on how to call compile and deploy scala code in sparklyr here: https://github.com/javierluraschi/sparkhello

[harryprince]

@TarasDerevianko I am seeking XGboost on sparklyr too

[datalee]

@harryprince i know h2o have the xgb,you can try.

[kevinykuo]

This should be possible utilizing https://github.com/dmlc/xgboost/blob/master/jvm-packages/xgboost4j-spark/src/main/scala/ml/dmlc/xgboost4j/scala/spark/XGBoostEstimator.scala after #1180 is resolved.

[albedan]

@kevinykuo Probably the latest release of Xgboost (0.80) has some interesting updates for integration in Sparklyr.

Consolidated APIs: It is now much easier to integrate XGBoost models into a Spark ML pipeline

[harryprince]

is anyone successfully deploy xgboost with sparklyr?

Currently, it not stable in sparklyr::spark_apply way.

[lorenzwalthert]

For future reference, it's here: https://github.com/rstudio/sparkxgb. Should this issue be closed?