Using Python trained LightGBM model to predict on Spark DataFrame (in Databricks)

[jmpanfil]

I am curious if there is a straightforward way to use a regular python trained LightGBM model on a distributed Spark DataFrame. The model was trained using version 4.3.0 and the input data is a scipy.sparse csr_matrix. I have used XGBoost in a similar fashion, and the solution there was creating a Spark dataframe where each row had an ID column and a feature column that was a org.apache.spark.ml.linalg.Vectors.sparse vector. The regular non-distributed XGBoost model was loaded into their Spark version model and then calling model.xgbRegressionModel.transform(df_sparse). Is there a similar solution for LightGBM where I can load in a non-distributed trained model into Spark and predict over a distributed dataframe?

I am using Databricks with LightGBM version 4.3.0 (4.5.0 also available). Python or Scala solutions are both welcome. Thank you!

[jameslamb]

Thanks for using LightGBM.

We don't ship a direct pyspark integration in the lightgbm Python package, the way that xgboost does with xgboost.pyspark (xgboost PySpark docs), and don't have any plans to.

Please try SynapseML, another project from Microsoft which supports using LightGBM with Apache Spark.

• docs: https://microsoft.github.io/SynapseML/docs/Explore%20Algorithms/LightGBM/Overview/
• code: https://github.com/microsoft/SynapseML

I think something synapse.ml.lightgbm.LightGBMRegressor.loadNativeModelFromFile() from this example will do what you're looking for: https://github.com/microsoft/SynapseML/blob/9e0fde448d2600e6383be008720d11fab6fb1658/docs/Explore%20Algorithms/LightGBM/Quickstart%20-%20Classification%2C%20Ranking%2C%20and%20Regression.ipynb#L215