mmlspark.lightgbm._LightGBMClassifier does not exist

[goodwanghan]

Describe the bug
mmlspark.lightgbm._LightGBMClassifier does not exist

To Reproduce
I git cloned the repo and sys.path.append the mmlspark python path, import mmlspark has no issue, but the classifier inside can't be used

There is no clear instruction on how to install mmlspark for python.

spark = pyspark.sql.SparkSession.builder.appName("MyApp") \
            .config("spark.jars.packages", "com.microsoft.ml.spark:mmlspark_2.11:1.0.0-rc1") \
            .getOrCreate()
from mmlspark.lightgbm import LightGBMClassifier
model = LightGBMClassifier(learningRate=0.3,
                           numIterations=100,
                           numLeaves=31).fit(train)
spark.stop()

Expected behavior
from mmlspark.lightgbm import LightGBMClassifier should work

Info (please complete the following information):

• MMLSpark Version: from latest repo
• Spark Version 2.4.4
• Spark Platform: custom platform not on azure

** Stacktrace**

ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-9-1b16cbc5ea7e> in <module>
      5             .config("spark.jars.packages", "com.microsoft.ml.spark:mmlspark_2.11:1.0.0-rc1") \
      6             .getOrCreate()
----> 7 from mmlspark.lightgbm import LightGBMClassifier
      8 model = LightGBMClassifier(learningRate=0.3,
      9                            numIterations=100,

/mnt/user-home/git/mmlspark/src/main/python/mmlspark/lightgbm/LightGBMClassifier.py in <module>
      9     basestring = str
     10 
---> 11 from mmlspark.lightgbm._LightGBMClassifier import _LightGBMClassifier
     12 from mmlspark.lightgbm._LightGBMClassifier import _LightGBMClassificationModel
     13 from pyspark import SparkContext

ModuleNotFoundError: No module named 'mmlspark.lightgbm._LightGBMClassifier'

Additional context
I tried this on Jupyter on a Linux machine. Does this only work on Azure?

[welcome]

👋 Thanks for opening your first issue here! If you're reporting a 🐞 bug, please make sure you include steps to reproduce it.

[imatiach-msft]

@goodwanghan sorry about the trouble you are having. You need to run the autogen to autogenerate the wrappers, _LightGBMClassifier is automatically generated from the scala API.
" I tried this on Jupyter on a Linux machine. Does this only work on Azure?"
LightGBM is open source and works anywhere, linux, windows, macos. The mmlspark integration can be run anywhere, Azure, Cloudera, AWS, etc. If it's not running somewhere then that's unexpected and probably a bug.

[candalfigomoro]

@imatiach-msft
Instead of generating the wrappers, because of restrictions on my business laptop, I usually extracted the wrappers from the JAR package that I got through maven. The problem is that for version 1.0-rc1 I can't download the file (while it worked fine for version 0.18.x). See #715 (comment)

Also, if I look here https://search.maven.org/search?q=g:com.microsoft.ml.spark the last version is 0.18.1. Why can't I find version 1.0-rc1?

Thank you

[mhamilton723]

@candalfigomoro see the fix i posted in #715 apologize for the confusion!

[arijeetm1]

facing similar issue with the LightGBMRegressor module as well.
mmlspark : 0.17
pyspark : 2.4.4
Scala: 2.11.12

[arijeetm1]

Looks like in version 0.17, the name has changed from mmlspark.lightgbm to specific module name repeated. It worked for me by using:
from mmlspark.LightGBMRegressor import LightGBMRegressor
Please update the sample in the documentation.

Also could not find vowpalWabbit module in the latest package from https://github.com/Azure/mmlspark/archive/bba5c10ff774a7541be4cde7438ba710bd51f5e6.zip
Although could find it in the master : https://github.com/arijeetm1/mmlspark/blob/master/src/main/python/mmlspark/vw/VowpalWabbitRegressor.py
Wondering if it's released yet? maybe in latest version 0.18 but could find it https://spark-packages.org/package/Azure/mmlspark

[surajiyer]

@goodwanghan sorry about the trouble you are having. You need to run the autogen to autogenerate the wrappers, _LightGBMClassifier is automatically generated from the scala API.
" I tried this on Jupyter on a Linux machine. Does this only work on Azure?"
LightGBM is open source and works anywhere, linux, windows, macos. The mmlspark integration can be run anywhere, Azure, Cloudera, AWS, etc. If it's not running somewhere then that's unexpected and probably a bug.

I am running into this issue as well but I do not understand what this means. Can you please explain step-by-step, on a linux machine with Python 3.7 Jupyter + Spark 2.4.5, how can I run the autogen?