Databricks Jobs failing to import mmlspark. ModuleNotFoundError: No module named 'mmlspark'

[nataliabernardo]

Hi, I'm able to run fine mmlspark on my notebook in Databricks. However, when I try to run this same notebook in a scheduled job, it fails and retrieves the message: ModuleNotFoundError: No module named 'mmlspark' .

The version currently of installed on my cluster is "Azure:mmlspark:0.17"

Do you have any recommendation on how to set up a Job to run mmlspark correctly?
Thanks

[mhamilton723]

Hey @nataliabernardo consider using our spark package install on databricks described in this section

https://github.com/Azure/mmlspark#databricks

This link gives a detailed step by step of the databricks UI

https://docs.microsoft.com/en-us/azure/cognitive-services/big-data/getting-started#azure-databricks

Please feel free to comment or re-open if this doesn't work!

[nataliabernardo]

Hi @mhamilton723,

I followed the steps and installed "com.microsoft.ml.spark:mmlspark_2.11:1.0.0-rc3" and attached to my cluster. Also uninstalled the other version ("Azure:mmlspark:0.17").

After that, both notebook and job are failing to run

• When try running the Job: ModuleNotFoundError: No module named 'mmlspark'
• When try running the Notebook: ImportError: cannot import name 'LightGBMRegressor' from 'mmlspark' (/local_disk0/spark-2d6934aa-583e-441f-a43b-05885373d248/userFiles-1aa6ffb5-d69b-479a-b75e-9640f9a7fbdf/addedFile7238710938107874708mmlspark_2_11_1_0_0_rc3-86e1e.jar/mmlspark/init.py)

Spark version: 6.6 (includes Apache Spark 2.4.5, Scala 2.11)