Machine Learning Pipeline Stages for Spark (exposed in Scala/Java + Python)
Switch branches/tags
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.

sparkling ml logo buildstatus


Machine Learning Pipeline Stages for Spark (exposed in Scala/Java + Python)


SparklingML's goal is to expose additional machine learning stages for Spark with the pipeline interface.


Super early! Come join!

Dev mailing list:!forum/sparklingml-dev


Sparkling ML consists of two components, a Python component and a Java/Scala component. The Python component depends on having the Java/Scala component pre-build which can be done by running ./build/sbt package.

The Python component depends on the package listed in requirements.txt (as well as part of Development and testing also requires spacy, nose, codecov, pylint, and flake8.

The script builds & tests both the Scala and Python code.

For now this only works with Spark 2.3.0, it needs some changes to support 2.3.1+.


Are your DocTests failing with

Expected nothing Got: Warning: no model found for 'en' Only loading the 'en' tokenizer.

Make sure you've installed spacy & the en language pack (python -m spacy download en)

Including in your build

SparklingML is not yet ready for production use.


SparklingML is licensed under the Apache 2 license. Some additional components may be under a different license.