Machine Learning Pipeline Stages for Spark (exposed in Scala/Java + Python)
Switch branches/tags
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.

README.md

sparkling ml logo buildstatus codecov.io

sparklingml

Machine Learning Pipeline Stages for Spark (exposed in Scala/Java + Python)

Why?

SparklingML's goal is to expose additional machine learning stages for Spark with the pipeline interface.

Status

Super early! Come join!

Dev mailing list: https://groups.google.com/forum/#!forum/sparklingml-dev

Building

Sparkling ML consists of two components, a Python component and a Java/Scala component. The Python component depends on having the Java/Scala component pre-build which can be done by running ./build/sbt package.

The Python component depends on the package listed in requirements.txt (as well as part of setup.py). Development and testing also requires spacy, nose, codecov, pylint, and flake8.

The script build_and_package.sh builds & tests both the Scala and Python code.

For now this only works with Spark 2.3.0, it needs some changes to support 2.3.1+.

Tests

Are your DocTests failing with

Expected nothing Got: Warning: no model found for 'en' Only loading the 'en' tokenizer.

Make sure you've installed spacy & the en language pack (python -m spacy download en)

Including in your build

SparklingML is not yet ready for production use.

License

SparklingML is licensed under the Apache 2 license. Some additional components may be under a different license.