Visualize streaming machine learning in Spark
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.
animations/databricks-blog-post Add cluster dying animation Jan 26, 2015
python - default tmp dir if not specified Mar 8, 2015
scala Change scala package name Jan 26, 2015
.gitignore Fix gitignore Jan 26, 2015
LICENSE Add license Jan 26, 2015 Update Mar 11, 2015

Join the chat at

Visualize streaming machine learning in Spark

two-dimensional-demo one-dimensional-demo


This Python app generates data, analyzes it in Spark Streaming, and visualizes the results with Lightning. The analyses use streaming machine learning algorithms included with Spark as of version 1.2. The demos are designed for local use, but the same algorithms can run at scale on a cluster with millions of records.

How to use

To run these demos, you need:

  • A working installation of Spark
  • A running Lightning server
  • An installation of Python with standard scientific computing libraries (NumPy, SciPy, ScikitLearn)

With those three things in place, install using:

pip install spark-ml-streaming

Then set SPARK_HOME to your Spark installation, and start an executable:

streaming-kmeans -l <lighting_host>

Where lightning_host is the address of your Lightning server. After it starts, your browser will open, and you should see data appear shortly.

Try running with different settings, for example, to run a 1-d version with 4 clusters and a half-life of 10 points:

streaming-kmeans -p <temporary_path> -l <lighting_host> -nc 4 -nd 1 -hl 10 -tu points

Where temporary_path is where data will be written / read, if not specified the current tmp directory will be used (See Python tempfile.gettempdir())

2D data will make a scatter plot and 1D data will make a line plot. You can set this with -nd.

To see all options type:

streaming-kmeans -h


The demo relies on a Scala package included pre-built inside python/mlstreaming/lib. To rebuild it, use sbt:

cd scala
sbt package