visualization server
JavaScript CSS Scala HTML Other
Latest commit fab6a4b Oct 13, 2015 @mighdoll updated gdoc links
Failed to load latest commit information.
avro-loader/src Move to 2.11 Jun 5, 2015
css bar chart Sep 17, 2015
dashboard/demos angular installed w/basic demo on admin page Mar 11, 2015
data-server/src disable some plot tests rather than ignoring them Sep 17, 2015
http-common/src quiet some logging in sparkle-http, by using testkit for unit tests Sep 17, 2015
js fix some js overview doc Sep 17, 2015
jsdoc minor fixes to component model doc Sep 17, 2015
kafka fixing testcase shutdown mem leaks Sep 17, 2015
log4j remove debug printout in log configuration Sep 17, 2015
project fix doc task issue where scaladoc complained about foreign links Sep 17, 2015
protocol/src/main initial pass at sparkle graph js doc Sep 17, 2015
servers create it tests for util-kafka Nov 15, 2014
spark-avro/src/main/scala/nest/sparkle/loader/spark/avro TaggedBlock and TaggedBlock2 cleanup Mar 11, 2015
spark/src quiet build by not trying to run integration tests on projects w/o in… Sep 17, 2015
sparkle-store-tests/src skip intermittent test Sep 17, 2015
sparkle-tests fixing more shutdown mem leaks Sep 17, 2015
sparkle revise & import lots of documentation Sep 17, 2015
store-test-kit/src/main quiet build by not trying to run integration tests on projects w/o in… Sep 17, 2015
store/src/main bump timeout for schema creation to 20 seconds to help busy integrati… Sep 17, 2015
test-kit quiet build by not trying to run integration tests on projects w/o in… Sep 17, 2015
util-kafka/src fixing more shutdown mem leaks Sep 17, 2015
util-tests/src/test Add request time metrics on api server http requests and websocket re… May 20, 2015
.gitignore ignore mac fs cruft Mar 11, 2015
LICENSE updated gdoc links Oct 13, 2015
ToDo adjust styling to make remove button more prominent May 20, 2015 revise & import lots of documentation Sep 17, 2015
scalastyle-config.xml Explicitly declare procedure return type as Unit. May 6, 2014


This is the 0.5.x version of Sparkle.


Sparkle is a small suite of libraries for making interactive data visualizations. Sparkle includes a javascript client library and a scala/jvm server library. The client and server are connected by websockets protocol for requesting and delivering data for visualization.

As a standalone tool, sparkle gives users an easy way to make visualize data from a directory of .csv files. The graphs are interactively zoomable in a d3 based web interface. With some easy customization, graphs can be aggregated into dashboards, and customized with simple declarative javascript, and data read in from other sources such as Apache Kafka.

See Sparkle Intro Talk for an introduction presentation. See Visualize the Things for a brief introduction to sparkle, lessons on garbage collection with reative streams, and comments on the future of the Lambda Architecture. See The Live Layer for a discussion of the Live Layer, a key component missing from the Lambda Architecture. See Sparkle Google Docs Folder for other docs.


Sparkle contains three core components:

Sparkle Data Server

sparkle-data-server is a mini web server that provides HTTP and websocket apis for data visualization. Sparkle-data-server collects data from an extensible set of inputs (Apache Kafka and .csv/.tsv data files, hadoop and netcat support coming soon). Sparkle-data-server stores data in Cassandra.
Sparkle-time also hosts an extensible set of data transformations that transform data on demand (e.g. for aggregation).

Sparkle-time can be run as a library inside another server, or it can run standalone if no customization is needed.


sg.js is a d3.js based javascript library for displaying interactive charts of potentially large data sets. Charts are customizable with declarative javascript, and extensible with d3. sg.js currently offers support for line charts, scatter plots, bar charts, etc. and is easily extensible. All charts support a drag-based zooming UI.

sg.js also includes a dashboard component for constructing pages that aggregate many charts on the same page. Graphs in the dashboard are resizable. The browser's back button works to undo/redo zoom navigation and chart resizing.

sg.js supports the sparkle-time api, enabling zooming charts of potentially huge server hosted data sets, the option of offloading data transformations to the server, and (soon) server pushed updates to locally displayed charts.

Sparkle Data API

To enable interoperation of multiple clients and servers, the sparkle-time server and sg.js javascript library speak a well-defined data protocol: the Sparkle Data Protocol. The protocol allows clients to request data transformations and data streams from a comformant server.

See Sparkle Data Protocol for details of the protocol. See Sparkle Transforms for the current set of built in transforms.


Sparkle is currently at version 0.5.x. Sparkle 0.6 will:

  • clean out legacy code
  • more fully support the Sparkle Data Protocol between visualization client and visualizaton server.

Many more future features are listed in the todo list

Using sparkle

See Using-sg

Contributing to Sparkle

See Contributing, Building, and javascript component model.