SnappyData - The Spark Database. Stream, Transact, Analyze, Predict in one cluster
Scala Java Shell Python Other
Latest commit 6680e09 Feb 25, 2017 @jramnara jramnara committed on GitHub Make working-with-objects example simpler
If the case class is available one doesn't have to iterate Row objects and map to Person. You can use DF.as[Person]
Permalink
Failed to load latest commit information.
.github * Renaming gemfirexd* jars to snappydata-store* jars. (#283) Jun 24, 2016
cluster SNAP-1342: TPCHDUnitTest. testSnappy fails Feb 2, 2017
core Fix for AQP-211 (#508) Feb 9, 2017
docs Make working-with-objects example simpler Feb 24, 2017
dtests changed snappy-shell to snappy (#504) Jan 31, 2017
dunit Match Xms and Xmx for dunit JVMs so store ResourceManager gets setup … Jan 12, 2017
examples changed snappy-shell to snappy (#504) Jan 31, 2017
gradle/wrapper [SNAP-606] Support for "spark.snappydata" properties (#231) May 9, 2016
python/pyspark a) Corrected Python objects to correctly use SparkSession APIs. (#460) Dec 9, 2016
release updating year in copyright header templates Jan 24, 2017
spark @ 6fd1706 linking spark, job server and store Feb 2, 2017
spark-jobserver @ 92f5a0c linking spark, job server and store Feb 2, 2017
store @ 174f10c linking spark, job server and store Feb 2, 2017
tests SNAP-1342: TPCHDUnitTest. testSnappy fails Feb 2, 2017
.gitignore Move to Spark 2.0 (#276) Aug 17, 2016
.gitmodules Move to Spark 2.0 (#276) Aug 17, 2016
LICENSE Move to Spark 2.0 (#276) Aug 17, 2016
NOTICE Move to Spark 2.0 (#276) Aug 17, 2016
README.md Fix broken link (#510) Feb 16, 2017
ReleaseNotes.txt * SnappyData Version 0.7 Dec 21, 2016
build.gradle * Update to gradle-scalatest version 0.13.1 Jan 25, 2017
codeStyleSettings.xml moving to mavenCentral() to jcenter() which is supposed to be faster … Oct 10, 2015
gradle.properties Adding top-level mavenPublish target which packages sources,javadocs … Jan 30, 2016
gradlew --TPCH Chnages Jul 26, 2016
gradlew.bat Move to Spark 2.0 (#276) Aug 17, 2016
mkdocs.yml Updated Documentation Jan 24, 2017
publish-site.sh Some basic sanity put in place to fail publishing of docs when api do… Feb 17, 2017
scalastyle-config.xml Adding support to run scalaStyle in product build (SNAP-120) Jan 29, 2016
settings.gradle [SNAP-1190] Changes for updates to spark layer (#435) Dec 3, 2016

README.md

SnappyData fuses Apache Spark with an in-memory database to deliver a data engine capable of processing streams, transactions and interactive analytics in a single cluster.

The Challenge with Spark and Remote Data Sources

Apache Spark is a general purpose parallel computational engine for analytics at scale. At its core, it has a batch design center and is capable of working with disparate data sources. While this provides rich unified access to data, this can also be quite inefficient and expensive. Analytic processing requires massive data sets to be repeatedly copied and data to be reformatted to suit Spark. In many cases, it ultimately fails to deliver the promise of interactive analytic performance. For instance, each time an aggregation is run on a large Cassandra table, it necessitates streaming the entire table into Spark to do the aggregation. Caching within Spark is immutable and results in stale insight.

The SnappyData Approach

At SnappyData, we take a very different approach. SnappyData fuses a low latency, highly available in-memory transactional database (GemFireXD) into Spark with shared memory management and optimizations. Data in the highly available in-memory store is laid out using the same columnar format as Spark (Tungsten). All query engine operators are significantly more optimized through better vectorization and code generation. The net effect is, an order of magnitude performance improvement when compared to native Spark caching, and more than two orders of magnitude better Spark performance when working with external data sources.

Essentially, we turn Spark into an in-memory operational database capable of transactions, point reads, writes, working with Streams (Spark) and running analytic SQL queries. Or, it is an in-memory scale out Hybrid Database that can execute Spark code, SQL or even Objects.

If you are already using Spark, experience 20x speed up for your query performance. Try out this test

Snappy Architecture

SnappyData Architecture

Getting Started

We provide multiple options to get going with SnappyData. The easiest option is, if you are already using Spark 2.0+. You can simply get started by adding SnappyData as a package dependency. You can find more information on options for running SnappyData here.

SnappyData in 5 Minutes!

Refer to the 5 minutes guide which is intended for both first time and experienced SnappyData users. It provides you with references and common examples to help you get started quickly!

Documentation

To understand SnappyData and its features refer to the documentation

Community Support

We monitor channels listed below for comments/questions.

Stackoverflow Stackoverflow SlackSlack Gitter Gitter Mailing List Mailing List Reddit Reddit JIRA JIRA

Link with SnappyData Distribution

Using Maven Dependency SnappyData artifacts are hosted in Maven Central. You can add a Maven dependency with the following coordinates:

groupId: io.snappydata
artifactId: snappydata-core_2.11
version: 0.7

groupId: io.snappydata
artifactId: snappydata-cluster_2.11
version: 0.7

Using sbt If you are using sbt, add this line to your build.sbt for core SnappyData artifacts:

libraryDependencies += "io.snappydata" % "snappydata-core_2.11" % "0.7"

For additions related to SnappyData cluster, use:

libraryDependencies += "io.snappydata" % "snappydata-cluster_2.11" % "0.7"

You can find more specific SnappyData artifacts here

Ad Analytics using SnappyData

Here is a stream + Transactions + Analytics use case example to illustrate the SQL as well as the Spark programming approaches in SnappyData - Ad Analytics code example. Here is a screencast that showcases many useful features of SnappyData. The example also goes through a benchmark comparing SnappyData to a Hybrid in-memory database and Cassandra.

Contributing to SnappyData

If you are interested in contributing, please visit the community page for ways in which you can help.