If you want to query Cassandra data via JDBC but you want to use the power of Spark SQL for data processing, you need this application.
This app (CSJB), which is a Spark app, will automatically registers all Cassandra tables as schema RDDs in Spark SQL and starts an embedded Apache HiveThriftServer to make the RDDs ready to be consumed via "jdbc:hive2" protocol.
With this bridge/server, you can use any BI tool like Jasper, Pentaho or Tableau to query Cassandra tables via Spark SQL. You will be able to do many different aggregation and/or filtering operations like group by, join, etc.. with Cassandra data.
This application uses the Datastax's spark-cassandra-connector
Notes:
-
Cassandra tables are registered in Spark SQL using naming format below: keyspace_tableName Example: "mykeyspace.test" table in Cassandra will be registered as "mykeyspace_mytable" schema RDD.
-
As of now, CSJB only supports Cassandra tables with the following data types:
AsciiType, BooleanType, BytesType,CounterColumnType, DateType, DecimalType, DoubleType, LongType, FloatType, Int32Type, IntegerType, LexicalUUIDType, UTF8Type, UUIDType, TimestampType, ReversedType (for TimestampType). More data types will be added in the near future (your contributions will be greatly appreciated).
============= Getting started
Prerequisites
- Install and start Cassandra 2.1
- Install and start Spark 1.2
Build
sbt assembly copyAssets
A fat assembly jar and config files will be generated in "target/dist"
Set environment variables
export INADCO_CSJB_HOME=`pwd`/target/dist
export SPARK_HOME=YOUR_SPARK_HOME_DIR
Enter Cassandra credentials and Spark master url
Default config values are in $INADCO_CSJB_HOME/dist/config/csjb-default.properties
spark.cassandra.connection.host=localhost
spark.cassandra.auth.username=CASSSANDRA_USER_NAME
spark.cassandra.auth.password=CASSSANDRA_PASSWORD
inadco.spark.master=local[2]
Start CSJB server
$INADCO_CSJB_HOME/bin/start.sh
The CSJB server is started at port 10000 (to stop it, run $INADCO_CSJB_HOME/bin/stop.sh) Status of CSJCB app is available via your Spark UI http://YOUR_SPARK_MASTER:8080/
Query your data
Run a client which supports "jdbc:hive2" protocol (e.g "beeline") and connect your CSJB server to query existing data in Cassandra (if you don't have the data yet, run cql and create a test table and insert some data in Cassandra first)
$SPARK_HOME/bin/beeline
beeline> !connect jdbc:hive2://localhost:10000
0: jdbc:hive2://localhost:10000> select id, sum(count) from mykeyspace_mytable group by id;