Home

Shark (Hive on Spark)

Shark is a large-scale data warehouse system for Spark designed to be compatible with Apache Hive. It can answer Hive QL queries up to 30 times faster than Hive without modification to the existing data nor queries. Shark supports Hive's query language, metastore, serialization formats, and user-defined functions.

Caching and Performance Optimizations

Analytical queries usually focus on a particular subset or time window, e.g., http logs from the previous month, touching only the (small) dimension tables and a small portion of the fact table. These queries exhibit strong temporal locality, and in many cases, it is plausible to fit the working set into a cluster’s memory.

Shark allows users to exploit this temporal locality by caching their working set of data, or in database terms, to create in-memory materialized views. Common data types can be cached in a columnar format (as Java primitives arrays), which is very efficient for storage and garbage collection, yet provides maximum performance (orders of magnitude faster than reading data from disk). Below is an example on how to cache data in Shark:

CREATE TABLE logs_last_month_cached AS
SELECT * FROM logs WHERE time > date(...);

SELECT page, count(*) c FROM logs_last_month_cached
GROUP BY page ORDER BY c DESC LIMIT 10;

In addition to caching, Shark employs a number of optimization techniques such as limit push downs and hash-based shuffle, which can provide significant speedups in query processing.

Spark/Machine Learning Integration

Shark provides a simple API for programmers to convert results from SQL queries into a special type of RDDs (Resilient Distributed Datasets). This integrates SQL query processing with machine learning, and provides a unified system for data analysis using both SQL and sophisticated statistical learning functions.

val youngUsers = sql2rdd("SELECT * FROM users WHERE age < 20")
println(youngUsers.count)
val featureMatrix = youngUsers.mapRows(extractFeatures(_))
kmeans(featureMatrix)

Hive Compatibility

List of unsupported Hive features provides list of Hive features that we don't support yet. If you are interested in trying out Shark and use some of the following features, please drop us a note and we will see how we can accommodate.

Hive Test Status: Coming soon with a list of tests passing/failing.

Deployment

EC2 Guide: We will provide a tutorial on setting up a Shark cluster on EC2 soon!

Developer Guide

Developer Guide: For people who are interested in contributing. We don't have much here yet. Will slowly add content to it.

!http://shark.cs.berkeley.edu/google-analytics/ga.php?utmac=MO-31076020-1&utmn=1579418404&utmr=-&guid=ON&utmp=%2Fwiki