Home

Shark (Hive on Spark)

Shark is a large-scale data warehouse system for Spark designed to be compatible with Apache Hive. It can answer Hive QL queries up to 30 times faster than Hive without modification to the existing data nor queries. Shark supports Hive's query language, metastore, serialization formats, and user-defined functions.

User Documentation

There is a small corpus of helpful documents on Shark. The shark-users mailing list is also very active and will be a helpful resource for beginners.

Running Shark Locally - getting Shark up and running on a single node for a quick spin in ~ 5 mins.

Running Shark on EC2 - getting a Shark cluster up and running on Amazon EC2 in ~ 10 mins, including examples on how to query data in S3.

Running Shark on a Cluster - getting a Shark cluster up and running on your own cluster.

Shark User Guide - an introduction to running Shark and its API.

Compatibility with Apache Hive - for those deploying Shark in existing Hive Warehouses.

Developer Documentation

Developer Guide - for people who are interested in contributing. We don't have much here yet. Will slowly add content to it.

Startup Tasks for New Contributors

Hive Patches - patches we made to Hive.

Related Projects

Spark