Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Data processing on Hadoop without the hassle.

This branch is 32 commits ahead, 666 commits behind nathanmarz:develop

Fetching latest commit…

Cannot retrieve the latest commit at this time

Failed to load latest commit information.
src
test/cascalog
.gitignore
.travis.yml
CHANGELOG.md
LICENSE.txt
README.md
apl.txt
example.job-conf.clj
gendoc.sh
project.clj

README.md

Cascalog

Build Status

Cascalog is a fully-featured data processing and querying library for Clojure. The main use cases for Cascalog are processing "Big Data" on top of Hadoop or doing analysis on your local computer from the Clojure REPL. Cascalog is a replacement for tools like Pig, Hive, and Cascading.

Cascalog operates at a significantly higher level of abstraction than a tool like SQL. More importantly, its tight integration with Clojure gives you the power to use abstraction and composition techniques with your data processing code just like you would with any other code. It's this latter point that sets Cascalog far above any other tool in terms of expressive power.

Follow the getting started steps, check out the tutorial, and you'll be running Cascalog queries on your local computer within 5 minutes.

Getting Started

The best way to get started with Cascalog is experiment with the toy datasets that ship with the project. These datasets are served from memory and can be played with purely from the REPL. Just follow these steps and you'll be on your way:

  1. Install leiningen
  2. Make sure you have Java 1.6 (run java -version)
  3. checkout the Cascalog project using Git
  4. lein deps && lein compile && lein repl
  5. Work through the examples in the introductory tutorials

Using Cascalog within a project

Cascalog is hosted at Clojars. Clojars is a maven repo that's easy to use with maven or leiningen.

To include Cascalog in your leiningen or cake project, add the following to your project.clj:

[cascalog "1.8.6"] ;; under :dependencies
[org.apache.hadoop/hadoop-core "0.20.2-dev"] ;; under :dev-dependencies

Note that Cascalog is compatible with Clojure 1.2.0, 1.2.1 and 1.3.0.

Documentation and Issue Tracker

Come chat with us in the Google group: cascalog-user

Or in the #cascalog or #cascading rooms on freenode!

Priorities for Cascalog development

  1. Replicated and bloom joins
  2. Cross query optimization: push constants and filters down into subqueries when possible

Acknowledgements

YourKit is kindly supporting open source projects with its full-featured Java Profiler. YourKit, LLC is the creator of innovative and intelligent tools for profiling Java and .NET applications. Take a look at YourKit's leading software products: YourKit Java Profiler and YourKit .NET Profiler.

Cascalog is based off of a very early branch of cascading-clojure project (http://github.com/clj-sys/cascading-clojure). Special thanks to Bradford Cross and Mark McGranaghan for their work on that project. Much of that code appears within Cascalog in either its original form or a modified form.

Something went wrong with that request. Please try again.