Thanks for using Cascading.
This repo branch is an early release WIP for Cascading 3.0.
3.0 includes a few major changes and additions:
- Complete re-write of the platform query planner and improvements to the planner API
- Addition of Apache Tez as a runtime platform
- Changes to the Tap/Scheme generic type signatures to support portability
These changes hope to simplify the creation of new bindings to new platform implementations and to improve the performance of resulting applications.
On the last point, we ask that users please spend time running their existing applications against 3.0 WIP releases and to open issues along with tests via pull requests against https://github.com/cwensel/cascading.
See below for more info on submitting issues.
For project documentation and community support, visit: cascading.org
To download a pre-built distribution, visit http://cascading.org/downloads/, or use Maven (described below).
The project includes six Cascading jar files:
cascading-core-x.y.z.jar- all Cascading Core class files
cascading-xml-x.y.z.jar- all Cascading XML operations class files
cascading-local-x.y.z.jar- all Cascading Local in-memory mode class files
cascading-hadoop-x.y.z.jar- all Cascading Hadoop 1.x MapReduce mode class files
cascading-hadoop2-mr1-x.y.z.jar- all Cascading Hadoop 2.x MapReduce mode class files
cascading-hadoop2-tez-x.y.z.jar- all Cascading Hadoop 2.x Tez mode class files
These class jars, along with, tests, source and javadoc jars, are all available via the Conjars.org Maven repository.
Hadoop 1.x mode is where the Cascading application should run on a Hadoop MapReduce cluster. Hadoop 2.x MR1 is the same but for Hadoop 2.x releases.
Hadoop 2.x Tez mode is where the Cascading application should run on an Apache Tez DAG cluster.
Local mode is where the Cascading application will run locally in memory without any Hadoop dependencies or cluster distribution. This implementation has minimal to no robustness in low memory situations, by design.
As of Cascading 3.x, all above jar files are built against Java 1.7. Prior versions of Cascading are built against Java 1.6.
Extensions, the SDK, and DSLs
There are a number of projects based on Cascading available. Visit the Cascading Extensions page for a current list.
Or download the Cascading SDK which includes pre-built binaries.
Of note are two top level projects:
- Lingual - ANSI SQL and JDBC with Cascading
- Pattern - Machine Learning and PMML support with Cascading
And new languages:
Cascading stable releases are always of the form
z is the current maintenance release.
x.y.z releases are maintenance releases. No public incompatible API changes will be made, but in an effort to fix bugs,
remediation may entail throwing new Exceptions.
x.y releases are minor releases. New features are added. No public incompatible API changes will be made on the
core processing APIs (Pipes, Functions, etc), but in an effort to resolve inconsistencies, minor semantic changes may be
It is important to note that we do reserve to make breaking changes to the new query planner API through the 3.x releases. This allows us to respond to bugs and performance issues without issuing new major releases. Cascading 4.0 will keep the public query planner APIs stable.
The source and tags for all stable releases can be found here: https://github.com/Cascading/cascading
WIP (work in progress) releases are fully tested builds of code not yet deemed fully stable. On every build by our continuous integration servers, the WIP build number is increased. Successful builds are then tagged and published.
The WIP releases are always of the form
x.y.z will be the next stable release version the WIP
releases are leading up to.
n is the current successfully tested build.
The source, working branches, and tags for all WIP releases can be found here: https://github.com/cwensel/cascading
Or downloaded from here: http://cascading.org/wip/
When a WIP is deemed stable and ready for production use, it will be published as a
x.y.z release, and made
available as a stable release from the cascading.org download page.
Writing and Running Tests
Comprehensive tests should be written against the
When running tests built against the PlatformTestCase, the local cluster can be disabled (if enabled by the test) by setting:
From Gradle, to run a single test case:
> gradle :cascading-hadoop2-mr1:platformTest --tests=*.FieldedPipesPlatformTest -i
or a single test method:
> gradle :cascading-hadoop2-mr1:platformTest --tests=*.FieldedPipesPlatformTest.testNoGroup -i
Debugging the 3.x Planner
The new 3.0 planner has a much improved debugging framework.
When running tests, set the following
If you are on Mac OS X and have installed GraphViz, dot files can be converted to pdf on the fly. To enable, set:
Optionally, for stand alone applications, statistics and tracing can be enabled selectively with the following properties:
cascading.planner.stats.path- outputs detailed statistics on time spent by the planner
cascading.planner.plan.path- basic planner information
cascading.planner.plan.transforms.path- detailed information for each rule
To report an issue, first ping the Cascading User mailing list with any questions.
If unresolved, look for a comparable test in the
cascading-platform sub-project, reproduce your issue, then issue
a pull request with the failing test added to one of the existing suites.
If you wish to allow your test code to be added to the Cascading test suite, please sign and return this contributor agreement.
Using with Maven/Ivy
It is strongly recommended developers pull Cascading from our Maven compatible jar repository Conjars.org.
You can find the latest public and WIP (work in progress) releases here:
When creating tests, make sure to add any of the relevant above dependencies to your
test scope or equivalent
configuration along with the
cascading-platform compile dependency has no classes, you must pull the tests dependency with the
See http://cascading.org/downloads/#maven for example Maven pom dependency settings.
Source and Javadoc artifacts (using the appropriate classifier) are also available through Conjars.
cascading-hadoop2-mr1 has a
provided dependency on the Hadoop jars so that it
won't get sucked into any application packaging as a dependency, typically.
Building and IDE Integration
For most cases, building Cascading is unnecessary as it has been pre-built, tested, and published to our Maven repository (above).
To build Cascading, run the following in the shell:
> git clone https://github.com/cascading/cascading.git > cd cascading > gradle build
Cascading requires Gradle 1.12 and Java 1.7 to build.
To use an IDE like IntelliJ, run the following to create IntelliJ project files:
> gradle idea
Similarly for Eclipse:
> gradle eclipse
Using with Apache Hadoop
First confirm you are using a supported version of Apache Hadoop by checking the Compatibility page.
To use Cascading with Hadoop, we suggest stuffing
jar files and all third-party libs into the
lib folder of your job jar and executing your job via
$HADOOP_HOME/bin/hadoop jar your.jar <your args>.
For example, your job jar would look like this (via:
jar -t your.jar)
/<all your class and resource files> /lib/cascading-core-x.y.z.jar /lib/cascading-hadoop-x.y.z.jar /lib/cascading-xml-x.y.z.jar /lib/<cascading third-party jar files>
Hadoop will unpack the jar locally and remotely (in the cluster) and add any libraries in
lib to the classpath. This
is a feature specific to Hadoop.