Cascalog Testing 2.0 gives a long discussion on various midje-cascalog idioms.
To use midje-cascalog in your own project, add the following two entries to
:dev-dependencies inside of your
[lein-midje "3.0.1"] [cascalog/midje-cascalog "1.10.1"]
Midje-Cascalog supports Clojure 1.3+ and Cascalog 1.8+. Add
(:use [midje sweet cascalog]) to your testing namespace to get started.
When you're all finished writing tests,
lein midje at the command line will run all Midje tests and generate a summary.
Let's say you want to test a Cascalog workflow that examines your user datastore and returns the user with the greatest number of followers. Your workflow's top level query will generate a single tuple containing that user's name and follower-count. Here's the code:
(defn max-followers-query [datastore-path] (let [src (name-vars (complex-subquery datastore-path) ["?user" "?follower-count"])] (cascalog.ops/first-n src 1 :sort ["?follower-count"] :reverse true)))
max-followers-query is a function that returns a Cascalog subquery. It works like this:
- The function accepts a path, (
datastore-path) and passes it into a function called
complex-subqueryreturns a subquery that generates 2-tuples; this subquery is passed into
name-varsbinds this subquery to
srcafter naming its output variables
first-nreturns a subquery that
- sorts tuples from
srcin reverse order by follower count, and
- returns a single 2-tuple with the name and follower-count of our most popular user.
- sorts tuples from
At a high level, the subquery returned by =max-followers-query= is responsible for a single piece of application logic:
- extracting the tuple with max
?follower-countfrom the tuples returned by
A correct test of
max-followers-query will test this piece of logic in isolation.
(fact "Query should return a single tuple containing [most-popular-user, follower-count]." (max-followers-query :path) => (produces [["richhickey" 2961]]) (provided (complex-subquery :path) => [["sritchie09" 180] ["richhickey" 2961]]))
Midje circumvents all extra complexity by mocking out the result of
(complex-subquery datastore-path) and forcing it to return a specific Clojure sequence of
[?user ?follower-count] tuples.
produces checks result from queries. The fact passes if these statements are true and fails otherwise. The above fact states that
max-followers-queryis called with the argument
- it will produce
[[ richhickey" 2961]],
[["sritchie09" 180] ["richhickey" 2961]].
Fact-based testing separates application logic from the way data is stored. By mocking out
complex-subquery, our fact tests
max-followers-query in isolation and proves it correct for all expected inputs.
This approach is not just better than the "state of the art" of MapReduce testing, as defined by Cloudera; it completely obliterates the old way of thinking, and makes it possible to build very complex workflows with a minimum of uncertainty.
Fact-based tests are the building blocks of rock-solid production workflows.