JobTest + specs appears to run the jobs many times.

[johnynek]

It has seemed to be the case in the past to me, but I never investigated carefully, but I think JobTest may run the job for each of the assertions.

Look at some of the jobs that intentionally throw exceptions to see how they are handled. Note how often they occur.

If we get to the bottom of this, if it is true, it might significantly speed up the tests.

[johnynek]

Examples like below create multiple run. For each in "case" in the entire job is run.

We should either structure tests to avoid this, or figure out a way to get JobTest to cache the previous results.

class MRMTest extends Specification {
  noDetailedDiffs() //Fixes an issue with scala 2.9
  "A MRMJob" should {
    val input = List((0, 1), (0, 2), (1, 3), (1, 1))

    JobTest("com.twitter.scalding.MRMJob")
      .source(Tsv("input"), input)
      .sink[(Int, Int)](Tsv("outputXor")) { outBuf =>
        "use reduce to compute xor" in {
          outBuf.toList.sorted must be_==(List((0, 3), (1, 2)))
        }
      }
      .sink[(Int, Int)](Tsv("outputSet")) { outBuf =>
        "use mapReduceMap to round-trip input" in {
          outBuf.toList.sorted must be_==(input.sorted)
        }
      }
      .sink[Int](Tsv("outputSetTo")) { outBuf =>
        "use flattenTo" in {
          outBuf.toList.sorted must be_==(input.map { _._2 }.sorted)
        }
      }
      .run
      .finish
  }
}

[etorreborre]

One reason for starting specs2 was that specs1 execution model was too convoluted. And in the case of JobTest, there is a bit of specification creation embedded in another object doing the execution. One thing I would try is to flatten the spec to just

class MRMTest extends Specification {
  noDetailedDiffs() //Fixes an issue with scala 2.9
  val input = List((0, 1), (0, 2), (1, 3), (1, 1))

    JobTest("com.twitter.scalding.MRMJob")
      .source(Tsv("input"), input)
      .sink[(Int, Int)](Tsv("outputXor")) { outBuf =>
        "A MRJob should use reduce to compute xor" in {
          outBuf.toList.sorted must be_==(List((0, 3), (1, 2)))
        }
      }
      .sink[(Int, Int)](Tsv("outputSet")) { outBuf =>
        "A MRJob should use mapReduceMap to round-trip input" in {
          outBuf.toList.sorted must be_==(input.sorted)
        }
      }
      .sink[Int](Tsv("outputSetTo")) { outBuf =>
        "A MRJob should use flattenTo" in {
          outBuf.toList.sorted must be_==(input.map { _._2 }.sorted)
        }
      }
      .run
      .finish
}

This might fix the execution if should happens to execute things twice.