TempSource: dumping data into a location to persist between Jobs

[johnynek]

In page rank, we run a job, write out a value, and before we start the next Job/Flow, we read the value and see if we should stop.

Currently, our implementation is ugly and requires the user to control that. What would be better is something like:

  val error = Temp() // can only write ONCE in the set of flows.
 // some place later in the same "family" of jobs:
  val result = error.read

So we need to keep a map of all the open temp files, which jobs write into them (we want write-once semantics) and then clean them up just before the Tool exits.

[johnynek]

See the code in TypedPipe.forceToDiskExecution: https://github.com/twitter/scalding/blob/0.13.1/scalding-core/src/main/scala/com/twitter/scalding/typed/TypedPipe.scala#L470

This should be refactored to make a temporary source. Ideally, we could delete these all when the entire execution is complete, something we don't do now.