Better support for Iterative Jobs / Temporary outputs

[johnynek]

If you want to do an iterative job, the pattern is:

Read some previous state, compute some new state, check if you should stop, if you should, next returns None else next returns Some(job) where job is probably a copy of the current job.

The problem is the programmer has to manage all the temporary sources that are created to check convergence.

I imagine something like a TempSourceFactory that can keep track of temporary sources, which are probably cascading sequence files, which you can pass between jobs.

After the last job is run, the TempSourceFactory cleans up all the allocated data on the disk (which is to say, all of this data is ephemeral).

My design thinking around this is to have a map inside the TempSourceFactory object that maps a UUID onto a Map[String,Source]. For each iterative job, there is one UUID, and this can be accessed from any job.

This should probably be plumbed through with an API on TempSourceFactory, something like:

object TempSourceFactory {
  def apply[T](name: String)(implicit mf: Manifest[T], args: Args): Mappable[T]
  def cleanup: Unit
}