scorch rollback semantics

[mschoch]

We have identified some issues with scorch rollback. Some conversation in #754

I'm not sure I have it yet, but I think I have identified one issue. The introduction of segments and the introduction of rollbacks both come in through the introducer. If you remember, prior to the introduction of segments, some optimistic work is done. Then during introduction we check that optimistic work and possibly do some fixup. But, I think if a rollback is applied, the fixup phase for the optimistic work isn't enough (ie, the optimistic work is wrong, not just incomplete). A similar problem may face the merger (though I haven't thought about it yet).

It almost seems like if we decide to rollback, it should flush the pipeline of not yet applied things (returning error in the case of batches, and just dropping the merges).

And additional thoughts from Steve that Rollback shouldn't honor unsafe_batch flag, and instead always do the right thing.

Additional discussion opened up possibility that Rollback could behave like Close(). So, it would first shut down all background goroutines, be in some highly deterministic state, then reset the state so that upon the next open, it's in the newly desired state, then shut down. This is just an idea at this point.