make rerun-on-error for short tasks behavior configurable

[jellevandenhooff]

When on the go I'd rather not burn battery rerunning failing tests repeatedly. Also, in an ideal world where I setup my watches correctly, I might not even want any rerun until I fix the problem.

[amonks]

Good idea.

I won't get to this immediately, and I wouldn't recommend submitting a PR until after I close #96 (which has turned into quite a big change), but I agree that this ought to happen.