ACCUMULO-4060 Run a timer task to restart failed FATE repo runner thr…

[joshelser]

…eads.

If ZK becomes unavailable for some period of time, it's possible that the
FATE repo runner threads inside of the master will terminate without
the master itself dying.

An attempt at an implementation to "recover gracefully" when the repo-runner threads die.

[keith-turner]

Seems like it would be simpler to modify transaction runner and add a try/catch/log just inside the while loop.

[joshelser]

Seems like it would be simpler to modify transaction runner and add a try/catch/log just inside the while loop.

We very well could do this as well. I was hoping to pick your brain on any worries in just eating those exceptions. I suppose in the end it's no different.

[keith-turner]

eating those exceptions

don't just eat them, log them.

I suppose in the end it's no different.

Yeah I can't see any differences between the approaches.

[joshelser]

Ok, thanks Keith. The only benefit I can see with what I did here is that it supports growing the number of repo runners. That's probably better done as a feature (with shrinking the number of them too), than as a bugfix.