view: auto-restart view worker on error

[hdevalence]

Currently, any transient error in the view worker permanently stops the server forever. Instead, restart after a delay.

[conorsch]

Refs #3732

[cronokirby]

We should revisit the use of error_slot and how the status of the view service is reported in other methods, otherwise I anticipate confusion when an upstream caller looks at the status during the sleep. Right now, the various methods for implementing ViewProtocolService will check the error slot, and fail if it's currently set, and so during the sleep period all these methods will fail.

In general, what should we do during the phase where the ViewService is restarting? Should the various service methods fail? Are callers supposed to know that they'll work soon? Should we have the methods block? Should we have them return stale answers?

[hdevalence]

Yeah, agreed that those are all tricky things. I think what we want to do is have a health check method that allows the caller to see if the service is healthy or not. There's one example of how to do an "internal" health check method in Hermes, though we might want something slightly different.

In the meantime I think we should merge this and track that separately.