validator: disable API while recovery stream is active

[sergerad]

Summary

Adds a RWLock to disable Validator RPCs when backup is occurring (and vice versa).

Status endpoint can still be used during backup.

Replaces #2283.

Changelog

[[entry]]
scope  = "validator"
impact = "changed"
description = "Disable API while recovery stream is active"

[Mirko-von-Leipzig]

Thank you!

[Mirko-von-Leipzig]

I think perhaps recovery is more accurate than backup for what's happening

[kkovaacs]

One potential issue is that since we allow subscribing with a future block (> current tip) and that subscription also takes the lock we're effectively deadlocked.

Should we maybe add a check that the subscription start block should be <= the current tip?

[Mirko-von-Leipzig]

One potential issue is that since we allow subscribing with a future block (> current tip) and that subscription also takes the lock we're effectively deadlocked.

Should we maybe add a check that the subscription start block should be <= the current tip?

Yeah I guess in a way this isn't really the same situation as the subscription streams we have in conventional RPC.