Destructive Chaos Testing: Sandbox healing complex cases

[Bolodya1997]

Overview

Extends #898.

Cover extra heal scenarios with sandbox tests:

1 death cases:

1. Local NSMgr:
   • restored:
     1. The same Forwarder selected.
     2. Another Forwarder selected.
2. Local Endpoint:
   • died:
     1. A new local Endpoint selected.
     2. A new remote Endpoint selected.
3. Remote NSMgr:
   • died:
     1. A new local Endpoint selected.
     2. A new remote Endpoint selected.
4. Remote Endpoint:
   • died:
     1. A new local Endpoint selected.
     2. A new remote Endpoint selected.
5. Registry:
   • restored

2 death cases:

1. Local NSMgr(r) +
   1. Local Endpoint(d)
   2. Remote NSMgr(d)
   3. Remote Forwarder(d)
   4. Remote Endpoint(d)
   5. Registry(r)

Should be tested (for all tests including simple cases):

1. Shutdown becomes when hop:
   1. doesn't pass any Request/Close.
   2. passes Request.
   3. passes Close.
2. If case includes selecting another Forwarder/Endpoint, during the healing should be selected:
   1. the same mechanism.
   2. another mechanism.

Blockers

1. simple cases task - Destructive Chaos Testing: Sandbox healing simple cases #898
2. clock in sandbox - [sdk#764] Rework sandbox to use clock.Clock #785

[Bolodya1997]

@edwarnicke
Do we want to skip some steps now and return to them later? Or probably there is something else I am missing?
Please share your thoughts.