{6}Restore a node from backup

[benjamin-actyx]

Job story

When:

• An Actyx node has died or broken

I want to:

• Restore it from a backup

So that:

• I can follow my standard admin operating procedure

->
The problem is that peers may have "newer" versions of the restored node’s chain than the node itself.
It would then re-use the same offsets to potentially write different data.
An easy way to work around this is to switch to a new node-id.
A more complicated way would be to try and restore state via the peers, then continue with the old node-id.

Acceptance criteria

• Script or guide or automatic behavior that makes restoring from backups easy and fail-safe

[mcamou]

See https://actyx.mantishub.io/view.php?id=494#c3470

[mcamou]

The issue with a new node-id is that local subscriptions would break (as would happen in https://actyx.mantishub.io/view.php?id=494#c3470). I think the second solution is better.

I think that this would mean that a node should not start emitting events until it receives at least one root map from the swarm. However, this brings up the problem of how a node would know that it's the first node in the swarm.

[rkuhn]

@rklaehn Someone found a way to break your “unbreakable” crypto offset reuse prevention scheme.

Waiting for foreign heartbeats before emitting events sounds reasonable to me, apart from the need of a setting to enable “lonely mode”. If we add that, then previously emitted state can be recovered, including crypto offsets. This is not bullet proof (e.g. against deliberately caused restores by someone who can also control the network), but it would make “restore from backup” quite a bit nicer for the admin.

[rkuhn]

I think we shouldn’t support this feature: when a node fails, make a new node with the same settings but a new keypair. Retrieving and restoring the settings is already possible with ax in a scripted fashion, so maybe the only thing needed is a documentation page explaining that this is the approach.