Node does not return to a healthy state after the database is temporarily unreachable

[fordN]

Do you want to request a feature or report a bug?
bug

What is the current behavior?
The graph-node is unable to successfully recover on its own from the being down for several minutes for, as an example, unexpected database maintenance.

If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem.
In order to reproduce the issue shutdown the database or otherwise make it unreachable to the graph-node for 10 minutes.
When the database comes back online the graph-node may continues to emit warnings from the ChainHeadUpdateListener (see below), and not receive new blocks. Though it does not seem to happen every time.
The bad state can be resolved by restarting the graph-node.

WARN Error receiving message, error: IO error: communication with the server has desynchronized due to an earlier IO error, channel: chain_head_updates, component: Store > ChainHeadUpdateListener > NotificationListener

What is the expected behavior?
Successfully reestablish a connection with the database when it comes back online.

[Jannis]

As for the hosted service, we'll do the following:

1. Identify whether scaling the db up to multiple zones prevents connection errors like this during/after upgrades.
2. Implement recovering from connection errors in graph-node.
3. Add a test that verifies that this works when e.g. the database is stopped and restarted.
4. Build alerts into graph-node for when there are connection errors.

[iameli]

Just ran into this on Livepeer's graph-node. A side effect, at least in our use case: these messages are extremely verbose, filling up 16gb of Papertrail logs in a day 😃

[lutter]

Looking into this, it would be easy to reconnect to the database after a listener loses the connection but getting disconnected from the database there means that we might have lost notifications, and what we think the world looks like does not match reality. In particular, we might have missed a subgraph assignment event, and therefore might not index something we should be indexing.

I think it would be much more robust and safer to panic if we lose connection in a listener rather than in effect silently dropping notifications; that then would let whatever process supervisor is watching graph-node restart the node, and we already make sure we have the right idea of what to index etc. on restart. It also has the nice effect that the supervisor is in charge of how often to restart, when to give up etc.