Restart app

[ebuchman]

It would be nice to recover from an app restart without having to restart tendermint - p2p reactors not dependent on the app could thus keep running. Implementing this requires changes in tmsp, consensus, and mempool:

• tmsp client:
  • when a request fails, flush the queue, close the connection, and drop back into the connect loop to attempt reconnect
  • expose a channel to alert consumer when we have reconnected
• consensus:
  • when execBlock fails due to TMSP request failure, stop the consensus state
  • wait for the reconnection and restart consensus state
• mempool:
  • stop caching txs while the app is disconnected or anti-replay will cause problems
  • eventually, buffer txs received while the app is off and replay when it comes on (buffer mempool txs for replay if app restarts #256)

If we can only stop the consensus state and mempool (and not the reactors) then we don't have to worry about re-adding peers (eg. looping over Switch.Peers and running AddPeer). However, some peer messages wont be able to be processed and may queue up.

Related

• Ability to replay txs from LevelDB on startup #227
• TMSP handshake message #211
• Better err/warn output upon app connection failure #199

[ebuchman]

Ok, from the top.

The goal of allowing the app to restart, vs. just restarting Tendermint, is two-fold:

• we don't lose any data (consensus votes, block parts, mempool txs)
• we can continue p2p gossip - including receiving new txs (that we can't process yet) and doing consensus on a height (we can do everything except run the block after we've seen the consensus).

The former we can solve with WALs - we already have one for consensus that saves everything we've pulled off the peerMsgQueue (so not everything received by the ConsensusReactor). We could move the WAL to the consensus reactor, or add a second one for peerState reactor msgs (ie. so we can debug gossipData and gossipVotes routines). Also, we'd need a WAL for the mempool, which will be more involved because we have to detect what's been committed or invalidated, which probably requires tx indexing or writing lots more to the file. All of this we should probably do anyways.

Using WALs, we could let the reactors quit, and let the Node package handle restarting the various parts, which should be simple since it already does that sequence on startup. We can remain connected to our peers, and add them to the new reactors when they start again (after we reconnect to the app), but then we wouldn't be able to gossip or receive new msgs while the app is down.

An alternative would be to let the reactors manage the reconnects using a GetProxyApp closure. This would require carefully managing dependencies between reactors (eg consensus depends on mempool) and buffering new msgs that we can't process until the app comes back, but it would allow gossip to continue unfettered in the background, and let us catchup quicker when the app restarts, making for a more robust gossip layer independent of application failures and upgrades. See https://github.com/tendermint/tendermint/tree/restart_squash for a start.

These are in contrast to a solution using OnReset, to have the tmsp client and consensus state restart themselves, which seems to be more confusing/difficult. See https://github.com/tendermint/tendermint/tree/restart_app

[jaekwon]

Not sure if we would want a WAL on the mempool... the logic is fairly simple, not sure what the benefit would be.
Moving the WAL to the consensus reactor seems problematic because the reactor isn't deterministic. And I'm not sure that we would want it to be deterministic. I think maybe it's fine to not have a WAL there, and test assertions about general data/vote syncing properties some otherhow. If we wanted to write any formally verifiable proofs, I would think that we can bake these data/vote syncing properties in... like, assume that if peer1 is in height h-1, and we're on height h, that peer1 will eventually receive a commit for h. Seems like anything more is overkill... well who knows...

The restart_squash approach seems great, but how is this implemented without a way to tell the app to load the state after block H? I'd think we need a new TMSP message for that... Curious, I'll take a look now.

[ebuchman]

the point of the WAL on the mempool is so we don't ever lose a tx thats been accepted into it (this is mostly for clients sake, could just have the rpc handle it instead).

I agree about not really needing the WAL on the reactor. Its not deterministic (unless we make a "deterministic reactor" module as part of the switch, and we could make a WAL an extra feature on that), but it could be useful for debugging.

As for loading the state, we need a handshake, and I'm trying to do the restart in a way that is forward compatible with some handshake strategy. But maybe we should spec the handshake first?

[ebuchman]

Plan:

• tmsp/client is a dumb tmsp client
• tendermint/proxy is a smart one that gives services a persistent AppConn which will handle reconnection internally; to the service a reconnection appears as latency
• reconnecting needs to be co-ordinated across consensus and mempool connections by a multiAppConn (handles handshakes and replay)
• if an AppConn's internal connection (its tmsp.Client) fails:
  • it notifies the multiAppConn
  • the multiAppConn stops both tmsp clients and creates new ones
  • upon reconnection, the multiAppConn handshakes on the consensus connection to determine the latest block height the app has processed.
  • all more recent blocks are replayed on the consensus connection
  • all current mempool transactions are replayed on the mempool connection
  • for each AppConn that noticed the broken connection, retry the failed tmsp client call
  • resume normal operation
• Note: we can use AppConnConsensus, AppConnMempool, AppConnQuery, etc as typed subsets of tmsp/client, meaning less code duplication, and conn capabilities enforced at type level.

[jaekwon]

I like this plan. Not sure how the last bit results in less code duplication though.

[ebuchman]

proxyapps don't need to implement the whole tmsp.Client

[ebuchman]

This ended up being more difficult than first anticipated. We should take another shot now that the handshake is more stable

[oaxjogi]

Any news? Would be great to make tendermint great again :-)

[ebuchman]

Oh! Is tendermint currently not great because of this feature?

It's a bit involved and requires some significant refactoring of various pieces to do it right, so it has not been priority for a while, especially since it's easy enough to just restart the node. Do you have some really pressing use-case?

[melekes]

The alternative solution could be calling panic and using Linux process supervisors like http://smarden.org/runit/ to restart tendermint. The downside is obviously it requires some devops work plus restart time. The advantage is once it's setup, process supervisor will ensure Tendermint is always running (despite possible errors)

[ebuchman]

Yeh good point, though I don't see the stack trace being useful here, so maybe we should just os.Exit ?

[oaxjogi]

tendermint is great, no question. However, if a node adapter is disconnected it would be great to get back to init mode and wait a time for a re-connection - otherwise it is hard for applications to handle that situations.

[melekes]

maybe we should just os.Exit ?

yes

[ebuchman]

Only thing is we can't clean up - would be nice if we could shut down the node everywhere rather than just fail. Can we send a signal to ourselves?

[melekes]

Can we send a signal to ourselves?

using syscall.Kill? yeah

[ebuchman]

Note that the resolution here was for tendermint to shutdown when the ABCI goes down. The OS can handle restarting both.

[ttmc]

Note that the resolution here was for tendermint to shutdown when the ABCI goes down. The OS can handle restarting both.

That should really be documented somewhere, e.g. in notes on how to run a production system.

[melekes]

@ttmc want to create a PR?

[ebuchman]

See #1494