fix(sequencer-relayer): avoid hanging while waiting for submitter task to return #1206
Conversation
github-actions
bot
added
the
sequencer-relayer
| info!("received shutdown signal"); | ||
| break Ok("shutdown signal received"); | ||
| } | ||
|
|
||
| _ = &mut submitter_task => break Err(eyre!("Celestia submission task returned")), |
There was a problem hiding this comment.
This is an unfortunate side-effect of changing to use child cancellation tokens - the submitter task (child) is no longer able to cancel the relayer task (parent). Since the submitter task can exit with error during init, we need to poll for that here or else we hang in this select! loop.
This also means we need to fuse the submitter task and outside the loop we need to check whether it's already terminated or not before awaiting it, or else we'd hang there.
There are other potential approaches to address that, but I couldn't see anything cleaner than this (except for my still-preferred approach of a single cancellation token for all tasks rather than this hierarchy).
| @@ -607,6 +608,7 @@ impl TestSequencerRelayer { | |||
| .and_then(|value: serde_json::Value| serde_json::from_value::<ZPage>(value).ok()) | |||
| .map_or("unknown".to_string(), |zpage| zpage.status); | |||
|
|
|||
| error!("timed out; context: `{context}`, state: `{state}`, healthz: `{healthz}`"); | |||
There was a problem hiding this comment.
This helps diagnose the point at which the test fails, since we can get several further log lines after this point before the panic is printed.
Summary
This fixes a bug whereby the relayer hangs during shutdown due to the submitter task not being instructed to shut down.
Background
Bugfix.
Changes
SequencerRelayer::newconstructs aShutdownHandlewhich wraps the top-level cancellation token. This handle is held as a member variable ofSequencerRelayerand is used to cancel all relayer child tasks if the API task exits. A clone of theShutdownHandleis also held inmainand is used to cancel all tasks if aSIGTERMis received.ShutdownHandle's token is provided to the relayer task, namedrelayer_shutdown_token. It is never actively cancelled - only ever via the parent token.relayer_shutdown_tokenis provided to the submitter task and also held as a member ofRelayer. It is actively cancelled after the relayer's mainselect!loop has exited if the submitter task is still running.Testing
Manual test only. I could see no way to artificially cause the relayer's
select!loop to exit. We could potentially try dropping in some#[cfg(test)]gated code in the loop to force this, but that would also mean we couldn't use the blackbox tests for it.Related Issues
Closes #1095.