Skip to content

feat(block-producer): detect and abort on chain desync#1520

Merged
Mirko-von-Leipzig merged 3 commits intonextfrom
mirko/desync-detection
Jan 16, 2026
Merged

feat(block-producer): detect and abort on chain desync#1520
Mirko-von-Leipzig merged 3 commits intonextfrom
mirko/desync-detection

Conversation

@Mirko-von-Leipzig
Copy link
Copy Markdown
Collaborator

This detects situations where the block-producer disagrees with the store on the current chain tip. This should only be possible under rare circumstances, but one possibility is:

  • block-producer submits block N+1 to store via gRPC
  • store processes and persists block N+1
  • block-producer cancels its gRPC request e.g. due to timeout
  • block-producer rolls back local state to N
  • block production spins endlessly as block N+1 is now rejected by store

There are multiple points to detect this. This code determines by comparing the block number being built against the input's header returned from the store. The block builder task panics, which should abort the block-producer process.

Note that this error isn't really recoverable; and requires a full reset of the block-producer. Aborting and restarting is the easiest way to implement this.

Closes #1513.

@Mirko-von-Leipzig Mirko-von-Leipzig marked this pull request as ready for review January 15, 2026 15:19
Copy link
Copy Markdown
Contributor

@bobbinth bobbinth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Thank you! I left a couple of small comments inline.

@Mirko-von-Leipzig Mirko-von-Leipzig merged commit ec27b95 into next Jan 16, 2026
7 checks passed
@Mirko-von-Leipzig Mirko-von-Leipzig deleted the mirko/desync-detection branch January 16, 2026 08:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Block producer desync recovery

2 participants