Skip to content

Fix RoutineBook.Acquire deadlock causing missing block rewards#233

Merged
leobago merged 1 commit into
masterfrom
dev
Mar 3, 2026
Merged

Fix RoutineBook.Acquire deadlock causing missing block rewards#233
leobago merged 1 commit into
masterfrom
dev

Conversation

@Zyra-V21
Copy link
Copy Markdown
Collaborator

Summary

  • RoutineBook.Acquire() returned without acquiring a semaphore slot after a 1-minute timeout, permanently leaking tokens until the pipeline deadlocked (all 32 slots lost). Now loops until a slot is acquired.
  • WaitForPrevState() blocked forever when epoch metrics were disabled, waiting for beacon states that were never downloaded. Added early return when epoch metrics are off.

Both bugs caused 2,383 slots with zeroed reward fields (f_cl_api_reward, f_reward_fees) in t_block_rewards on both testing and production databases (slots 13,337,408 – 13,597,822).

Closes #232

Test plan

  • Compiled patched binary and deployed to eth-archive
  • Reprocessing job running successfully on affected slot range (previously deadlocked at slot ~13,337,440)
  • Verified blocks download past the former deadlock point
  • Verified block rewards are being calculated (no "gas fees not calculated" warnings after initial epochs)

Acquire() used a select with a 1-minute timeout that returned without
acquiring a semaphore slot. Over time all tokens were lost, freezing
the processing pipeline. Now loops until a slot is acquired.

WaitForPrevState() blocked forever when epoch metrics were disabled
because it waited for states that were never downloaded. Added an
early return when epoch metrics are off.

Closes #232
Copy link
Copy Markdown
Member

@leobago leobago left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@leobago leobago merged commit f2326f2 into master Mar 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RoutineBook.Acquire() deadlock causes missing block reward data

2 participants