Medalla Madness

[ajsutton]

Description

Parent issue to track the work required to handle the non-finalisation on Medalla.
Quick notes may be made inline but should be moved to their own issue once we start to flesh them out so we have a clear record of what was wrong and what we did to fix that specific issue.

CPU and Memory Consumption

• Add state regeneration queue to limit concurrent regenerations #2589 An excessive number of state regenerations were in progress concurrently resulting in excessive CPU and memory usage
• Process gossip off of the netty thread #2583 Netty threads were blocked processing gossip messages. Moved to thread pool.
• P2P Executor Queue Filling Up #2598 P2P executor queue is filling up
• Use cache to get block and state #2610 Same state is being regenerated multiple times
• Use cache to get block and state #2610 Significant number of state regenerations that replay 1 block. Why are these not hitting cache?
• Persist hot states periodically #2608 Persist hot states periodically to reduce the number of blocks that need to be replayed when regenerating states that have dropped out of the in-memory cache
• Don't reprocess blocks at startup #2615 Skip processing blocks at startup if possible
• Optimize checkpoint processing #2609 Optimize checkpoint state generation
• State Cache Size Exceeding Limit #2612 State cache exceeding maximum size

Fork choice

• ProtoNode: Delta to be subtracted is greater than node weight. #2562 Error in fork choice. Was happening prior to Medalla hitting issues but seems much more common now.
• Use cache to get block and state #2610 Many fork choice updates are being skipped with message "Skipping update of chain head" because of interim updates.
• Teku following incorrect chain #2614 Reports that the Teku instances that are in sync are following a different chain to what should be canonical head

Sync Issues

• Be more lenient before deciding a peer is excessively throttling requests #2590 Excessive throttling checks were kicking in too often because of all the empty slots on Medalla
• Check latest peer status while syncing #2586 Check latest peer status while syncing
• find a common ancestor before initiating a sync. #2600 Each time we select a new peer to sync from we start from the finalised checkpoint. We should find a more recent common ancestor if the finalised checkpoint is significantly in the past

Other

• Avoid accessing store each time metrics are retrieved #2584 Metrics were creating high contention on the Store locks
• Import/Export Validator Signing Records #2601 Support import/export of slashing protection databases to make changing clients easier.
• Don't block while regenerating states to archive during finalization #2616 Archive mode nodes will pause for a long period when finalisation is reached while they regenerate the finalised states to store.
• deadlock when updating finalized checkpoint #2625 deadlock when updating finalized checkpoint

[ajsutton]

We're running smoothly again on Medalla. There's still #2601 and #2616 to complete as follow up but will close this tracking ticket to keep things tidy. Those individual tickets will remain open and get done when we can.