[BUG] - Incremental ledger snapshot calculations are causing missed slot leader checks and block propagation delays

[orpheus-antpool]

Internal/External
External

Area
Other: Stake Pool Operation

Summary
With increasing number of transactions, the incremental ledger snapshot calculations are becoming more complex and lead to missed slot leader checks and block propagation delays. This is independent of the major garbage collections which they trigger due to high memory usage and which also contribute to missed slot leader checks and block propagation delays. A particularly large difference is observable when comparing epoch 303 and epoch 304.

Expected behavior
Incremental ledger snapshot calculations should be performed in the background in a separate thread (to avoid blocking for longer than 1s) and at finer granularity (to avoid sudden memory spikes that trigger a chain of major garbage collections). If this is already running in a forkIO, then perhaps a foreign unsafe call is blocking something (in which case it would make sense to force it in a separate bound thread). Continuations could be a solution for finer grain calculations.

[Jimbo4350]

Closing this. If this is still relevant please reopen.