ingest v2: WAL truncation blocks persist/init_shards, causing ingest timeouts under load

[aanufriev-celonis]

Describe the bug

In ingest v2, the ingester's shard-positions gossip handler holds the ingester's global lock across an entire WAL-truncation pass, which blocks ingestion on that node while the pass runs.

EventSubscriber<ShardPositionsUpdate>::handle_event acquires the full ingester lock once (lock_fully) and holds it while it truncates/deletes every shard in the update.

persist and init_shards take that same lock. So the truncation pass is effectively a stop-the-world for that ingester, each truncate/delete performs WAL disk I/O, so on a node with many shards and/or a WAL on slower storage it can hold the lock for tens of seconds.

During that window:

• incoming persist requests queue behind the lock and (our case) exceed client ingest timeouts;

• a concurrent init_shards, auto-scale-up triggered by the same traffic surge, also blocks and times out, so the source cannot add capacity exactly when it needs to.

Steps to reproduce (if applicable)

This is a contention/timing issue, so it reproduces under load. Conditions that trigger it:

1. Run an ingester (ingest v2) that leads many shards, ideally with the WAL on slower storage.
2. Drive a sustained ingestion surge into a source so the control plane attempts to scale the source up while shard positions are actively gossiped (frequent ShardPositionsUpdate truncations).
3. Observe on that ingester: WAL lock-acquire warnings, ShardPositionsUpdate handler-budget warnings, failed to init shards ... request timed out.

Expected behavior

Background WAL cleanup driven by gossiped shard positions must not block ingestion for the duration of the whole pass. Ingest latency (and init_shards) should not spike to seconds/tens-of-seconds (up to 50 in our case) because a truncation pass is in progress. The cleanup should yield the ingester lock frequently enough that queued persist/init_shards calls can interleave, performing small units of work rather than the entire update.

Configuration:

quickwit:d33dd237f8250899c223bb2cd7997f005ed616fe

[guilload]

The last time I look into those warnings, tracing clearly showed that the issue was actually slow disks: all the time was spent in mrecordlog. RPCs were indeed waiting on the lock but there was no lock contention. I increased the provisioned IOPS and throughput for our EBS and the issue went away on a cluster with hundred of shards.

Before we do anything, we need a better diagnostic via tracing. I'm also not a fond of the fix you suggest. I think we should do batching at another layer. Maybe we could directly add a batch API in chitchat or, in Quickwit, push the updates in a channel and do some short lingering there.

[aanufriev-celonis]

Thanks @guilload, in our case the time is dominated by mrecordlog truncation while the full ingester lock is held, with persist queued behind it. Agree, it is not classic lock contention, and faster disks shorten the hold.

I think you are right, the ingester handler is the wrong layer. Forgive me my lack of knowledge of Quickwit's architecture, I am still exploring it. Considering ShardPositionsService implementation, cluster-originated updates are published one shard per ShardPositionsUpdate, so on a large cluster the ingester takes the full lock once per shard, and my per-event batching does not reduce that. Coalescing at the aggregation layer is the right move.

Suggested direction: move the batching to the aggregation layer instead of the ingester handler. In ShardPositionsService, briefly buffer incoming updates and apply them together: keep the latest position per shard over a small time window, then apply them in a single pass. That collapses the many single-shard gossip events into far fewer lock acquisitions and drops redundant truncations, targeting the mrecordlog-under-lock cost directly.