Store IO throttling throttles far more than asked #6018
I've been digging into the "merges can fall behind" at high indexing
Net/net I think when we ask for 50 MB/sec today we are really
I indexed a bunch of small log-file type docs into 1 shard, 0
With no throttling (index.store.throttle.type=none), the merge takes
With the default 50 MB/sec merge throttling, it takes 72.1 sec, which
So I dug in and discovered one problem, I think caused by the
System.nanoTime is actually quite costly, so I suspect the overhead of
This really improved it: now the merge takes 25.7 sec at 50 MB/sec
I'll also open a Lucene issue to fix this, and make an XRateLimiter
It will solve it in that place: the BufferedChecksumIndexOutput. Note that RateLimitedIndexOutput itself has a similar construction, we should ensure its not doing anything stupid too.
Another similar expensive call is any digester's updateBytes(). We want to avoid calling this native method frequently (e.g. for zero bytes) for the same reasons.