Join GitHub today
GitHub is home to over 36 million developers working together to host and review code, manage projects, and build software together.Sign up
Add regular file flushes whilst writing out scavenged chunk #1918
After our scavenge performance improvements we've been experiencing intermittent commit timeouts and general spikes in write time during a scavenge. This is caused by the large file write direct from memory at the end of a chunk scavenge or chunks merge.
The file is written quickly and then flushed/closed resulting in up to 256MB being put on the disk write queue. We can see in our monitoring that the disk transfer latency spikes resulting in the slow writes for new events.
This PR adds a regular flush to the scavenge and merge process so that the individual flushes are smaller. Looking at stats when testing locally this results in avg write size of ~60KB instead of 1MB.
I've run a custom build with this change (plus #1914) and run a full scavenge on one of our large databases. There was still occasional impact, but there were no commit timeouts and write times were much more stable.
I played with adding a Thread.Sleep after each flush to throttle the write even more. This gave good results locally, but added many seconds to the chunk write. I left it out as I'm not sure it's necessary.
I'm sure there's a similar issue with the index scavenge and merge processes. I haven't touched them here.
No the possibility of delays from a source code perspective. There are a few such operations that could possibly want a delay/priority added. Scavenge, Index Merge, Checkpoints, Version Upgrade etc.
It is also that we don't really know enough such as if these things are actually on shared/remote disks etc and what might possibly be conflicting in terms of iops. In other words we might want to centralize the decision so we know what the index is doing when we are deciding what a chunk should do. Does that make more sense? There has been some work towards this already ... think IO scheduling.