Query latency impact from linux disk read ahead option

[jasperjiaguo]

Recently we have discovered that Pinot query latency can be impact by the value of Linux's read_ahead_kb. Specifically we saw a very high page fault count and severe tail latency shootup when the read_ahead_kb was set to a larger value in certain Linux distributions. As read_ahead_kb controls the read ahead during the access of mmap files, we think larger read ahead value harms the queries having more random data access patterns. Theoretically it might benefit the opposite pattern but we have yet to see such a case. I think there are a few things that might worth doing:

• In the short term we add this as a tip for Pinot admin in the OSS doc, so that it becomes a public knowledge
• In the long term we may explore controlling this programmatically (like madvise in c), but it might be harder to do in Java
• Revisit the mmap based segment cache

Similar issues/analysis:
https://smalldatum.blogspot.com/2014/05/the-impact-of-read-ahead-and-read-size.html
elastic/elasticsearch#27748