[Performance] Modify FD flags as data become progressively cold #4465
Description
RocketMQ is designed to be write friendly, using commit log file sequence to store all topics served by the broker. This works well since the majority of the read are served with data from page-cache, thus random read is not involved.
Unfortunately, it is not always the case as more workloads are served. If a few subscribers read data that were kicked out of page-cache, severe read amplification and page-cache pollution are observed, bringing substantial impact to IO bandwidth, resulting in broker busy from time to time.
Page-cache pollution, at present, cannot be fixed unless we turn to Direct IO and manage cache manually, now that flag POSIX_FADV_NOREUSE of posix_madvise in Linux is no-op, which is not in sync with its doc. Read amplification may be mitigated through posix_madvise, yet by way of JNI.
Along the way to identifying the performance issue, I found Lucene/elastic-search developers came across this issue years ago
elastic/elasticsearch#27748
https://blog.mikemccandless.com/2010/06/lucene-and-fadvisemadvise.html