runtime: MADV_HUGEPAGE causes stalls when allocating memory #61718
Description
Environment: linux/amd64
I've bisected stalls in one of my applications to 8fa9e3b — after discussion with @mknyszek, the stalls seem to be caused by Linux directly reclaiming pages, and taking significant time to do so (100+ ms in my case.)
The direct reclaiming is caused by the combination of Go setting memory as MADV_HUGEPAGE and Transparent Huge Pages being configured as such on my system (which AFAICT is a NixOS default; I don't recall changing this:)
$ cat /sys/kernel/mm/transparent_hugepage/enabled
always [madvise] never
$ cat /sys/kernel/mm/transparent_hugepage/defrag
always defer defer+madvise [madvise] never
In particular, the madvise setting for defrag has the following effect:
will enter direct reclaim like always but only for regions that are have used madvise(MADV_HUGEPAGE). This is the default behaviour.
with always meaning
means that an application requesting THP will stall on allocation failure and directly reclaim pages and compact memory in an effort to allocate a THP immediately. This may be desirable for virtual machines that benefit heavily from THP use and are willing to delay the VM start to utilise them
It seems to me that one of the reasons for setting MADV_HUGEPAGE is to undo setting MADV_NOHUGEPAGE and that there is no other way to do that.