runtime: use process_madvise to batch scavenger memory releases on Linux

[Dorbmon]

On Linux, Go's page scavenger calls sysUnused once for each candidate range selected by scavengeOne. This normally results in a separate madvise(MADV_FREE) or madvise(MADV_DONTNEED) syscall for each range.

Since Linux 6.13, process_madvise permits these advice values for the caller's own address space and accepts an iovec containing discontiguous ranges. Could the scavenger collect a small, bounded batch of candidate ranges and advise them with one process_madvise call, while retaining the existing per-range madvise path when the syscall or advice is unavailable or denied?

This could reduce syscall overhead on fragmented heaps. Linux 6.16 and later can additionally batch the associated TLB flushes across the vector.

An implementation would need to:

• keep all selected ranges unavailable to allocation until the batch completes;
• use PIDFD_SELF on Linux 6.15 and later, or a cached self pidfd on Linux 6.13–6.14;
• detect support at runtime and preserve the existing fallback behavior;
• use bounded batches to preserve scavenger pacing and stop points; and
• handle short positive returns from process_madvise correctly.

I would like to prototype this and benchmark it directly against the current madvise path, using several batch sizes and both MADV_FREE and MADV_DONTNEED, if this direction seems reasonable.

[gabyhelp]

Related Issues

• runtime: use MADV_FREE on linux as well #23687 (closed)
• runtime: the scavenger doesn't call `MADV_NOHUGEPAGE` like it used to #55328 (closed)
• runtime: use MADV_REMOVE on Linux #61688 (closed)
• runtime: default to MADV_DONTNEED on Linux #42330 (closed)
• runtime: improve heap hugepage utilization #59960 (closed)
• runtime: smarter scavenging #30333 (closed)
• runtime: make the scavenger more prompt #16930 (closed)
• proposal: x/exp/mmap: support madvise syscall #73342

Related Code Changes

• runtime: default to MADV_DONTNEED on Linux
• runtime: use MADV_FREE on Linux if available

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[Dorbmon]

I implemented a prototype of this approach.

The prototype:

• keeps all selected ranges unavailable to allocation until the batched OS operation and any scalar fallback complete;
• batches Linux scavenging paths without a stop callback, using bounded batches of at most 8 ranges and a normal 64 KiB target on 4 KiB-page systems;
• keeps allocation-time scavenging and single-range releases on the existing scalar path; and
• detects support once, uses PIDFD_SELF or a cached self pidfd as appropriate, and falls back safely on unsupported, failed, or short operations.

I benchmarked the syscall path in a hardware-accelerated KVM guest running Linux 6.16.12 with one Intel Xeon Silver 4114 vCPU and 4 KiB pages. The direct self-process process_madvise functional test passed without skipping.

Each benchmark operation advises a fixed total of 64 KiB, divided into 1, 2, 4, or 8 discontiguous ranges. The scalar case makes one madvise syscall per range, while the vector case makes one process_madvise syscall. Resident touches the target pages outside the timed interval; AlreadyAdvised isolates repeated syscall and kernel bookkeeping overhead.

After two discarded full warm-up passes, I collected 10 one-second samples per method per leaf, alternating method order. All expected samples were present. The table shows the process_madvise change in time/op relative to madvise:

Advice / state	2 ranges	4 ranges	8 ranges
MADV_FREE / AlreadyAdvised	-24.87%	-44.21%	-53.13%
MADV_FREE / Resident	-9.15%	-25.09%	-41.02%
MADV_DONTNEED / AlreadyAdvised	-28.15%	-48.75%	-58.06%
MADV_DONTNEED / Resident	-16.49%	-20.24%	-40.68%

All comparisons in the table were significant (p < 0.001, n=10). The geomean across these cases improved from 18.94 us to 12.14 us (-35.90%).

The one-range control showed about 13% overhead for repeatedly advising an already-advised range, while the Resident difference was not significant. The prototype therefore intentionally uses the original scalar sysUnused path when there is only one range.

This is a syscall-level microbenchmark, not an end-to-end scavenger benchmark, and the one-vCPU setup does not measure multi-CPU TLB shootdown savings. Within that scope, the results support batching from two ranges onward, with increasing benefit at four and eight ranges.

[gopherbot]

Change https://go.dev/cl/799901 mentions this issue: runtime: batch scavenger releases with process_madvise on Linux

[thediveo]

do you keep fallback to the old syscall usage for older kernels? there are many 6.1 and even 5.x kernels in industry and infrastructure.

[prattmic]

Do you have a real-world application where CPU time spent in sysUnused is noticeable?

I took a look at some Google production workloads and sysUnused is (I kid you not) 0.001% of CPU time.

cc @golang/runtime

[mknyszek]

Even if it doesn't affect existing CPU time, this could allow us to increase the scavenge rate when it's supported.

The scavenge rate is currently heuristically based on ~1 page / ms, which is based on 1% of 1 CPU (the backgrounds scavenger, which runs serially), which is based on worst case numbers I took from 2019: ~5 µs to madvise(DONT_NEED) an individual 8 KiB page (so, fixed costs dominate) including the cost of touching that memory again later, rounded up to 10µs to conservatively include costs of actually finding a page to scavenge (though that tends to be very cheap with the current algorithm) and account for variance of different platforms.

On the other hand, 1 page / ms in the implementation means 1 8 KiB page or 1 2 MiB hugepage, since madvising a huge page is not really much more expensive. In cases where there's a ton of free memory, the scavenger will assume those are all hugepages and operate at a much higher byte-rate. So maybe, in fact, there's no reason to try and increase the rate, since it's already quite high for cases where there's just a ton of free memory sitting, waiting to be madvised (it certainly wasn't the case for many years at this point).

Keep in mind, we also don't return memory below a certain threshold -- we know the GC is going to use it. Excessively returning memory to the OS also has significant negative consequences; there was a release where this was a big problem. The one case where we ignore this, in the face of an exceeded memory limit, I don't think returning memory faster is really going to make things better. We're already going to start thrashing on paging memory back in anyway on first touch. So from this perspective, I'm wary of changing a policy that we know has stood the test of time, and getting ourselves into new regressions.

So, I guess I don't know where I stand on this. I think we've engineered something that deals with so many situations that this might not make any visible difference at all, even if we increase the scavenge rate, in which case it's just additional complexity.