[release/10.0] Fix Linux madvise() advice values

[janvorli]

Backport of #126966 to release/10.0

Customer Impact

• Customer reported
• Found internally

VirtualReset in GC was combining MADV_FREE (8) and MADV_DONTDUMP (16) via bitwise OR and passing the result (24) to a single madvise call. madvise takes a single advice constant — not a bitmask — so this was either rejected as EINVAL (kernels < 5.18) or silently matched MADV_DONTNEED_LOCKED (kernels ≥ 5.18), a completely different operation.

Regression

• Yes - introduced in .NET 9 by Reduce use of madvise DODUMP/DONTDUMP #95643 (an optimization to reduce syscall count).
• No

Testing

CI tests, local testing using targeted repro app from the customer, GC tests

Risk

Low. It is a simple targeted fix that just makes two calls to madvise with correct arguments where we were making one call with a combination of those values that had no effect or was outright wrong.

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @dotnet/gc
See info in area-owners.md if you want to be subscribed.

[Copilot]

Pull request overview

This backport fixes GCToOSInterface::VirtualReset on Unix by ensuring Linux madvise() receives a single advice constant per call (rather than an invalid bitwise-OR combination), restoring correct page-reclaim behavior and avoiding unintended advice interpretation on newer kernels.

Changes:

• Split the combined madvise() call into two independent calls: one for MADV_DONTDUMP (coredump hint) and one for MADV_FREE (page-reclaim hint).
• Use #ifdef MADV_FREE as the compile-time guard (instead of HAVE_MADV_FREE) and make posix_madvise(POSIX_MADV_DONTNEED) a proper fallback only when MADV_FREE isn’t available.

[JulieLeeMSFT]

LGTM.
cc @a74nh, @SwapnilGaikwad, @dhartglassMSFT, @EgorBo.

[JulieLeeMSFT]

/ba-g known issues.