Repository navigation
runtime: make Windows VirtualAlloc failure more informative #70558
Description
When a Windows build fails due to OOM, the error log looks like this:
https://logs.chromium.org/logs/golang/buildbucket/cr-buildbucket/8730352753718672849/+/u/step/19/log/2
It contains a thread dump of the go test command at the moment after VirtualAlloc failed, but I suspect the real culprits here are the test child processes, among which is x/tools/go/ssa, which is known to have a big appetite for memory. Unfortunately that information is somewhat obscure in the log.
The task of this issue is to consider whether there are ways that the Windows OOM crash could be made more informative, for example by including the committed size of the current process. Alternatively, whether there are changes we could make to go test or the builders that would point the finger of blame more clearly.
Activity
prattmic commentedon Nov 25, 2024
cc @golang/runtime @golang/windows
alexbrainman commentedon Nov 26, 2024
From the log
1455 is ERROR_COMMITMENT_LIMIT - see https://learn.microsoft.com/en-us/windows/win32/debug/system-error-codes--1300-1699- .
I suspect you need to increase your memory or page file on your computer. See for example, #33716 (comment) which had the same error 1455.
I do not know what memory figures would be helpful here. Windows memory is complicated. But I am not an expert in this area. Perhaps others might have better suggestions.
Alex
mknyszek commentedon Nov 26, 2024
See also #52546.
I became less motivated to try to improve this because failures due to a failure to map on Linux usually don't indicate a true OOM (thanks to demand paging and the OOM killing policy...), but something more silly like VMA limits or address space limits (for example, 32-bit mode under 64-bit Linux).
For Windows I think this is slightly more tractable because of the explicit commit step required, but as @alexbrainman states, it's still somewhat hard, unfortunately, since a failure to commit memory could still come from some hidden places.
(As an aside, it's really a shame that errors like this are so difficult to debug on modern systems. In theory it shouldn't be, the OS is doing all the accounting already, but it's generally not exposed in a way that's easy for anyone to interpret. Linux's OOM-killing also makes this obnoxiously hard because it force-kills which results in the destruction of information.)
Lastly, there's a ton of information that could be relevant to dump, but it's also hard to fetch and dump it when you can't even mmap (or VirtualAlloc) anymore. I suppose we could reserve some memory for, for example, reading
/proc/<pid>/statuson Linux and whatever we can use on Windows.prattmic commentedon Nov 26, 2024
When Alan brought this up, I was thinking that we could call
GetProcessMemoryInfowhen throwing and printWorkingSetSize.I don't know Windows well enough to know if this is a good idea. It was just a thought to try to give a hint whether the problem was with this process using tons of memory, or other processes using tons of memory.
prattmic commentedon Nov 26, 2024
We might even get close enough to that by just printing some of our own internal metrics, like live heap size.
runtime: print error code on Windows VirtualAlloc failure
gopherbot commentedon Jan 11, 2026
Change https://go.dev/cl/735360 mentions this issue:
runtime: print error code on Windows VirtualAlloc failureawmorgan commentedon Jan 17, 2026
Hi, I'm working on the current CL 735360. Different allocation failure paths currently print different sets of information.
sysReserveOS failure: Running
_ = make([]byte, 200*1024*1024*1024*1024)currently prints: runtime: out of memory: cannot allocate 219902325555200-byte block (4096000 in use) (Missing: The specific syscall failure and OS errno.)sysUsedOS failure: The log linked in the original issue description currently prints: runtime: VirtualAlloc of 8192 bytes failed with errno=1455 (Missing: The Go heap in-use context.)
I propose unifying the error messages across sysAllocOS, sysReserveOS, and sysUsedOS to include both the OS error and the Go heap context.
@alexbrainman suggested the following format:
runtime: VirtualAlloc of <N> bytes failed with errno=<ERRNO>: cannot allocate <N>-byte block (<INUSE> in use)Does this format look acceptable to everyone?