runtime: unactionable "runtime: failed to create new OS thread (13)" error messages during Chromium build

[randomascii]

What version of Go are you using (go version)?

Unknown

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

Windows 10 19041

What did you do?

Sometimes when building Chromium the message "runtime: failed to create new OS thread (13)" is printed. This is presumed to come from here:

https://github.com/golang/go/blame/master/src/runtime/cgo/gcc_windows_amd64.c#L29

What did you expect to see?

I expect that the Go programs which run as part of the Chromium build should succeed. If they do not succeed I expect that they will return an error code such that the build is halted. It could also be useful if they record a crash dump (intentionally crashing will do this) or print enough information (process name and command line) to give some hope of diagnosing the problem.

What did you see instead?

The message is printed and the ninja build of Chromium continues. No failure is seen which suggests that the error is, somehow, non-fatal. Here is typical output:

"c:\src\depot_tools\ninja.exe" -C out\release chrome -j 1000
ninja: Entering directory `out\release'
[1000 processes, 18048/41525 @ 54.1/s : 333.332s ] CXX obj/components/invalidation/impl/impl/mock_ack_handler.obj
runtime: failed to create new OS thread (13)
[1 processes, 41374/41374 @ 39.2/s : 1056.572s ] STAMP obj/chrome/chrome.stamp

Apologies for the lack of details but I don't know anything about how Go is used in the Chromium build system. This is happening on Google workstations with 96+ GB of RAM so resource limitations are unlikely to be the problem.

In addition to the non-ideal failure mode the failure itself is of interest - it is not clear why create thread is failing and that seems worth understanding.

[ianlancetaylor]

I don't understand why ninja keeps running. This error should cause the program to fail, by calling the C function abort. That part of this error seems related to ninja, or the way that the Chromium build procedure uses ninja, not to Go. I recommend that you file that part of this bug report elsewhere. I don't think there is anything that the Go team can do to fix it.

As far as Go is concerned, it's possible that Windows should adopt a version of the _cgo_try_pthread_create function. It looks like errno 13 is EACCES, and it looks like _beginthread returns that value if there are insufficient resources to create a new thread. Retrying as _cgo_try_pthread_create does seems worth trying.

[randomascii]

That's fair. I also don't understand why ninja keeps running. ninja is 100% reliable about detecting and reporting process failures so this suggests that there is some intermediate step that runs the failing process, or else the process is cleanly killed before the abort call completes (seems unlikely).

Unfortunately I'm not sure how to make progress because I don't know which process is hitting this failure.

It would be nice if the failure message include more information. The process name would be easiest to use, but with some preparation I could use the PID.