Insufficient system resources exist to complete the requsted service. #8676

KrzysztofKarol · 2019-06-19T11:40:44Z

Description of the problem / feature request:

Bazel installed by scoop or downlaoded binary crashes on run.
Prerequisites + curl + versions/python27 installed.

PS H:\> bazel --version
bazel 0.27.0
PS H:\> bazel help
FATAL: ExecuteProgram(C:\Users\krzyskar/_bazel_krzyskar/install/388981886003af3376cd5aa448ae98ff/_embedded_binaries/embe
dded_tools/jdk/bin/java.exe) failed: ERROR: src/main/native/windows/process.cc(184): CreateProcessW("C:\Users\krzyskar\_
bazel_krzyskar\install\388981886003af3376cd5aa448ae98ff\_embedded_binaries\embedded_tools\jdk\bin\java.exe"  -XX:+HeapDu
mpOnOutOfMemoryError -XX:HeapDumpPath=c:\\\users\\\krzy(...)): Insufficient system resources exist to complete the reque
sted service.

PS H:\> bazel
FATAL: ExecuteProgram(C:\Users\krzyskar/_bazel_krzyskar/install/388981886003af3376cd5aa448ae98ff/_embedded_binaries/embe
dded_tools/jdk/bin/java.exe) failed: ERROR: src/main/native/windows/process.cc(184): CreateProcessW("C:\Users\krzyskar\_
bazel_krzyskar\install\388981886003af3376cd5aa448ae98ff\_embedded_binaries\embedded_tools\jdk\bin\java.exe"  -XX:+HeapDu
mpOnOutOfMemoryError -XX:HeapDumpPath=c:\\\users\\\krzy(...)): Insufficient system resources exist to complete the reque
sted service.

PS H:\>

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

bazel

What operating system are you running Bazel on?

OS Name:                   Microsoft Windows 7 Enterprise
OS Version:                6.1.7601 Service Pack 1 Build 7601

Processor(s):              1 Processor(s) Installed.
                           [01]: Intel64 Family 6 Model 58 Stepping 9 GenuineIntel ~3401 Mhz

Total Physical Memory:     32,655 MB
Available Physical Memory: 13,288 MB
Virtual Memory: Max Size:  32,653 MB
Virtual Memory: Available: 12,946 MB
Virtual Memory: In Use:    19,707 MB

What's the output of `bazel info release`?

PS H:\> bazel info release
FATAL: ExecuteProgram(C:\Users\krzyskar/_bazel_krzyskar/install/388981886003af3376cd5aa448ae98ff/_embedded_binaries/embe
dded_tools/jdk/bin/java.exe) failed: ERROR: src/main/native/windows/process.cc(184): CreateProcessW("C:\Users\krzyskar\_
bazel_krzyskar\install\388981886003af3376cd5aa448ae98ff\_embedded_binaries\embedded_tools\jdk\bin\java.exe"  -XX:+HeapDu
mpOnOutOfMemoryError -XX:HeapDumpPath=c:\\\users\\\krzy(...)): Insufficient system resources exist to complete the reque
sted service.

Have you found anything relevant by searching the web?

https://stackoverflow.com/questions/56564539/how-do-i-give-bazel-more-resources

Temporary solution:
Install bazel@0.25.0, 0.26.0 is not working as well.

The text was updated successfully, but these errors were encountered:

irengrig · 2019-06-19T16:21:41Z

/cc @meisterT

meisterT · 2019-06-19T16:33:56Z

Can you try with an explicit argument for -Xmx, e.g.
bazel --host_jvm_args=-Xmx4g info?

KrzysztofKarol · 2019-06-19T18:08:07Z

Great idea! I didn't think about it. Unfortunately I will test it on Monday.

KrzysztofKarol · 2019-06-24T12:16:23Z

Unfortunately

PS H:\> bazel --host_jvm_args=-Xmx4g info
FATAL: ExecuteProgram(C:\Users\krzyskar/_bazel_krzyskar/install/388981886003af3376cd5aa448ae98ff/_embedded_binaries/embe
dded_tools/jdk/bin/java.exe) failed: ERROR: src/main/native/windows/process.cc(184): CreateProcessW("C:\Users\krzyskar\_
bazel_krzyskar\install\388981886003af3376cd5aa448ae98ff\_embedded_binaries\embedded_tools\jdk\bin\java.exe"  -XX:+HeapDu
mpOnOutOfMemoryError -XX:HeapDumpPath=c:\\\users\\\krzy(...)): Insufficient system resources exist to complete the reque
sted service.

krzyskar@COMPUTERNAME MINGW64 /c/projects
$ bazel --host_jvm_args=-Xmx4g info
FATAL: ExecuteProgram(C:\Users\krzyskar/_bazel_krzyskar/install/388981886003af3376cd5aa448ae98ff/_embedded_binaries/embedded_tools/jdk/bin/java.exe) failed: ERROR: src/main/native/windows/process.cc(184): CreateProcessW("C:\Users\krzyskar\_bazel_krzyskar\install\388981886003af3376cd5aa448ae98ff\_embedded_binaries\embedded_tools\jdk\bin\java.exe"  -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=c:\\\users\\\krzy(...)): Insufficient system resources exist to complete the requested service.

I tried also setting the initial memory allocation pool without luck

PS H:\> bazel --host_jvm_args=-Xms2G --host_jvm_args=-Xmx16G info
FATAL: ExecuteProgram(C:\Users\krzyskar/_bazel_krzyskar/install/388981886003af3376cd5aa448ae98ff/_embedded_binaries/embe
dded_tools/jdk/bin/java.exe) failed: ERROR: src/main/native/windows/process.cc(184): CreateProcessW("C:\Users\krzyskar\_
bazel_krzyskar\install\388981886003af3376cd5aa448ae98ff\_embedded_binaries\embedded_tools\jdk\bin\java.exe"  -XX:+HeapDu
mpOnOutOfMemoryError -XX:HeapDumpPath=c:\\\users\\\krzy(...)): Insufficient system resources exist to complete the reque
sted service.

More info:

PS H:\> C:\Users\krzyskar\_bazel_krzyskar\install\388981886003af3376cd5aa448ae98ff\_embedded_binaries\embedded_tools\jdk
\bin\java.exe -version
openjdk version "11.0.2" 2019-01-15 LTS
OpenJDK Runtime Environment Zulu11.29+3-CA (build 11.0.2+7-LTS)
OpenJDK 64-Bit Server VM Zulu11.29+3-CA (build 11.0.2+7-LTS, mixed mode)

KrzysztofKarol · 2019-07-03T08:13:57Z

Update

PS H:\> bazel --version
bazel 0.27.1
PS H:\> bazel info
FATAL: ExecuteProgram(C:\Users\krzyskar/_bazel_krzyskar/install/9470fb600225157f1e34c45c6d4dc834/_embedded_binaries/embe
dded_tools/jdk/bin/java.exe) failed: ERROR: src/main/native/windows/process.cc(184): CreateProcessW("C:\Users\krzyskar\_
bazel_krzyskar\install\9470fb600225157f1e34c45c6d4dc834\_embedded_binaries\embedded_tools\jdk\bin\java.exe"  -XX:+HeapDu
mpOnOutOfMemoryError -XX:HeapDumpPath=c:\\\users\\\krzy(...)): Insufficient system resources exist to complete the reque
sted service.

0.27.1 is not working as well.

Is it possible to use "external" JDK?

meisterT · 2019-07-03T08:19:30Z

You can point Bazel to a different JDK with --server_javabase (https://docs.bazel.build/versions/master/command-line-reference.html#flag--server_javabase).
Is that message actually coming from the JVM or from Windows? cc @meteorcloudy in case he has further ideas.

KrzysztofKarol · 2019-07-03T08:41:57Z

Thank you @meisterT . I should check out the doc first.

PS H:\> echo $env:JAVA_HOME
C:\Users\krzyskar\scoop\apps\openjdk\current

PS H:\> Invoke-Expression $($env:JAVA_HOME + "\bin\java -version")
openjdk version "12.0.1" 2019-04-16
OpenJDK Runtime Environment (build 12.0.1+12)
OpenJDK 64-Bit Server VM (build 12.0.1+12, mixed mode, sharing)

PS H:\> bazel --server_javabase $env:JAVA_HOME
FATAL: ExecuteProgram(c:\users\krzyskar\scoop\apps\openjdk\current/bin/java.exe) failed: ERROR: src/main/native/windows/
process.cc(184): CreateProcessW("c:\users\krzyskar\scoop\apps\openjdk\current\bin\java.exe"  -XX:+HeapDumpOnOutOfMemoryE
rror -XX:HeapDumpPath=c:\\\users\\\krzyskar\\\_bazel_krzyskar\\\u5mzpase -XX:+UseParallelOldGC --add-opens=(...)): Insuf
ficient system resources exist to complete the requested service.

Is that message actually coming from the JVM or from Windows?
How to differentiate that?

Interesting thing.
bazel fails on PS, cmd, ConEmu with Git Bash but works on Git Bash

krzyskar@COMPUTERNAME MINGW64 ~
$ bazel
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
                                                          [bazel release 0.27.1]
Usage: bazel <command> <options> ...

laszlocsomor · 2019-07-04T09:28:29Z

To my own surprise, I can repro this!

Created a GCP VM with Windows Server 2008, 1 CPU + 32GB RAM
Installed Bazel 0.27.1 through Chocolatey
Installed Zulu JDK 8, 64bit
Installed VC redist DLLs
Created a hello world java_binary
bazel run //:hello fails the same way @KrzysztofKarol described

I'll try to investigate.

laszlocsomor · 2019-07-04T09:45:43Z

More info:

error is not Java specific: bazel run fails also for py_binary
also not Bash specific: bazel run runs all binaries through bash.exe by default, but it also fails with --incompatible_windows_bashless_run_command, i.e. when not running through Bash
there are enough resources: using any of the binaries in genrule.tools and building the genrule works

CreateProcess fails with ERROR_NO_SYSTEM_RESOURCES.

And https://bugzilla.mozilla.org/show_bug.cgi?id=1460995 seems very relevant:

CreateProcess fails with ERROR_NO_SYSTEM_RESOURCES on Windows 7. It looks like the reason why is because PROC_THREAD_ATTRIBUTE_HANDLE_LIST doesn't like console handles.

We can fix this by only setting that policy (and its associated inheritable handles) when there is one or more standard handles that are not console handles.

Note that if all handles are console handles, we don't need to pass STARTF_USESTDHANDLES either, since they'll automatically be set to the current console.

laszlocsomor · 2019-07-04T10:05:04Z

And that is indeed the problem!

Here:

bazel/src/main/native/windows/process.cc

Lines 166 to 187 in 6d0b14b

    
           PROCESS_INFORMATION process_info = {0}; 
        
           STARTUPINFOEXW info; 
        
           attr_list->InitStartupInfoExW(&info); 
        
           if (!CreateProcessW( 
        
                   /* lpApplicationName */ NULL, 
        
                   /* lpCommandLine */ mutable_commandline.get(), 
        
                   /* lpProcessAttributes */ NULL, 
        
                   /* lpThreadAttributes */ NULL, 
        
                   /* bInheritHandles */ attr_list->InheritAnyHandles() ? TRUE : FALSE, 
        
                   /* dwCreationFlags */ (create_window ? 0 : CREATE_NO_WINDOW) | 
        
                       CREATE_NEW_PROCESS_GROUP  // So that Ctrl-Break isn't propagated 
        
                       | CREATE_SUSPENDED  // So that it doesn't start a new job itself 
        
                       | EXTENDED_STARTUPINFO_PRESENT | CREATE_UNICODE_ENVIRONMENT, 
        
                   /* lpEnvironment */ env, 
        
                   /* lpCurrentDirectory */ cwd.empty() ? NULL : cwd.c_str(), 
        
                   /* lpStartupInfo */ &info.StartupInfo, 
        
                   /* lpProcessInformation */ &process_info)) { 
        
             DWORD err = GetLastError(); 
        
             *error = MakeErrorMessage(WSTR(__FILE__), __LINE__, L"CreateProcessW", 
        
                                       cmd_sample, err); 
        
             return false; 
        
           }

when I don't use the attribute list, remove EXTENDED_STARTUPINFO_PRESENT, and use a STARTUPINFOW instead of STARTUPINFOEXW and let the subprocess inherit none of the handles, bazel run works. (And also prints to the console.)

On Windows 7, the subprocess cannot inherit console handles via the attribute list (CreateProcessW fails with ERROR_NO_SYSTEM_RESOURCES). However, if we: - don't specify STARTF_USESTDHANDLES in STARTUPINFOW::dwFlags, and - set CreateProcessW:bInheritHandles to FALSE, then we disallow inheriting the current process' open and inheritable handles. Confusingly though, if on top of that we: - don't specify CREATE_NEW_CONSOLE nor DETACHED_PROCESS in CreateProcessW, and then, in this special case, the subprocess *does* inherit the console handles, but nothing else (i.e. none of the inheritable, open file handles for example). This is arguably confusing but seems to work and do exactly what we need: for the subprocess to inherit the console handles and nothing else. bazelbuild#8676

On Windows 7, the subprocess cannot inherit console handles via the attribute list (CreateProcessW fails with ERROR_NO_SYSTEM_RESOURCES). However, if we: - don't specify STARTF_USESTDHANDLES in STARTUPINFOW::dwFlags, and - set CreateProcessW:bInheritHandles to FALSE, then we disallow inheriting the current process' open and inheritable handles. However, if on top of that we also: - don't specify CREATE_NEW_CONSOLE nor DETACHED_PROCESS in CreateProcessW:dwCreationFlags, then, in this special case, the subprocess *does* inherit the console handles, but nothing else (i.e. none of the inheritable, open file handles for example). This is rather confusing, but seems to work and do exactly what we need: for the subprocess to inherit the console handles and nothing else. #8676 Closes #8793. PiperOrigin-RevId: 256546818

On Windows 7, the subprocess cannot inherit console handles via the attribute list (CreateProcessW fails with ERROR_NO_SYSTEM_RESOURCES). However, if we: - don't specify STARTF_USESTDHANDLES in STARTUPINFOW::dwFlags, and - set CreateProcessW:bInheritHandles to FALSE, then we disallow inheriting the current process' open and inheritable handles. However, if on top of that we also: - don't specify CREATE_NEW_CONSOLE nor DETACHED_PROCESS in CreateProcessW:dwCreationFlags, then, in this special case, the subprocess *does* inherit the console handles, but nothing else (i.e. none of the inheritable, open file handles for example). This is rather confusing, but seems to work and do exactly what we need: for the subprocess to inherit the console handles and nothing else. bazelbuild#8676 Closes bazelbuild#8793. PiperOrigin-RevId: 256546818

laszlocsomor · 2019-07-05T07:05:55Z

Fixed by #8793

On Windows 7, the subprocess cannot inherit console handles via the attribute list (CreateProcessW fails with ERROR_NO_SYSTEM_RESOURCES). However, if we: - don't specify STARTF_USESTDHANDLES in STARTUPINFOW::dwFlags, and - set CreateProcessW:bInheritHandles to FALSE, then we disallow inheriting the current process' open and inheritable handles. However, if on top of that we also: - don't specify CREATE_NEW_CONSOLE nor DETACHED_PROCESS in CreateProcessW:dwCreationFlags, then, in this special case, the subprocess *does* inherit the console handles, but nothing else (i.e. none of the inheritable, open file handles for example). This is rather confusing, but seems to work and do exactly what we need: for the subprocess to inherit the console handles and nothing else. bazelbuild#8676 Closes bazelbuild#8793. PiperOrigin-RevId: 256546818

irengrig added area-EngProd Bazel CI, infrastructure, bootstrapping, release, and distribution tooling untriaged labels Jun 19, 2019

meisterT added area-Windows Windows-specific issues and feature requests and removed area-EngProd Bazel CI, infrastructure, bootstrapping, release, and distribution tooling labels Jul 3, 2019

laszlocsomor added P2 We'll consider working on this in future. (Assignee optional) type: bug and removed untriaged labels Jul 4, 2019

laszlocsomor mentioned this issue Jul 4, 2019

Windows: fix "insufficient resources" on Windows 7 #8793

Closed

laszlocsomor self-assigned this Jul 4, 2019

laszlocsomor closed this as completed Jul 5, 2019

philwo added the team-OSS Issues for the Bazel OSS team: installation, release processBazel packaging, website label Jun 15, 2020

alexbrainman mentioned this issue May 8, 2021

os: ERROR_NO_SYSTEM_RESOURCES when making child process use os.Stdout on Windows 7 golang/go#45914

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Insufficient system resources exist to complete the requsted service. #8676

Insufficient system resources exist to complete the requsted service. #8676

KrzysztofKarol commented Jun 19, 2019

irengrig commented Jun 19, 2019

meisterT commented Jun 19, 2019

KrzysztofKarol commented Jun 19, 2019

KrzysztofKarol commented Jun 24, 2019 •

edited

Loading

KrzysztofKarol commented Jul 3, 2019

meisterT commented Jul 3, 2019

KrzysztofKarol commented Jul 3, 2019 •

edited

Loading

laszlocsomor commented Jul 4, 2019

laszlocsomor commented Jul 4, 2019 •

edited

Loading

laszlocsomor commented Jul 4, 2019

laszlocsomor commented Jul 5, 2019

Insufficient system resources exist to complete the requsted service. #8676

Insufficient system resources exist to complete the requsted service. #8676

Comments

KrzysztofKarol commented Jun 19, 2019

Description of the problem / feature request:

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

What operating system are you running Bazel on?

What's the output of bazel info release?

Have you found anything relevant by searching the web?

irengrig commented Jun 19, 2019

meisterT commented Jun 19, 2019

KrzysztofKarol commented Jun 19, 2019

KrzysztofKarol commented Jun 24, 2019 • edited Loading

KrzysztofKarol commented Jul 3, 2019

meisterT commented Jul 3, 2019

KrzysztofKarol commented Jul 3, 2019 • edited Loading

laszlocsomor commented Jul 4, 2019

laszlocsomor commented Jul 4, 2019 • edited Loading

laszlocsomor commented Jul 4, 2019

laszlocsomor commented Jul 5, 2019

What's the output of `bazel info release`?

KrzysztofKarol commented Jun 24, 2019 •

edited

Loading

KrzysztofKarol commented Jul 3, 2019 •

edited

Loading

laszlocsomor commented Jul 4, 2019 •

edited

Loading