Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

child_process.spawn() has undocumented 200kb buffer limit #4236

Closed
gladys123 opened this issue Dec 11, 2015 · 10 comments

Comments

@gladys123
Copy link

commented Dec 11, 2015

The child_process.spawn() function will pause a script's execution if it writes more than 200kb of stdout that isn't captured. I discovered this when my spawned process would suspiciously pause after exactly a certain amount of output.

Adding a listener for stdout that does nothing fixed it and allowed the script to run:

myProcess.stdout.on('data', function (data) {});

This is not documented in the docs. Is this intentional (and hence should be documented) or a bug?

@bnoordhuis

This comment has been minimized.

Copy link
Member

commented Dec 11, 2015

That depends on with what options you spawned the child process. With { stdio: 'pipe' } (the default), the child's stdout and stderr file descriptors are pipes and those have limited (and platform-specific) capacity. If you don't plan on consuming the output from the child, create it with { stdio: 'ignore' }.

It's possible the documentation could be clearer (pull requests welcome) but it's working as intended, it implements flow control the same way foo | bar does in the shell.

@Fishrock123

This comment has been minimized.

Copy link
Member

commented Dec 11, 2015

It's possible the documentation could be clearer

Probably, I've found that option confusing in the past.

@Fishrock123 Fishrock123 added doc and removed question labels Dec 11, 2015

@thebergamo

This comment has been minimized.

Copy link

commented Jun 29, 2016

I got this problem in 2015 and write a post on medium, maybe I can help to document this :)

@thebergamo

This comment has been minimized.

Copy link

commented Jun 29, 2016

@bnoordhuis I see the limit buffer for this cases are stored in max_buffer_string as I can see here:

node/src/spawn_sync.cc

Lines 766 to 770 in ed3d8b1

Local<Value> js_max_buffer = js_options->Get(env()->max_buffer_string());
if (IsSet(js_max_buffer)) {
if (!js_max_buffer->IsUint32())
return UV_EINVAL;
max_buffer_ = js_max_buffer->Uint32Value();
and the error comes here:

node/src/spawn_sync.cc

Lines 596 to 598 in ed3d8b1

if (max_buffer_ > 0 && buffered_output_size_ > max_buffer_) {
SetError(UV_ENOBUFS);
Kill();

I think for better documentation will be nice, if I can provide a value of the max bytes we can write in the std/out/err by default.

@tinybug

This comment has been minimized.

Copy link

commented Jul 12, 2016

it's really a problem, it will cause the process hang-up if you do not handle the stdout or stderr. I think the document should be clearer, or it will be really weird for someone who haven't handled the stdout or stderr.

@thebergamo

This comment has been minimized.

Copy link

commented Jul 12, 2016

To fix the docs just need to know whats the real limit as I say in my last comment :(

@cjihrig

This comment has been minimized.

Copy link
Contributor

commented Jul 12, 2016

I think (but I could be wrong), that @bnoordhuis comment "those have limited (and platform-specific) capacity" means that the limits you're looking for come from the operating system, and not values defined in Node/libuv.

@thebergamo

This comment has been minimized.

Copy link

commented Jul 13, 2016

Yeah, I see this comment @cjihrig, but looking in the code I'm little confused about this. :/ Will try found this information in some place :/

@sam-github sam-github added doc and removed doc docs-requested labels Dec 1, 2016

@Trott

This comment has been minimized.

Copy link
Member

commented Jul 15, 2017

AriLFrankel added a commit to AriLFrankel/node that referenced this issue Mar 1, 2018

AriLFrankel added a commit to AriLFrankel/node that referenced this issue Mar 1, 2018

doc: update child_process.md
Add an explanation of the risk of exceeding platform pipe capacity with uncaptured output in child_process.spawn with stdio of pipe

Fixes: nodejs#4236

@AriLFrankel AriLFrankel referenced this issue Mar 1, 2018

Closed

doc: update child_process.md #19075

2 of 2 tasks complete

AriLFrankel added a commit to AriLFrankel/node that referenced this issue Mar 1, 2018

doc: update child_process.md
Add an explanation of the risk of exceeding platform pipe capacity with uncaptured output in child_process.spawn with stdio of pipe

Fixes: nodejs#4236

AriLFrankel added a commit to AriLFrankel/node that referenced this issue Mar 2, 2018

doc: update child_process.md
Add an explanation of the risk of exceeding platform pipe capacity with uncaptured output in child_process.spawn with stdio of pipe

Fixes: nodejs#4236

AriLFrankel added a commit to AriLFrankel/node that referenced this issue Mar 2, 2018

doc: update child_process.md
Add an explanation of the risk of exceeding platform pipe capacity with uncaptured output in child_process.spawn with stdio of pipe

Fixes: nodejs#4236

@addaleax addaleax closed this in 5fdee52 Mar 23, 2018

FallenRiteMonk added a commit to FallenRiteMonk/node that referenced this issue Mar 23, 2018

doc: update child_process.md
Add an explanation of the risk of exceeding platform pipe
capacity with uncaptured output in child_process.spawn
with stdio of pipe

PR-URL: nodejs#19075
Fixes: nodejs#4236
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Gireesh Punathil <gpunathi@in.ibm.com>
Reviewed-By: James M Snell <jasnell@gmail.com>

targos added a commit that referenced this issue Mar 24, 2018

doc: update child_process.md
Add an explanation of the risk of exceeding platform pipe
capacity with uncaptured output in child_process.spawn
with stdio of pipe

PR-URL: #19075
Fixes: #4236
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Gireesh Punathil <gpunathi@in.ibm.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
@kopax

This comment has been minimized.

Copy link

commented Apr 20, 2018

I am using spawn in CI and for some reason it doesn't redirect to stdout directly while it does on my host.

This result to the following message:

[ERROR] stderr maxBuffer exceeded

is there a way to increase maxBuffer or to force npx to work the stdout ?

BethGriggs added a commit that referenced this issue Dec 3, 2018

doc: update child_process.md
Add an explanation of the risk of exceeding platform pipe
capacity with uncaptured output in child_process.spawn
with stdio of pipe

PR-URL: #19075
Fixes: #4236
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Gireesh Punathil <gpunathi@in.ibm.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.