Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Fix blocking / non-blocking stdio woes #3584

Closed
piscisaureus opened this Issue · 49 comments
@piscisaureus

Currently process.stdin / stdout / stderr is blocking, except when it is a pipe on windows. Weird and surprising. Very unpractical in cases where stdio is used as an IPC mechanism between node processes.

Discussion:http://logs.nodejs.org/libuv/2012-06-29#00:40:38.256

Preliminary results: have process.stdout.setBlocking(true|false) or process.stdout.writeSync.

Feel free to post thoughts here.

@cowboy

I've experienced a similar issue with stdio in Windows, and have somewhat worked around it in the meantime with this horrible, horrible hack.

While my mini exit lib works well in place of process.exit in some cases, it completely fails to help in this case:

var exitHack = require('./exit').exit;

// If there are pending stdout / stderr writes when "condition" is met...
if (condition) {
  exitHack();
}

// ...this code will still do something. Whoops!
doSomething();

I haven't seen this problem in OS X or Linux. It would be great if Node.js behaved the same cross-OS.

@bpasero

Any update? Will this land in 0.10?

@isaacs
Owner

This issue has been stagnant for months. In order for this to happen, it will need a champion.

Are you that champion? Would you like to see stdio have a more consistently blocking/nonblocking interface, which works the same on Windows and Unix?

Heed the call. Build the bits that need to go into libuv. Sketch out an API in node for it.

It seems that it won't make the 0.10 cut-off, I'm afraid. Maybe next pass.

@HenryRawas

I have started to look into this issue. Here is what I have found so far.
If stdout is not piped, then the output goes through libuv tty code which is blocking as expected.
If stdout is piped, the output goes through libuv pipe code.
The pipe code checks to see if the pipe supports overlapped IO.
In the case of piped stdout it does not - NtQueryInformationFile returns a mode of FILE_SYNCHRONOUS_IO_NONALERT, which means:
All operations on the file are performed synchronously.
Wait requests in the system that must synchronize I/O queuing and completion are not subject to alerts
In this case the pipe code uses the thread pool to perform all IO.
So when the process exits, the write requests are queued up for the thread pool to execute, but they are never executed.

I made a quick hack to simply let the Write operations execute synchronously when the pipe is in this mode.
In the tests that I have, this fixes the problem of data being lost when the process exits.

This approach means that stdout would have blocking IO behavior even when piped, without requiring any new API.

Other pipes that have mode FILE_SYNCHRONOUS_IO_NONALERT or FILE_SYNCHRONOUS_IO_ALERT would also end up being blocking.
These modes may be set on a pipe or file when creating it, but libuv never sets it.

Does this seem like an acceptable way of fixing this issue?

@bpasero

Excellent! Any chance of getting this into node 0.10.x? This fix will allow me to get rid of ugly workarounds I need to use currently to not loose process output.

@bnoordhuis

@piscisaureus and/or @sblom: request for comment, please.

@HenryRawas HenryRawas referenced this issue from a commit in MSOpenTech/node
@HenryRawas HenryRawas Issue #3584. If pipe is synchronous, such as stdio, then write data s…
…ynchronously.

Add tests to check piped stdout does not lose data on exit.
2566720
@HenryRawas HenryRawas referenced this issue from a commit in MSOpenTech/node
@HenryRawas HenryRawas Issue #3584. If pipe is synch, use synch write
When pipe used for stdout or stderr on Windows,
pip does not allow IOCP. In this case do write
on calling thread.
Added test test-child-process-stdout-flush-exit
d1f31dc
@HenryRawas

Second pull request fixes commit message problems.

@HenryRawas HenryRawas referenced this issue from a commit in MSOpenTech/node
@HenryRawas HenryRawas stdio: Fix issue #3584. Use synch write in pipe
When pipe used for stdout or stderr on Windows,
pipe does not allow IOCP. In this case do write
on calling thread.
Added test test-child-process-stdout-flush-exit
6955260
@sblom
Owner

@HenryRawas works with me. I'll look at thiS.

@HenryRawas HenryRawas referenced this issue from a commit
Commit has since been removed from the repository and is no longer available.
@HenryRawas

Updated the proposed fix. Added API uv_pipe_setblocking() to libuv and changed windows pipe to use synchronous writes if this is set. Updated node code to call uv_pipe_setblocking() for stdio over pipe.

@HenryRawas HenryRawas referenced this issue from a commit in MSOpenTech/node
@HenryRawas HenryRawas stdio: Fix issue #3584. Use synch write in pipe
When pipe used for stdout or stderr on Windows,
pipe does not allow IOCP. In this case do write
on calling thread.
Added test test-child-process-stdout-flush-exit
0b3021e
@bpasero

@HenryRawas is it realistic to get this fix in for 0.10.x?

@HenryRawas
@jdalton

Just wanted to :+1: this issue. Was just bit by it in Windows 8 with Node v0.10.12. I'm still looking for a workaround atm.

@dcherman

@jdalton Did you try the pipe to file workaround?

@jdalton

@dcherman No I haven't. Did I miss it already covered?

Here is what I'm trying to do. I noticed that the stdout output was truncated:

var foo = cp.spawn(fooPath, fooOptions);

var output = '';
foo.stdout.on('data', function(data) {
  output += data;
});

foo.on('exit', function() {
  fs.writeFileSync(outputPath, output, 'utf8');
});
@cowboy

No matter what, don't do string concat like that. Use Buffer instances and .concat them.

@jdalton

@dcherman Woo doing:

var logStream = fs.createWriteStream(outputPath);
foo.stdout.pipe(logStream);
foo.on('exit', function() {
  // do some cleanup
});

seems to work :)

@jdalton

Actually nix my issue. I tried it manually via the console and for some reason > isn't capturing all the stdout that's normally being displayed to the console. Looks like it's not a node issue.

@vladikoff

Anyone got any updates on this?

@bpasero

Will this fix Land in 0.12? I hear 0.12 is nearing completion.

@tonylukasavage

I promise to make grand offerings I'll never follow through on for whoever resolves this. Anybody else who experiences this in the meantime, fs.writeSync() worked well for me in my CLI tool. I just do this when I'm running my tests now on Windows and I always get my stdout/stderr back from exec() and spawn():

console.log = console.info = console.warn = function(m) {
    fs.writeSync(1, m);
};
console.error = function(m) {
    fs.writeSync(2, m);
};
@feklee

I ran into the issue when trying to execute Node code from Emacs on Windows. Apparently EShell, the built in shell, always uses piping for executing commands. So I didn't get output from ordinary Node apps, and I had to write wrappers, using temporary files.

Test case for cmd.exe (i.e. not Emacs' EShell):

  • hello.js:

    console.log('Hello world!');
    process.exit(5);
    
  • Run as: C:\>node hello.js | MORE

  • No output.

@elieux

I got the same problem today and after a while of searching the nets, I got here. I'm confused though. Is this fixed? If yes, why is the patch not included in the release? If not, what is missing?

@seanmonstar

@elieux the issue is still open, so it's not likely fixed.

@vladikoff

Please test https://npmjs.org/package/exit from @cowboy for a possible solution to this issue. Thanks!

@bpasero

I don't see how that module would help unless all modules would adopt it.

@cowboy

Yeah, https://npmjs.org/package/exit is not a solution to the issue, it's just a possible workaround. I'm not even sure it works everywhere yet, but hey--that's what testing is for.

@OrenMe OrenMe referenced this issue in jonnyzzz/TeamCity.Node
Open

Grunt log is truncated #29

@vladikoff

exit 0.1.1 was just published: https://npmjs.org/package/exit
0.1.1 addresses some exit code issues has a few fixes.

Help test this workaround if you get a chance, thanks!

@vladikoff vladikoff referenced this issue in gruntjs/grunt
Closed

Fixing process exit and output issues #921

@zhiwww

node-exit is not working here. Win 7, node 0.10.20. Seems like the drain event has never been triggered, so the script never exits.

@refack

@zhiwww same here (win8x64 / node 0.10.20x64)
I added a PR that solves it for me: cowboy/node-exit#3

@ErisDS ErisDS referenced this issue from a commit in ErisDS/Ghost
@ErisDS ErisDS Removing error handling using process.stderr
- process.stdin/out/error is problematic, see joyent/node#3584
84ce7cc
@kasprownik kasprownik referenced this issue from a commit
Commit has since been removed from the repository and is no longer available.
@yeputons yeputons referenced this issue in oortcloud/meteorite
Open

Windows support: first attempt #224

@yeputons

@HenryRawas any news with this one?

@thasmo

@isaacs, @sblom, @HenryRawas - sorry for spamming - any updates on this? Thanks! :)

@Panya Panya referenced this issue in stylus/stylus
Closed

version output race condition #1356

@korczis

So any updates?

@nitroduna

So is it known that writing to stdout in unix/linux is blocking even when it refers to pipes?

After analyzing a logging issue in our code, I found out this behavior and today I wrote a post in the node google group: https://groups.google.com/d/msg/nodejs/Ua4nmiNPZXY/Vq3pepnM9Q8J

That behaviour is not consistent with the API documentation http://nodejs.org/api/process.html#process_process_stdout that says it is always non-blocking when stdout refers to a pipe.

I've just seen that the libuv code was fixed to make stdout writings non-blocking in linux joyent/libuv@8fe4ca6 but then it was reverted again to blocking mode joyent/libuv@8c9cbee

I suppose it was reverted because, as I can see in older comments, your are trying to make it work blocking in windows, but meanwhile I think the API documentation should be changed to reflect the actual behaviour, because the current one causes much confusion.

@nitroduna nitroduna referenced this issue from a commit in joyent/libuv
@bnoordhuis bnoordhuis Revert "unix: set O_NONBLOCK in uv_pipe_open()"
It turns out that node.js relies on the blocking behavior of pipes in
some cases, notably when forking worker processes.  Reopens #941.

This reverts commit 8fe4ca6.
8c9cbee
@orangemocha orangemocha referenced this issue from a commit in MSOpenTech/node
@orangemocha orangemocha Windows: make stdout/sterr pipes blocking
See joyent#3584

I took 0b3021e and modified slightly
to avoid exposing the 'blocking' option. Instead, stdout and stderr
pipes are always implicitly set as blocking.

This fixes test-stream2-stderr-sync.js on Windows.
e916b64
@orangemocha orangemocha referenced this issue from a commit in MSOpenTech/node
@orangemocha orangemocha Windows: make stdout/sterr pipes blocking
See joyent#3584

I took 0b3021e and modified slightly
to avoid exposing the 'blocking' option. Instead, stdout and stderr
pipes are always implicitly set as blocking.

This fixes test-stream2-stderr-sync.js on Windows.
2e99f87
@orangemocha orangemocha referenced this issue from a commit in MSOpenTech/node
@orangemocha orangemocha Windows: make stdout/sterr pipes blocking
See joyent#3584

I took 0b3021e and modified slightly
to avoid exposing the 'blocking' option. Instead, stdout and stderr
pipes are always implicitly set as blocking.

This fixes test-stream2-stderr-sync.js on Windows.
5d1269a
@orangemocha orangemocha referenced this issue from a commit in MSOpenTech/node
@orangemocha orangemocha Windows: make stdout/sterr pipes blocking
See joyent#3584

I took 0b3021e and modified slightly
to avoid exposing the 'blocking' option. Instead, stdout and stderr
pipes are always implicitly set as blocking.

This fixes test-stream2-stderr-sync.js on Windows.
60bee33
@orangemocha orangemocha referenced this issue from a commit in MSOpenTech/node
@orangemocha orangemocha Windows: make stdout/sterr pipes blocking
See joyent#3584

I took 0b3021e and modified slightly
to avoid exposing the 'blocking' option. Instead, stdout and stderr
pipes are always implicitly set as blocking.

This fixes test-stream2-stderr-sync.js on Windows.
ceddead
@orangemocha orangemocha referenced this issue from a commit in MSOpenTech/node
@orangemocha orangemocha Windows: make stdout/sterr pipes blocking
See joyent#3584

I took 0b3021e and modified slightly
to avoid exposing the 'blocking' option. Instead, stdout and stderr
pipes are always implicitly set as blocking.

This fixes test-stream2-stderr-sync.js on Windows.
1215c45
@orangemocha orangemocha referenced this issue from a commit in MSOpenTech/node
@orangemocha orangemocha Windows: make stdout/sterr pipes blocking
See joyent#3584

I took 0b3021e and modified slightly
to avoid exposing the 'blocking' option. Instead, stdout and stderr
pipes are always implicitly set as blocking.

This fixes test-stream2-stderr-sync.js on Windows.
f4a5a0d
@orangemocha orangemocha referenced this issue from a commit in MSOpenTech/node
@orangemocha orangemocha Windows: make stdout/sterr pipes blocking
See joyent#3584

I took 0b3021e and modified slightly
to avoid exposing the 'blocking' option. Instead, stdout and stderr
pipes are always implicitly set as blocking.

This fixes test-stream2-stderr-sync.js on Windows.
b3ed2d2
@orangemocha orangemocha closed this issue from a commit
@orangemocha orangemocha src: make stdout/sterr pipes blocking
Expose `setBlocking` on Pipe's and if a pipe is being created for stdio
on windows then make the pipes blocking.

This fixes test-stream2-stderr-sync.js on Windows.

Fixes #3584
20176a9
@vladikoff

Well this is exciting, thanks @orangemocha

@dougwilson

And there was much rejoicing

@cowboy

omg

@fresheneesz

omg yay? Is it really happening?

@tonylukasavage

Glorious

@Bartvds

Sweet! :sparkles:

@yeputons

Calm down, it's just a pull request :) Not merged yet.

@tjfontaine
Owner
@gaohailang gaohailang referenced this issue from a commit in gaohailang/blog
@ErisDS ErisDS Removing error handling using process.stderr
- process.stdin/out/error is problematic, see joyent/node#3584
9b85376
@gaohailang gaohailang referenced this issue from a commit in gaohailang/blog
@gaohailang gaohailang Removing error handling using process.stderr
- process.stdin/out/error is problematic, see joyent/node#3584
8785a73
@bajtos bajtos referenced this issue from a commit in strongloop/loopback-sdk-angular-cli
@bajtos bajtos Fix unit-tests failing on windows
Execute `node bin/lb-ng` instead of `bin/lb-ng`, because Windows does not
recognize hasbang notation.

Modify lb-ng to exit in the next tick, so that stdout & stderr are flushed
before exit (workaroud for joyent/node#3584).
f917263
@RReverser RReverser referenced this issue from a commit in RReverser/acorn
@RReverser RReverser Workaround for joyent/node#3584. 6ff004a
@marijnh marijnh referenced this issue from a commit in marijnh/acorn
@RReverser RReverser Workaround for joyent/node#3584. 5ab6837
@takueof takueof referenced this issue from a commit
Commit has since been removed from the repository and is no longer available.
@takueof takueof referenced this issue from a commit
Commit has since been removed from the repository and is no longer available.
@takueof takueof referenced this issue from a commit
Commit has since been removed from the repository and is no longer available.
@takueof takueof referenced this issue from a commit
Commit has since been removed from the repository and is no longer available.
@markelog markelog referenced this issue from a commit in jscs-dev/node-jscs
@takueof takueof Avoid node.js 0.10.x exit code bug for MS Windows
Fixes #587

see also:
joyent/node#3584
98c08ed
@morficus morficus referenced this issue from a commit in morficus/Ghost
@ErisDS ErisDS Removing error handling using process.stderr
- process.stdin/out/error is problematic, see joyent/node#3584
f741b35
@lahmatiy lahmatiy referenced this issue from a commit in lahmatiy/node-jscs
@takueof takueof Avoid node.js 0.10.x exit code bug for MS Windows
Fixes #587

see also:
joyent/node#3584
4e8a72c
@SonicHedgehog SonicHedgehog referenced this issue from a commit in SonicHedgehog/brave-mouse
@SonicHedgehog SonicHedgehog Use exit on Windows to ensure stdio is flushed c378c98
@avital avital referenced this issue from a commit in meteor/meteor
@avital avital Flush buffer on `process.exit` in Windows
This works around a bug that's present in Node
0.8 and 0.10. This will likely be fixed in 0.12.
joyent/node#3584
ed06b75
@amavisca amavisca referenced this issue from a commit in jasmine/jasmine-npm
Christopher Amavisca Use exit module instead of process.exit for Windows workaround
- process.exit isn't properly waiting for buffers to finish writing in Windows
- issue in node fixed in v0.11.12: joyent/node#3584

Fixes #20
92f47d3
@agebert agebert referenced this issue from a commit in e2ebridge/e2e-conf
@agebert agebert bug: Fix for Windows' drain error on process.exit (Issue 3584)
Output to stderr or stdout is not always completly written if
you exit the process. Issue happens only on Windows using pipes.

Issue joyent/node#3584 will be fixed in
node.js 0.12.
9ac7558
@pmeijer pmeijer referenced this issue from a commit in webgme/webgme
@pmeijer pmeijer Add error-handling for rj-build gulp task.
This is currently not working, awaits joyent/node#3584
4071833
@samuelbjohnson samuelbjohnson referenced this issue in RetireJS/retire.js
Merged

Fix console.error process.exit flush issue #88

@pgilad

Hi, is this considered resolved? Is the use of https://github.com/cowboy/node-exit still recommended for some versions of node?

@orangemocha
Owner

Yes, it is resolved. It was fixed with 20176a9.
You shouldn't need the workaround anymore.

@pgilad

Great, thanks!

@gigapromoters

Issue still exists for me in 0.12.4, Windows 8.1:

var child = require('child_process').spawn('php',
['-S', 'localhost:8007']);
child.stdout.on('data', function(data) {
console.log("child: "+data.toString());
});

Not sure if this is a bug, but if the process is infinite - then the child.stdout's callback is never triggered.

@gigapromoters

I wrapped php process in another child script, now it works fine.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.