Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

stdio buffered writes (chunked) issues & process.exit() truncation #6456

Open
eljefedelrodeodeljefe opened this issue Apr 28, 2016 · 171 comments
Open

Comments

@eljefedelrodeodeljefe
Copy link
Contributor

@eljefedelrodeodeljefe eljefedelrodeodeljefe commented Apr 28, 2016

If this is currently breaking your program, please use this temporary fix:

[process.stdout, process.stderr].forEach((s) => {
  s && s.isTTY && s._handle && s._handle.setBlocking &&
    s._handle.setBlocking(true)
})

  • Version: v6, (likely all and backportable)
  • Platform: all
  • Subsystem: process

As noted in #6297 async stdio will not be flushed upon immediate process.exit(). This may lay open general deficiencies around C exit() from C++ functions not being properly unwound and is probably not just introduced by latest libuv updates. It should be considered to add flushing, providing graceful exit and/or improving unwinding C++ stacks.

cc @jasnell, @kzc, @Qix-, @bnoordhuis

Issues

Discussion has been already taking place at several places, e.g. #6297, #6456, #6379

Summaries of Proposals

proposals are not exclusive and could lead to semantically unrelated contributions.

  • aid with process.stdout.flush()
  • process.setBlocking(true)
  • node --blocking-stdio
  • longjmp() towards main at exit in C++
  • move parts of process.exit() / process.reallyExit() to new method os.exit()
  • golang panic()- or c++ throw-like stack unwinding

Discussions by Author (with content)


@ChALkeR
I tried to discuss this some time ago at IRC, but postponed it for quite a long time. Also I started the discussion of this in #1741, but I would like to extract the more specific discussion to a separate issue.

I could miss some details, but will try to give a quick overview here.

Several issues here:

  1. Many calls to console.log (e.g. calling it in a loop) could chew up all the memory and die — #1741, #2970#3171.
  2. console.log has different behavior while printing to a terminal and being redirected to a file. — #1741 (comment).
  3. Output is sometimes truncated — #6297, there were other ones as far as I remember.
  4. The behaviour seems to differ across platforms.

As I understand it — the output has an implicit write buffer (as it's non-blocking) of unlimited size.

One approach to fixing this would be to:

  1. Introduce an explicit cyclic write buffer.
  2. Make writes to that cyclic buffer blocking.
  3. Make writes from the buffer to the actual output non blocking.
  4. When the cyclic buffer reaches it's maximum size (e.g. 10 MiB) — block further writes to the buffer until a corresponding part of it is freed.
  5. On (normal) exit, make sure the buffer is flushed.

For almost all cases, except for the ones that are currently broken, this would behave as a non-blocking buffer (because writes to the buffer are considerably faster than writes from the buffer to file/terminal).

For cases when the data is being piped to the output too quickly and when the output file/terminal does not manage to output it at the same rate — the write would turn into a blocking operation. It would also be blocking at the exit until all the data is written.

Another approach would be to monitor (and limit) the size of data that is contained in the implicit buffer coming from the async queue, and make the operations block when that limit is reached.

@Qix-
Copy link

@Qix- Qix- commented Apr 28, 2016

Perhaps a list of issues this would address and/or close would be helpful to include since this seems to be a sprawling issue with a lot of fragmented discussion.

@eljefedelrodeodeljefe
Copy link
Contributor Author

@eljefedelrodeodeljefe eljefedelrodeodeljefe commented Apr 28, 2016

Yes, just a little late in Europe :( keep 'em coming and I add them above.

@vsemozhetbyt
Copy link
Contributor

@vsemozhetbyt vsemozhetbyt commented Apr 28, 2016

Also see #6410

@addaleax addaleax added the process label Apr 28, 2016
@vsemozhetbyt
Copy link
Contributor

@vsemozhetbyt vsemozhetbyt commented Apr 28, 2016

Considering all the clarification in the #6410, is there also a theoretical possibility that not only several I/O calls to stdout could not make it, but even one simple console.log() before process.exit() could be truncated or discarded?

@Qix-
Copy link

@Qix- Qix- commented Apr 28, 2016

@vsemozhetbyt that is especially correct if I'm understanding your question correctly.

@addaleax
Copy link
Member

@addaleax addaleax commented Apr 28, 2016

@vsemozhetbyt If it’s big enough, definitely. See e.g. test/known_issues/test-stdout-buffer-flush-on-exit.js.

@eljefedelrodeodeljefe
Copy link
Contributor Author

@eljefedelrodeodeljefe eljefedelrodeodeljefe commented Apr 28, 2016

To reproduce you can do

require('crypto').randomBytes(100000000, function(err, buffer) {
  var token = buffer.toString('hex');
  console.log(token);
  process.exit(0)
});

Edit: @addaleax's hint: test does a similar thing. Sorry @addaleax

@kzc
Copy link

@kzc kzc commented Apr 28, 2016

@vsemozhetbyt This output is truncated with node 6.0.0 on Mac after approx 40 lines:

node -e 'console.log("The quick brown fox jumps.\n".repeat(40000)); process.exit(7);'

node 5.x and earlier output all 40000 lines on Mac.

@vsemozhetbyt
Copy link
Contributor

@vsemozhetbyt vsemozhetbyt commented Apr 28, 2016

So now if user does not want to reflow the code all one has is to write something like

const err = {name: 'Error', message: 'something wrong'};
throw err;

instead of

console.log('Error: something wrong');
process.exit(1);

and to deal with all the uncontrolled clutter of debug output?

@kzc
Copy link

@kzc kzc commented Apr 28, 2016

throw err;
and to deal with all the uncontrolled clutter of debug output?

For dev code, sure, but uncaught exceptions in production code is not very elegant or professional.

@kzc
Copy link

@kzc kzc commented Apr 29, 2016

Related: #6379

Also discusses process.stdio.setBlocking(Boolean)

@eljefedelrodeodeljefe
Copy link
Contributor Author

@eljefedelrodeodeljefe eljefedelrodeodeljefe commented Apr 29, 2016

added @chalkers thread and updated this issue with some summaries and stuff.

@Qix-
Copy link

@Qix- Qix- commented Apr 29, 2016

@kzc

node 5.x and earlier output all 40000 lines on Mac.

Not sure what you're talking about.

#!/usr/bin/env bash
. ~/.nvm/nvm.sh

uname -a
echo

function do_buffer_test {
    node <<< 'console.log((new Array(40000)).join("Hello! this is a test!\n"));' | wc -l
    node <<< 'console.log((new Array(40000)).join("Hello! this is a test!\n")); process.exit(1)' | wc -l
    node <<< 'console.log((new Array(40000)).join("Hello! this is a test!\n")); process.reallyExit(1)' | wc -l
    node <<< 'console.log((new Array(40000)).join("Hello! this is a test!\n")); process.abort()' | wc -l
    node <<< 'for (var i = 0; i < 40000; i++) console.log("Hello! this is a test!");' | wc -l
    node <<< 'for (var i = 0; i < 40000; i++) console.log("Hello! this is a test!"); process.exit(1)' | wc -l
    node <<< 'for (var i = 0; i < 40000; i++) console.log("Hello! this is a test!"); process.reallyExit(1)' | wc -l
    node <<< 'for (var i = 0; i < 40000; i++) console.log("Hello! this is a test!"); process.abort()' | wc -l
}

nvm install 0.10
do_buffer_test

nvm install 0.12
do_buffer_test

nvm install 1
do_buffer_test

nvm install 2
do_buffer_test

nvm install 3
do_buffer_test

nvm install 4
do_buffer_test

nvm install 5
do_buffer_test

nvm install 6
do_buffer_test
$ ./test-buffers.sh
Darwin JunonBox.local 15.4.0 Darwin Kernel Version 15.4.0: Fri Feb 26 22:08:05 PST 2016; root:xnu-3248.40.184~3/RELEASE_X86_64 x86_64

v0.10.44 is already installed.
Now using node v0.10.44 (npm v2.15.0)
   40000
   40000
   40000
   40000
   40000
   40000
   40000
   40000
v0.12.13 is already installed.
Now using node v0.12.13 (npm v2.15.0)
   40000
   40000
   40000
   40000
   40000
   40000
   40000
   40000
iojs-v1.8.4 is already installed.
Now using io.js v1.8.4 (npm v2.9.0)
   40000
    2849
    2849
    2849
   40000
   40000
   40000
   40000
iojs-v2.5.0 is already installed.
Now using io.js v2.5.0 (npm v2.13.2)
   40000
    2849
    2849
    2849
   40000
   40000
   40000
   40000
iojs-v3.3.1 is already installed.
Now using io.js v3.3.1 (npm v2.14.3)
   40000
    2849
    2849
    2849
   40000
   40000
   40000
   40000
v4.4.3 is already installed.
Now using node v4.4.3 (npm v2.15.1)
   40000
    2849
    2849
    2849
   40000
   40000
   40000
   40000
v5.11.0 is already installed.
Now using node v5.11.0 (npm v3.8.6)
   40000
    2849
    2849
    2849
   40000
   40000
   40000
   40000
v6.0.0 is already installed.
Now using node v6.0.0 (npm v3.8.6)
   40000
    2849
    2849
    2849
   40000
   40000
   40000
   40000
$ ./test-buffers.sh
Linux -snip- 3.18.27 #1 SMP Wed Feb 17 01:14:23 UTC 2016 x86_64 GNU/Linux

######################################################################## 100.0%
Now using node v0.10.44 (npm v2.15.0)
Creating default alias: default -> 0.10 (-> v0.10.44)
40000
40000
40000
40000
40000
40000
40000
40000
######################################################################## 100.0%
Now using node v0.12.13 (npm v2.15.0)
40000
40000
40000
40000
40000
40000
40000
40000
Downloading https://iojs.org/dist/v1.8.4/iojs-v1.8.4-linux-x64.tar.gz...
######################################################################## 100.0%
Now using io.js v1.8.4 (npm v2.9.0)
40000
2849
2849
2849
40000
40000
40000
40000
Downloading https://iojs.org/dist/v2.5.0/iojs-v2.5.0-linux-x64.tar.xz...
######################################################################## 100.0%
Now using io.js v2.5.0 (npm v2.13.2)
40000
2849
2849
2849
40000
40000
40000
40000
Downloading https://iojs.org/dist/v3.3.1/iojs-v3.3.1-linux-x64.tar.xz...
######################################################################## 100.0%
Now using io.js v3.3.1 (npm v2.14.3)
40000
2849
2849
2849
40000
40000
40000
40000
Downloading https://nodejs.org/dist/v4.4.3/node-v4.4.3-linux-x64.tar.xz...
######################################################################## 100.0%
Now using node v4.4.3 (npm v2.15.1)
40000
2849
2849
2849
40000
40000
40000
40000
Downloading https://nodejs.org/dist/v5.11.0/node-v5.11.0-linux-x64.tar.xz...
######################################################################## 100.0%
Now using node v5.11.0 (npm v3.8.6)
40000
2849
2849
2849
40000
40000
40000
40000
Downloading https://nodejs.org/dist/v6.0.0/node-v6.0.0-linux-x64.tar.xz...
######################################################################## 100.0%
Now using node v6.0.0 (npm v3.8.6)
40000
2849
2849
2849
40000
40000
40000
40000

Looks to me whenever io.js forked is when this started happening. Perhaps @indutny can shed some light on the subject.

@eljefedelrodeodeljefe
Copy link
Contributor Author

@eljefedelrodeodeljefe eljefedelrodeodeljefe commented Apr 29, 2016

Adding two other possibilities.

  • move parts of process.exit() / process.reallyExit() to new method os.exit(), which may fit what actually is happening better
  • golang panic()- or c++ throw-like stack unwinding.
@kzc
Copy link

@kzc kzc commented Apr 29, 2016

@Qix- @eljefedelrodeodeljefe Please understand that when you pipe the results you are changing the test. It follows a different code path in node and libuv. You have to observe it on the terminal. So in that regard, the test is more difficult to automate.

I am observing this behavior of Mac OS X 10.9.5. The behavior is different on Mac and Linux. Mac stdout appears to have had blocking writes to the tty historically. See #6297 (comment)

@kzc
Copy link

@kzc kzc commented Apr 29, 2016

+1 for new function os.exit(Boolean) that drains stdout/stderr upon exit and leave process.exit() as is.

Actually may have to leave process.exit as is because of the prevalence of workarounds to the stdout flushing problem such as node-exit which might break if the behavior of process.exit changes.

@saghul
Copy link
Member

@saghul saghul commented Apr 29, 2016

If memory serves right, and by looking at the results posted by @Qix- I think this is where things started to change: libuv/libuv@b197515 Because we started to open writable TTYs in non-blocking mode. Follow the commit trail for reasonin, reverting is not an option.

@kzc
Copy link

@kzc kzc commented Apr 29, 2016

reverting is not an option.

Yes, and that particular commit also introduced the tty redirection bug in src/unix/tty.c that was fixed in libuv 1.9.0.

@saghul
Copy link
Member

@saghul saghul commented Apr 29, 2016

@kzc your point being?

@kzc
Copy link

@kzc kzc commented Apr 29, 2016

Just adding weight to reverting is not an option.

@kzc
Copy link

@kzc kzc commented Apr 29, 2016

@saghul I will add that prior to the tty redirection fix, Mac stdout appeared to be blocking based on my observations with process.exit tests never truncating tty output on Mac. As of that tty redirect fix it is now non-blocking to make it on par with Linux behavior.

@bnoordhuis
Copy link
Member

@bnoordhuis bnoordhuis commented Apr 29, 2016

Can I suggest closing out all other issues with a "discussion continues in #6456" comment?

@jasnell
Copy link
Member

@jasnell jasnell commented Apr 29, 2016

+1 ... we don't need multiple issues covering the same thing.

jasnell added a commit to jasnell/node that referenced this issue Apr 29, 2016
When set, an internal timer will be set that will exit the
process at the given timeout. In the meantime, the registered
listeners for process.on('exitingSoon') will be invoked and
passed a callback to be called when the handler is ready for
the process to exit. The process will exit either when the
internal timer fires or all the callbacks are called, whichever
comes first.

This is an attempt to deal more intelligently with resource
cleanup and async op completion on exit

(see nodejs#6456).
@jasnell jasnell mentioned this issue Apr 29, 2016
0 of 4 tasks complete
@jasnell
Copy link
Member

@jasnell jasnell commented Apr 29, 2016

See: #6477

@Trott
Copy link
Member

@Trott Trott commented May 28, 2017

@Fishrock123 I'm removing this note from the post at the top:

CTC Note (Fishrock123): We are working on resolving this. Unfortunately it's not the most trivial thing to investigate or fix well.

I don't think anyone is actually working on this actively at the current time. If that's wrong, feel free to put the note back, of course.

kachkaev added a commit to kachkaev/run-elm that referenced this issue Feb 16, 2018
Related to nodejs/node#6456
jfairbank added a commit to jfairbank/run-elm that referenced this issue Feb 21, 2018
* Support long output from Elm

Related to nodejs/node#6456

* Update CHANGELOG

* Configure tests with test-config.js instead of input.txt and output.txt
@Trott
Copy link
Member

@Trott Trott commented Mar 4, 2018

Is this still an issue? (If so, is anyone working on it? I'll add a help wanted label at least if that's the case.) Should this remain open?

@isiahmeadows
Copy link

@isiahmeadows isiahmeadows commented Mar 4, 2018

@Trott Based on some of these recent issues/commits linking to here, I suspect it still is an issue. (I've long had to move on to fix async writes, since it was causing tests to fail.)

@silverwind
Copy link
Contributor

@silverwind silverwind commented Jul 16, 2018

Is this still an issue?

Yes, I can confirm 10.6.0 still shows this issue. Simple repro:

$ node -p 'process.stdout.write("x".repeat(5e6)); process.exit()' | wc -c
65536
xzyfer added a commit to sass/node-sass that referenced this issue Jul 19, 2018
jamesdonoh referenced this issue in davisjam/vuln-regex-detector Feb 13, 2019
1. Compute deathSignal correctly
2. Redirect output to tmp file.
   Very long piped strings sometimes get mishandled.
@silverwind silverwind mentioned this issue Jun 20, 2019
3 of 3 tasks complete
michaelgoin pushed a commit to newrelic/node-newrelic that referenced this issue Jan 27, 2020
…ng process.stdout to blocking. See nodejs/node#6456 for a discussion of the issue and a version of the fix used here.

Signed-off-by: mrickard <mrickard@newrelic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.