Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

child_process: change the defaults maxBuffer size #27179

Closed
wants to merge 1 commit into from

Conversation

@koh110
Copy link
Contributor

commented Apr 10, 2019

I changed the defaults for all the maxBuffer sizes to a much larger value.

Refs: #23027

Requested in #23027 (review) by @mcollina

Checklist
  • make -j4 test (UNIX), or vcbuild test (Windows) passes
  • tests and/or benchmarks are included
  • documentation is changed or added
  • commit message follows commit guidelines
@koh110

This comment has been minimized.

Copy link
Contributor Author

commented Apr 10, 2019

@sam-github sam-github requested a review from mcollina Apr 10, 2019

@sam-github

This comment has been minimized.

Copy link
Member

commented Apr 10, 2019

Strikes me as an enormously large value, but I don't have a good feel for what value is reasonable. This could lead to sharply increased memory usage for some use cases... but then, if people care, the maxBuffer option should be used to set limits appropriate to the use.

@bnoordhuis
Copy link
Member

left a comment

See #9829 and #11196, notably #9829 (comment) - it's been discussed before.

Changing the default isn't completely out of the question but since it's intended as a containment mechanism for runaway processes, increasing it 320x from 200k to 64M seems like a tremendously bad idea.

@mcollina

This comment has been minimized.

Copy link
Member

commented Apr 11, 2019

Changing the default isn't completely out of the question but since it's intended as a containment mechanism for runaway processes, increasing it 320x from 200k to 64M seems like a tremendously bad idea.

We had Infinity before. This is smaller. From a user point of view, if we are executing a binary and it suddenly got output truncated because of a Node.js update, it would be a bad user experience.

@koh110

This comment has been minimized.

Copy link
Contributor Author

commented Apr 11, 2019

The option was set to Infinity only for spawnSync. This PR changed option of exec too. It is bad point in this PR. But I think that exec and spawnSync should be set same value.

I found this test. How about 1000 * 1024?
https://github.com/nodejs/node/blob/master/test/pummel/test-child-process-spawn-loop.js

I think changing Infinity to 200 * 1024 make a major impact.
Would I check value that CITGM passed?

@sam-github

This comment has been minimized.

Copy link
Member

commented Apr 11, 2019

I doubt that citgm would find these things, its likely going to only show up at load (not that more testing is bad).

But I think that exec and spawnSync should be set same value.

Definitely.

I like the 1 meg suggestion (#9829 (comment)).

Its really hard to get these numbers right, I'd also be OK with no limit, or a child_process.DEFAULT_MAX_BUFFER that was writable.

@sam-github

This comment has been minimized.

Copy link
Member

commented Apr 11, 2019

To follow on, 1meg is only 5 times larger, which seems to deal with output that is larger than a normal hand-created text file, more log-file size. Bigger enough it may cover some more use cases, but not so big its useless as protection against a run-away process, and not so controversial as removing all limits, and letting the only limit be node's ability to allocate memory (aka "infininite").

@koh110

This comment has been minimized.

Copy link
Contributor Author

commented Apr 11, 2019

I like this suggestion too. I think It is suitable.

I like the 1 meg suggestion

@bnoordhuis How about this idea?

maxBuffer should not be setted Infinity for run-away process. But I think it is necessary to set maxBuffer to a higher value, because changing value of spawnSync give big impact.

@bnoordhuis

This comment has been minimized.

Copy link
Member

commented Apr 13, 2019

1M seems like a good middle ground.

@koh110 koh110 force-pushed the koh110:change_default branch from 1292e4a to 40c28d9 Apr 15, 2019

@koh110

This comment has been minimized.

Copy link
Contributor Author

commented Apr 15, 2019

@bnoordhuis @sam-github @mcollina I changed 64 * 1024 * 1024 to 1024 * 1024. Would you check it?

@BridgeAR

This comment has been minimized.

Copy link
Member

commented Apr 16, 2019

Changing a default should likely be semver-major.

@sam-github

This comment has been minimized.

Copy link
Member

commented Apr 16, 2019

Changing a default should likely be semver-major.

I'm on the fence about that. Its increasing the maxBuffer, is it a breaking change that some apps will stop failing? But then, it will use more memory. I guess there is no hurry to land in 12.x, it'll be in 13.x in a month.

@koh110

This comment has been minimized.

Copy link
Contributor Author

commented Apr 16, 2019

I think it should be landed in 12.x. Because, it will have an impact on developers who are using over 200 * 1024.
I think that we should decrease impact with this version up.

@bnoordhuis
Copy link
Member

left a comment

LGTM. Semver-major seems appropriate.

@sam-github

This comment has been minimized.

Copy link
Member

commented Apr 16, 2019

@BethGriggs @nodejs/tsc Should this land in 12.x? Its not a complex change, so I think its low risk of instability, even though it is late in 12.x rc cycle.

@nodejs-github-bot

This comment has been minimized.

@BethGriggs

This comment has been minimized.

Copy link
Member

commented Apr 16, 2019

@sam-github, @mhdawson suggested that this should land with #23027 (which is not in v12.x yet). I'm +1 on them both going into 12.x.

@sam-github

This comment has been minimized.

Copy link
Member

commented Apr 17, 2019

@mcollina @nodejs/tsc: this requires 2 TSC approvals, it currently has 0

@targos

targos approved these changes Apr 17, 2019

@mhdawson
Copy link
Member

left a comment

LGTM

@mhdawson

This comment has been minimized.

Copy link
Member

commented Apr 17, 2019

@sam-github there are 2 now.

@sam-github

This comment has been minimized.

Copy link
Member

commented Apr 17, 2019

Landed in 652877e

@sam-github sam-github closed this Apr 17, 2019

sam-github added a commit that referenced this pull request Apr 17, 2019

child_process: change the defaults maxBuffer size
PR-URL: #27179
Refs: #23027
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: Sam Roberts <vieuxtech@gmail.com>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Reviewed-By: Beth Griggs <Bethany.Griggs@uk.ibm.com>
Reviewed-By: Michael Dawson <michael_dawson@ca.ibm.com>

@BethGriggs BethGriggs referenced this pull request Apr 18, 2019

Merged

v12.0.0 proposal #26930

BethGriggs added a commit that referenced this pull request Apr 22, 2019

2019-04-23, Version 12.0.0 (Current)
Notable changes:

* assert:
  * improve performance to instantiate errors (Ruben Bridgewater)
    [#26738](#26738)
  * validate required arguments (Ruben Bridgewater)
    [#26641](#26641)
  * adjust loose assertions (Ruben Bridgewater)
    [#25008](#25008)
* async_hooks:
  * remove deprecated emitBefore and emitAfter (Matteo Collina)
    [#26530](#26530)
  * remove promise object from resource (Andreas Madsen)
    [#23443](#23443)
* bootstrap
  * make Buffer and process non-enumerable (Ruben Bridgewater)
    [#24874](#24874)
* buffer:
  * use stricter range checks (Ruben Bridgewater)
    [#27045](#27045)
  * harden SlowBuffer creation (ZYSzys)
    [#26272](#26272)
  * harden validation of buffer allocation size (ZYSzys)
    [#26162](#26162)
  * do proper error propagation in addon methods (Anna Henningsen)
    [#23939](#23939)
* child_process:
  * change the defaults maxBuffer size (kohta ito)
    [#27179](#27179)
  * harden fork arguments validation (ZYSzys)
    [#27039](#27039)
  * use non-infinite maxBuffer defaults (kohta ito)
    [#23027](#23027)
* console:
  * don't use ANSI escape codes when TERM=dumb (Vladislav Kaminsky)
    [#26261](#26261)
* crypto:
  * remove legacy native handles (Tobias Nießen)
    [#27011](#27011)
  * decode missing passphrase errors (Tobias Nießen)
    [#25208](#25208)
  * move DEP0113 to End-of-Life (Tobias Nießen)
    [#26249](#26249)
  * remove deprecated crypto.\_toBuf (Tobias Nießen)
    [#25338](#25338)
  * set `DEFAULT\_ENCODING` property to non-enumerable
    (Antoine du Hamel)
    [#23222](#23222)
* deps:
  * silence irrelevant V8 warning (Michaël Zasso)
    [#26685](#26685)
  * update postmortem metadata generation script (cjihrig)
    [#26685](#26685)
  * V8: un-cherry-pick bd019bd (Refael Ackermann)
    [#26685](#26685)
  * V8: cherry-pick 6 commits (Michaël Zasso)
    [#26685](#26685)
  * V8: cherry-pick d82c9af (Anna Henningsen)
    [#26685](#26685)
  * V8: cherry-pick e5f01ba (Anna Henningsen)
    [#26685](#26685)
  * V8: cherry-pick d5f08e4 (Anna Henningsen)
    [#26685](#26685)
  * V8: cherry-pick 6b09d21 (Anna Henningsen)
    [#26685](#26685)
  * V8: cherry-pick f0bb5d2 (Anna Henningsen)
    [#26685](#26685)
  * V8: cherry-pick 5b0510d (Anna Henningsen)
    [#26685](#26685)
  * V8: cherry-pick 91f0cd0 (Anna Henningsen)
    [#26685](#26685)
  * V8: cherry-pick 392316d (Anna Henningsen)
    [#26685](#26685)
  * V8: cherry-pick 2f79d68 (Anna Henningsen)
    [#26685](#26685)
  * sync V8 gypfiles with 7.4 (Ujjwal Sharma)
    [#26685](#26685)
  * update V8 to 7.4.288.13 (Ujjwal Sharma)
    [#26685](#26685)
  * bump minimum icu version to 63 (Ujjwal Sharma)
    [#25852](#25852)
  * silence irrelevant V8 warnings (Michaël Zasso)
    [#25852](#25852)
  * V8: cherry-pick 7803fa6 (Jon Kunkee)
    [#25852](#25852)
  * V8: cherry-pick 58cefed (Jon Kunkee)
    [#25852](#25852)
  * V8: cherry-pick d3308d0 (Michaël Zasso)
    [#25852](#25852)
  * V8: cherry-pick 74571c8 (Michaël Zasso)
    [#25852](#25852)
  * cherry-pick fc0ddf5 from upstream V8 (Anna Henningsen)
    [#25852](#25852)
  * sync V8 gypfiles with 7.3 (Ujjwal Sharma)
    [#25852](#25852)
  * sync V8 gypfiles with 7.2 (Michaël Zasso)
    [#25852](#25852)
  * update V8 to 7.3.492.25 (Michaël Zasso)
    [#25852](#25852)
  * add s390 asm rules for OpenSSL-1.1.1 (Shigeki Ohtsu)
    [#19794](#19794)
  * sync V8 gypfiles with 7.1 (Refael Ackermann)
    [#23423](#23423)
  * update V8 to 7.1.302.28 (Michaël Zasso)
    [#23423](#23423)
* doc:
  * update behaviour of fs.writeFile
    (Sakthipriyan Vairamani (thefourtheye))
    [#25080](#25080)
  * add internal functionality details of util.inherits
    (Ruben Bridgewater)
    [#24755](#24755)
* errors:
  * update error name (Ruben Bridgewater)
    [#26738](#26738)
* fs:
  * use proper .destroy() implementation for SyncWriteStream
    (Matteo Collina)
    [#26690](#26690)
  * improve mode validation (Ruben Bridgewater)
    [#26575](#26575)
  * harden validation of start option in createWriteStream (ZYSzys)
    [#25579](#25579)
  * make writeFile consistent with readFile wrt fd
    (Sakthipriyan Vairamani (thefourtheye))
    [#23709](#23709)
* http:
  * validate timeout in ClientRequest() (cjihrig)
    [#26214](#26214)
  * return HTTP 431 on HPE\_HEADER\_OVERFLOW error (Albert Still)
    [#25605](#25605)
  * switch default parser to llhttp (Anna Henningsen)
    [#24870](#24870)
  * change DEP0066 to a runtime deprecation (Morgan Roderick)
    [#24167](#24167)
  * else case is not reachable (szabolcsit)
    [#24176](#24176)
* lib:
  * move DEP0021 to end of life (cjihrig)
    [#27127](#27127)
  * remove Atomics.wake (Gus Caplan)
    [#27033](#27033)
  * validate Error.captureStackTrace() calls (Ruben Bridgewater)
    [#26738](#26738)
  * refactor Error.captureStackTrace() usage (Ruben Bridgewater)
    [#26738](#26738)
  * move DTRACE\_\* probes out of global scope (James M Snell)
    [#26541](#26541)
  * deprecate \_stream\_wrap (Sam Roberts)  [#26245]
  (#26245)
  * don't use `util.inspect()` internals (Ruben Bridgewater)
    [#24971](#24971)
  * improve error message for MODULE\_NOT\_FOUND (Ali Ijaz Sheikh)
    [#25690](#25690)
  * requireStack property for MODULE\_NOT\_FOUND (Ali Ijaz Sheikh)
    [#25690](#25690)
  * move DEP0029 to end of life (cjihrig)
    [#25377](#25377)
  * move DEP0028 to end of life (cjihrig)
    [#25377](#25377)
  * move DEP0027 to end of life (cjihrig)
    [#25377](#25377)
  * move DEP0026 to end of life (cjihrig)
    [#25377](#25377)
  * move DEP0023 to end of life (cjihrig)
    [#25280](#25280)
  * move DEP0006 to end of life (cjihrig)
    [#25279](#25279)
  * remove unintended access to deps/ (Anna Henningsen)
    [#25138](#25138)
  * move DEP0120 to end of life (cjihrig)
    [#24862](#24862)
  * use ES6 class inheritance style (Ruben Bridgewater)
    [#24755](#24755)
  * remove `inherits()` usage (Ruben Bridgewater)
    [#24755](#24755)
* module:
  * remove dead code (Ruben Bridgewater)
    [#26983](#26983)
  * mark DEP0019 as End-of-Life (Ruben Bridgewater)
    [#26973](#26973)
  * throw an error for invalid package.json main entries
    (Ruben Bridgewater)
    [#26823](#26823)
  * don't search in require.resolve.paths (cjihrig)
    [#23683](#23683)
* n-api:
  * remove code from error name (Ruben Bridgewater)
    [#26738](#26738)
* net:
  * do not manipulate potential user code (Ruben Bridgewater)
    [#26751](#26751)
  * emit "write after end" errors in the next tick (Ouyang Yadong)
    [#24457](#24457)
  * deprecate \_setSimultaneousAccepts() undocumented function
    (James M Snell)
    [#23760](#23760)
* net,http2:
  * merge setTimeout code (ZYSzys)
    [#25084](#25084)
* os:
  * implement os.type() using uv\_os\_uname() (cjihrig)
    [#25659](#25659)
* process:
  * global.process, global.Buffer getters (Guy Bedford)
    [#26882](#26882)
  * move DEP0062 (node --debug) to end-of-life (Joyee Cheung)
    [#25828](#25828)
  * exit on --debug and --debug-brk after option parsing (Joyee Cheung)
    [#25828](#25828)
  * improve `--redirect-warnings` handling (Ruben Bridgewater)
    [#24965](#24965)
* readline:
  * support TERM=dumb (Vladislav Kaminsky)
    [#26261](#26261)
* repl:
  * add welcome message (gengjiawen)
    [#25947](#25947)
  * fix terminal default setting (Ruben Bridgewater)
    [#26518](#26518)
  * check colors with .getColorDepth() (Vladislav Kaminsky)
    [#26261](#26261)
  * deprecate REPLServer.rli (Ruben Bridgewater)
    [#26260](#26260)
* src:
  * remove unused INT\_MAX constant (Sam Roberts)
    [#27078](#27078)
  * update NODE\_MODULE\_VERSION to 72 (Ujjwal Sharma)
    [#26685](#26685)
  * remove `AddPromiseHook()` (Anna Henningsen)
    [#26574](#26574)
  * update NODE\_MODULE\_VERSION to 71 (Michaël Zasso)
    [#25852](#25852)
  * clean up MultiIsolatePlatform interface (Anna Henningsen)
    [#26384](#26384)
  * properly configure default heap limits (Ali Ijaz Sheikh)
    [#25576](#25576)
  * remove icuDataDir from node config (GauthamBanasandra)
    [#24780](#24780)
  * explicitly allow JS in ReadHostObject (Yang Guo)
    [#23423](#23423)
  * update postmortem constant (cjihrig)
    [#23423](#23423)
  * update NODE\_MODULE\_VERSION to 68 (Michaël Zasso)
    [#23423](#23423)
* tls:
  * support TLSv1.3 (Sam Roberts)
    [#26209](#26209)
  * return correct version from getCipher() (Sam Roberts)
    [#26625](#26625)
  * check arg types of renegotiate() (Sam Roberts)
    [#25876](#25876)
  * add code for ERR\_TLS\_INVALID\_PROTOCOL\_METHOD (Sam Roberts)
    [#24729](#24729)
  * emit a warning when servername is an IP address (Rodger Combs)
    [#23329](#23329)
  * disable TLS v1.0 and v1.1 by default (Ben Noordhuis)
    [#23814](#23814)
  * remove unused arg to createSecureContext() (Sam Roberts)
    [#24241](#24241)
  * deprecate Server.prototype.setOptions() (cjihrig)[
    #23820](#23820)
  * load NODE\_EXTRA\_CA\_CERTS at startup (Ouyang Yadong)
    [#23354](#23354)
* util:
  * change inspect compact and breakLength default (Ruben Bridgewater)
    [#27109](#27109)
  * improve inspect edge cases (Ruben Bridgewater)
    [#27109](#27109)
  * only the first line of the error message (Simon Zünd)
    [#26685](#26685)
  * don't set the prototype of callbackified functions
    (Ruben Bridgewater)
    [#26893](#26893)
  * rename callbackified function (Ruben Bridgewater)
    [#26893](#26893)
  * increase function length when using `callbackify()`
    (Ruben Bridgewater)
    [#26893](#26893)
  * prevent tampering with internals in `inspect()` (Ruben Bridgewater)
    [#26577](#26577)
  * fix proxy inspection (Ruben Bridgewater)
    [#26241](#26241)
  * prevent leaking internal properties (Ruben Bridgewater)
    [#24971](#24971)
  * protect against monkeypatched Object prototype for inspect()
    (Rich Trott)
    [#25953](#25953)
  * treat format arguments equally (Roman Reiss)
    [#23162](#23162)
* win, fs:
  * detect if symlink target is a directory (Bartosz Sosnowski)
    [#23724](#23724)
* zlib:
  * throw TypeError if callback is missing (Anna Henningsen)[
    #24929](#24929)
  * make “bare” constants un-enumerable (Anna Henningsen)
    [#24824](#24824)

PR-URL: #26930
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.