-
Notifications
You must be signed in to change notification settings - Fork 29k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
util: recover from maximum call stack size #20725
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM if it doesn’t affect the benchmarks negatively
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think at the very least this should be a separate option for util.inspect()
. I would not expect to get back something I did not request. It should be all or nothing (meaning either the stack overflow error should occur or an error be thrown -- something to indicate the request could not be completed as asked).
If the user wants to explicitly opt-in to this behavior, that's fine because then they know what they could be getting into at that point.
Here’s a benchmark CI: https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks/191/ But yes, I’m generally +1 on this PR. I don’t see the point of emitting a stack overflow error if we can avoid it at little cost. I don’t think setting depths to these high values makes sense in real-world applications anyway, but if somebody does that, then |
Seems like TypedArrays take a minor hit of up to 4%. The other entries seem to be sough (I reran the benchmarks). I am going to play around with it to see if I am able to get that further down. I am a bit surprised that it has impact on those but not on e.g. nested objects. About having depth levels like these in real world applications: This should indeed be highly unlikely as the output would mainly just be whitespace. I can not imagine that anyone would look at this and I can also not think of a use case where this would be useful programmatically. It would just not be useful anymore. |
I don't believe this should be a factor in deciding whether the PR should land. Agreed with @addaleax on user experience vs. cost. |
I had another look at this and found a solution that does not show the regression.
|
a743cbc
to
630360c
Compare
The benchmarks clearly show no regression (it will only effect very deeply nested objects - if at all). I believe the user experience is better by returning a value instead of throwing. Inspecting any deeper will not provide a result that would still be really useful. @mscdex are you strongly -1 on this? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. I think this will help new users.
I am -0.5 on this. In real-world applications that I've seen issues like this, even if the stack overflow doesn't happen, the process may run out of memory because of the complexity of the object. String concatenation in V8 is memory-expensive (each |
@joyeecheung the max stack size with be exceeded much much before the heap memory is. When I ran a different implementation that was recovering from the max stack size I was able to inspect a depth of almost 100.000. This will end inspection at a depth of ~1500 for the very simple case and earlier for more complex cases. |
IIUC, this ends the inspection early in the sense that it will return the formatted subtree to be concatenated later, it does not mean that the whole inspection process will end once the formatter finds one deep subtree in the object, so more memory will be consumed if the object is not skewed (containing multiple deep subtrees). |
@joyeecheung that is correct but a very unlikely case. To have a object like this will likely cause problems one way or the other. The problem you speak about should be solved with #19994. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm +1 on this as this is a debugging tool. Returning something is better than the stack-overflow.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Just in case we start getting into vote-counting territory for the TSC (since it does seem unlikely that the impasse here will be resolved):
|
(UPDATE: I'll be updating the preliminary vote counts as TSC people weigh in.) Since this is at an impasse that seems unlikely to be resolved, I think calling for a vote at this time is appropriate. The way things stand now, as I see it at least:
There are 18 TSC members total, so this will require participation by at least another 2 TSC members to achieve a resolution. @nodejs/tsc |
@joyeecheung I did consider it but we already have a lot of options for |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This seems like reasonable behavior to me
I am not +1 but I would not block this. We can see how the users react to this in the wild and revert if necessary. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM but I would like Anna's suggestion to be implemented before landing
To break the impasse, we had a vote in the TSC meeting today. This PR can land (assuming green CI). Vote results:
Votes add up to 14 rather than 18 because 4 members were not involved in the vote:
I'll remove the |
Using util.inspect should still return values in case the maximum call stack size is reached. This is important to inspect linked lists and similar.
8214e97
to
7748619
Compare
I just pushed the requested change @addaleax |
Dimissing the review due to the TSC vote.
CI again 2: https://ci.nodejs.org/job/node-test-pull-request/15577/ ✔️ |
Landed in b26506b |
Using util.inspect should still return values in case the maximum call stack size is reached. This is important to inspect linked lists and similar. PR-URL: #20725 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Anatoli Papirovski <apapirovski@mac.com> Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Ali Ijaz Sheikh <ofrobots@google.com> Reviewed-By: Michael Dawson <michael_dawson@ca.ibm.com> Reviewed-By: Daniel Bevenius <daniel.bevenius@gmail.com> Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com> Reviewed-By: Jeremiah Senkpiel <fishrock123@rocketmail.com> Reviewed-By: Michaël Zasso <targos@protonmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>
Using util.inspect should still return values in case the maximum call stack size is reached. This is important to inspect linked lists and similar. PR-URL: #20725 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Anatoli Papirovski <apapirovski@mac.com> Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Ali Ijaz Sheikh <ofrobots@google.com> Reviewed-By: Michael Dawson <michael_dawson@ca.ibm.com> Reviewed-By: Daniel Bevenius <daniel.bevenius@gmail.com> Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com> Reviewed-By: Jeremiah Senkpiel <fishrock123@rocketmail.com> Reviewed-By: Michaël Zasso <targos@protonmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>
Notable changes: * build: * Node.js should now be about 60% faster to startup than the previous version, thanks to the use V8's code cache feature for core modules. [#21405](#21405) * dns: * An experimental promisified version of the dns module is now available. Give it a try with `require('dns').promises`. [#21264](#21264) * fs: * `fs.lchown` has been undeprecated now that libuv supports it. [#21498](#21498) * lib: * `Atomics.wake` is being renamed to `Atomics.notify` in the ECMAScript specification ([reference](tc39/ecma262#1220)). Since Node.js now has experimental support for worker threads, we are being proactive and added a `notify` alias, while emitting a warning if `wake` is used. [#21413](#21413) [#21518](#21518) * n-api: * Add API for asynchronous functions. [#17887](#17887) * util: * `util.inspect` is now able to return a result instead of throwing when the maximum call stack size is exceeded during inspection. [#20725](#20725) * vm: * Add `script.createCachedData()`. This API replaces the `produceCachedData` option of the `Script` constructor that is now deprecated. [#20300](#20300) * worker: * Support for relative paths has been added to the `Worker` constructor. Paths are interpreted relative to the current working directory. [#21407](#21407) PR-URL: #21629
Notable changes: * dns: * An experimental promisified version of the dns module is now available. Give it a try with `require('dns').promises`. [#21264](#21264) * fs: * `fs.lchown` has been undeprecated now that libuv supports it. [#21498](#21498) * lib: * `Atomics.wake` is being renamed to `Atomics.notify` in the ECMAScript specification ([reference](tc39/ecma262#1220)). Since Node.js now has experimental support for worker threads, we are being proactive and added a `notify` alias, while emitting a warning if `wake` is used. [#21413](#21413) [#21518](#21518) * n-api: * Add API for asynchronous functions. [#17887](#17887) * util: * `util.inspect` is now able to return a result instead of throwing when the maximum call stack size is exceeded during inspection. [#20725](#20725) * vm: * Add `script.createCachedData()`. This API replaces the `produceCachedData` option of the `Script` constructor that is now deprecated. [#20300](#20300) * worker: * Support for relative paths has been added to the `Worker` constructor. Paths are interpreted relative to the current working directory. [#21407](#21407) PR-URL: #21629
Notable changes: * dns: * An experimental promisified version of the dns module is now available. Give it a try with `require('dns').promises`. [#21264](#21264) * fs: * `fs.lchown` has been undeprecated now that libuv supports it. [#21498](#21498) * lib: * `Atomics.wake` is being renamed to `Atomics.notify` in the ECMAScript specification ([reference](tc39/ecma262#1220)). Since Node.js now has experimental support for worker threads, we are being proactive and added a `notify` alias, while emitting a warning if `wake` is used. [#21413](#21413) [#21518](#21518) * n-api: * Add API for asynchronous functions. [#17887](#17887) * util: * `util.inspect` is now able to return a result instead of throwing when the maximum call stack size is exceeded during inspection. [#20725](#20725) * vm: * Add `script.createCachedData()`. This API replaces the `produceCachedData` option of the `Script` constructor that is now deprecated. [#20300](#20300) * worker: * Support for relative paths has been added to the `Worker` constructor. Paths are interpreted relative to the current working directory. [#21407](#21407) PR-URL: #21629
Using util.inspect should still return values in case the maximum
call stack size is reached. This is important to inspect linked
lists and similar.
Checklist
make -j4 test
(UNIX), orvcbuild test
(Windows) passes