Prevent stackoverflow caused by recursive deconstruction - #1436
Conversation
|
How is this PR related to #1431? |
|
Wow! I didn't notice there is already an open PR on this. Such an active community! Skimming through the code, the general idea is the same as this PR. Both use heap-allocated stack to unfold the JSON object. I haven't tested it yet though. How should we proceed now? |
|
@nlohmann Sorry for the long delay. While this PR and #1431 are in the same spirit, I believe this PR is more concise and easier to read. Also, it is accompanied by a basic unit test to prevent regression. Regarding the other PR, I don't quite understand the implementation thus I cannot verify its validity. Specifically, I am not sure why there is a |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
stale
Bot
added
the
state: stale
stale
Bot
removed
the
state: stale
|
@nlohmann Yes, this will probably fix #1835. I will check it with bad_json_parsers to make sure. Thanks for reminding me! |
|
I can confirm it passes bad_json_parsers without crashing on any depths (1 to 5,000,000) provided in their test. |
|
Travis CI stalled on one of the jobs (https://travis-ci.org/nlohmann/json/jobs/609598763?utm_medium=notification&utm_source=github_status). Restarting it might help. |
|
Once again, a timeout issue with CI :-). Travis CI terminates This is caused by creating a very deep JSON object in unit-test, that requires a lot of memory allocations hence imposing a heavy burden on valgrind to check for memory leaks. To mitigate this problem, I moved this test to a separate unit-test ( Meanwhile, breaking down |
|
Thanks for taking care about this. I would have taken a more complex approach and would have used the parser's callback mechanism to produce some output to tame Travis. But your approach is much cleaner. Thanks a lot! Sorry for not have merged this issue earlier in January. I have no idea why it slipped through. But I did remember it the moment I saw the stacktrace in #1835. |
🔖 Release itemThis issue/PR will be part of the next release of the library. This template helps preparing the release notes. Type
Description
|
|
No problem at all! I had forgotten about it too, thanks for reminding me :-)
That's interesting, I didn't know that is possible. BTW, there were more considerations on Travis build time which posted in #1836 but I doubt if that qualify for an issue. |
|
There seems to be a problem with the implementation: Calling vector::reserve in a loop can be big pessimization.It effectively disables exponential resizing, and makes everything O(n^2). See a benchmark here: http://quick-bench.com/KT304ZLCCZpN8sd1tVzrGaPkl4A Another question is: an array is cleared after it is inserted to the stack, but not an object. I suppose it will cause only one level of recursion (because values are now moved-from), so maybe it is fine. |
|
Thanks for the feedback.
Very good point! I will look into it. PRs are welcome too :-)
That's correct. I can't remember why the object wasn't explicitly cleared. While it might not make any difference, it's nice to clear it for the sake of consistency at least. |
|
Update: Refer to #1436 (comment) for actual benchmark on object destruction. While the use of Nevertheless, we still might have to get rid of Benchmark results on Release 3.7.1 (recursive destruction): aacdc6b **Benchmark results on Release 3.7.2 (non-recursive destruction): 56109ea ** |
|
Current parsing benchmarks specifically exclude destruction time from measurements. json/benchmarks/src/benchmarks.cpp Lines 22 to 25 in 411158d |
|
I modified the benchmarks to only measure The performance of recursive deconstruction is significantly higher, but I believe avoiding the stack-overflow worth this performance penalty in destruction. non-recursive with non-recursive without recursive: |
|
I made a benchmark that demonstrates the issue. To avoid this being buried in the comments to a closed PR, I created a separate issue #1837 |
This includes the following fixes: nlohmann/json#1436 > For a deeply-nested JSON object, the recursive implementation of json_value::destroy function causes stack overflow. nlohmann/json#1708 nlohmann/json#1722 Stack size nlohmann/json#1693 (comment) Integer Overflow nlohmann/json#1447 UTF8, json dump out of bounds nlohmann/json#1445 Possibly influences #7532
This includes the following fixes: nlohmann/json#1436 > For a deeply-nested JSON object, the recursive implementation of json_value::destroy function causes stack overflow. nlohmann/json#1708 nlohmann/json#1722 Stack size nlohmann/json#1693 (comment) Integer Overflow nlohmann/json#1447 UTF8, json dump out of bounds nlohmann/json#1445 Possibly influences #7532
This includes the following fixes: nlohmann/json#1436 > For a deeply-nested JSON object, the recursive implementation of json_value::destroy function causes stack overflow. nlohmann/json#1708 nlohmann/json#1722 Stack size nlohmann/json#1693 (comment) Integer Overflow nlohmann/json#1447 UTF8, json dump out of bounds nlohmann/json#1445 Possibly influences #7532
Serializing a container serializes its elements, so dump() descended into one call per nesting level. A value nested deeply enough exhausted the call stack and terminated the process with a segmentation fault - no exception, nothing the caller could catch. Parsing such a value works, as the parser is iterative, and so does destroying one, as #1436 made destruction iterative. Bound how far the descent goes rather than take the call stack away from it. The first 128 levels are written by exactly the code that always wrote them, and only below that does dump_iteratively write out what is left, keeping the containers it has entered on an explicit stack. Serializing can therefore no longer exhaust the stack, however deeply a value is nested, while a value nested less deeply than the bound pays only for one comparison per container. Writing every value that way instead measured between 2% and 20% slower - 20% on object-heavy documents - which is why the descent is kept for all but the values that cannot afford it. The bound costs nothing measurable: between -1.4% and +1.2% across compact and pretty output of number, integer, string, object-heavy, wide-object and deeply nested documents. The output is unchanged for every value. Both ways of writing a container emit the separator in front of every element but the first, rather than after every element but the last, which puts exactly one between each pair and none at the end. This fixes #5387 for dump(). The copy constructor is fixed in #5389.
Serializing a container serializes its elements, so dump() descended into one call per nesting level. A value nested deeply enough exhausted the call stack and terminated the process with a segmentation fault - no exception, nothing the caller could catch. Parsing such a value works, as the parser is iterative, and so does destroying one, as #1436 made destruction iterative. Bound how far the descent goes rather than take the call stack away from it. The first 128 levels are written by exactly the code that always wrote them, and only below that does dump_iteratively write out what is left, keeping the containers it has entered on an explicit stack. Serializing can therefore no longer exhaust the stack, however deeply a value is nested, while a value nested less deeply than the bound pays only for one comparison per container. Writing every value that way instead measured between 2% and 20% slower - 20% on object-heavy documents - which is why the descent is kept for all but the values that cannot afford it. The bound costs nothing measurable: between -1.4% and +1.2% across compact and pretty output of number, integer, string, object-heavy, wide-object and deeply nested documents. The output is unchanged for every value. Both ways of writing a container emit the separator in front of every element but the first, rather than after every element but the last, which puts exactly one between each pair and none at the end. This fixes #5387 for dump(). The copy constructor is fixed in #5389. Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Serializing a container serializes its elements, so dump() descended into one call per nesting level. A value nested deeply enough exhausted the call stack and terminated the process with a segmentation fault - no exception, nothing the caller could catch. Parsing such a value works, as the parser is iterative, and so does destroying one, as #1436 made destruction iterative. Bound how far the descent goes rather than take the call stack away from it. The first 128 levels are written by exactly the code that always wrote them, and only below that does dump_iteratively write out what is left, keeping the containers it has entered on an explicit stack. Serializing can therefore no longer exhaust the stack, however deeply a value is nested, while a value nested less deeply than the bound pays only for one comparison per container. Writing every value that way instead measured between 2% and 20% slower - 20% on object-heavy documents - which is why the descent is kept for all but the values that cannot afford it. The bound costs nothing measurable: between -1.4% and +1.2% across compact and pretty output of number, integer, string, object-heavy, wide-object and deeply nested documents. The output is unchanged for every value. Both ways of writing a container emit the separator in front of every element but the first, rather than after every element but the last, which puts exactly one between each pair and none at the end. This fixes #5387 for dump(). The copy constructor is fixed in #5389. Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Serializing a container serializes its elements, so dump() descended into one call per nesting level. A value nested deeply enough exhausted the call stack and terminated the process with a segmentation fault - no exception, nothing the caller could catch. Parsing such a value works, as the parser is iterative, and so does destroying one, as #1436 made destruction iterative. Bound how far the descent goes rather than take the call stack away from it. The first 128 levels are written by exactly the code that always wrote them, and only below that does dump_iteratively write out what is left, keeping the containers it has entered on an explicit stack. Serializing can therefore no longer exhaust the stack, however deeply a value is nested, while a value nested less deeply than the bound pays only for one comparison per container. Writing every value that way instead measured between 2% and 20% slower - 20% on object-heavy documents - which is why the descent is kept for all but the values that cannot afford it. The bound costs nothing measurable: between -1.4% and +1.2% across compact and pretty output of number, integer, string, object-heavy, wide-object and deeply nested documents. The output is unchanged for every value. Both ways of writing a container emit the separator in front of every element but the first, rather than after every element but the last, which puts exactly one between each pair and none at the end. This fixes #5387 for dump(). The copy constructor is fixed in #5389. Signed-off-by: Niels Lohmann <mail@nlohmann.me>
4 participants
This pull request fixes #832, #1419, #1835.
For a deeply-nested JSON object, the recursive implementation of
json_value::destroyfunction causes stack overflow.This is problematic for users that wish to expose a public API since an attacker can crash their systems by sending a deeply nested JSON request (e.g. a request containing 512K bytes of
[).In this pull request:
First, a regression test has been added for this particular issue. We should probably add more regression tests for more complex scenarios.
json_value::destroymethod has been modified to deconstruct children of the JSON value iteratively to prevent triggering a recursive deconstruction. The computational complexity and memory requirement of this approach isO(n)which is equivalent to previous recursive deconstruction. Also,std::movehas been used to minimize the overhead of flattening original JSON object.One question: The iterative deconstruction can be also implemented inside
~basic_json::basic_json. I am not sure which place is more appropriate. I have currently implemented it insidejson_value::destorymerely because I find it easier.Pull request checklist
Read the Contribution Guidelines for detailed information.
include/nlohmanndirectory, runmake amalgamateto create the single-header filesingle_include/nlohmann/json.hpp. The whole process is described here.Please don't
#ifdefs or other means.