-
-
Notifications
You must be signed in to change notification settings - Fork 6.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Prevent stackoverflow caused by recursive deconstruction #1436
Conversation
How is this PR related to #1431? |
Wow! I didn't notice there is already an open PR on this. Such an active community! Skimming through the code, the general idea is the same as this PR. Both use heap-allocated stack to unfold the JSON object. I haven't tested it yet though. How should we proceed now? |
@nlohmann Sorry for the long delay. While this PR and #1431 are in the same spirit, I believe this PR is more concise and easier to read. Also, it is accompanied by a basic unit test to prevent regression. Regarding the other PR, I don't quite understand the implementation thus I cannot verify its validity. Specifically, I am not sure why there is a |
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
@nlohmann Yes, this will probably fix #1835. I will check it with bad_json_parsers to make sure. Thanks for reminding me! |
I can confirm it passes bad_json_parsers without crashing on any depths (1 to 5,000,000) provided in their test. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me.
Travis CI stalled on one of the jobs (https://travis-ci.org/nlohmann/json/jobs/609598763?utm_medium=notification&utm_source=github_status). Restarting it might help. |
Once again, a timeout issue with CI :-). Travis CI terminates This is caused by creating a very deep JSON object in unit-test, that requires a lot of memory allocations hence imposing a heavy burden on valgrind to check for memory leaks. To mitigate this problem, I moved this test to a separate unit-test ( Meanwhile, breaking down |
Thanks for taking care about this. I would have taken a more complex approach and would have used the parser's callback mechanism to produce some output to tame Travis. But your approach is much cleaner. Thanks a lot! Sorry for not have merged this issue earlier in January. I have no idea why it slipped through. But I did remember it the moment I saw the stacktrace in #1835. |
🔖 Release itemThis issue/PR will be part of the next release of the library. This template helps preparing the release notes. Type
Description
|
No problem at all! I had forgotten about it too, thanks for reminding me :-)
That's interesting, I didn't know that is possible. BTW, there were more considerations on Travis build time which posted in #1836 but I doubt if that qualify for an issue. |
There seems to be a problem with the implementation: Calling vector::reserve in a loop can be big pessimization.It effectively disables exponential resizing, and makes everything O(n^2). See a benchmark here: http://quick-bench.com/KT304ZLCCZpN8sd1tVzrGaPkl4A Another question is: an array is cleared after it is inserted to the stack, but not an object. I suppose it will cause only one level of recursion (because values are now moved-from), so maybe it is fine. |
Thanks for the feedback.
Very good point! I will look into it. PRs are welcome too :-)
That's correct. I can't remember why the object wasn't explicitly cleared. While it might not make any difference, it's nice to clear it for the sake of consistency at least. |
Update: Refer to #1436 (comment) for actual benchmark on object destruction. While the use of Nevertheless, we still might have to get rid of Benchmark results on Release 3.7.1 (recursive destruction): aacdc6b
**Benchmark results on Release 3.7.2 (non-recursive destruction): 56109ea **
|
Current parsing benchmarks specifically exclude destruction time from measurements. json/benchmarks/src/benchmarks.cpp Lines 22 to 25 in 411158d
|
I modified the benchmarks to only measure The performance of recursive deconstruction is significantly higher, but I believe avoiding the stack-overflow worth this performance penalty in destruction. non-recursive with
non-recursive without
recursive:
|
I made a benchmark that demonstrates the issue. To avoid this being buried in the comments to a closed PR, I created a separate issue #1837 |
This includes the following fixes: nlohmann/json#1436 > For a deeply-nested JSON object, the recursive implementation of json_value::destroy function causes stack overflow. nlohmann/json#1708 nlohmann/json#1722 Stack size nlohmann/json#1693 (comment) Integer Overflow nlohmann/json#1447 UTF8, json dump out of bounds nlohmann/json#1445 Possibly influences #7532
This includes the following fixes: nlohmann/json#1436 > For a deeply-nested JSON object, the recursive implementation of json_value::destroy function causes stack overflow. nlohmann/json#1708 nlohmann/json#1722 Stack size nlohmann/json#1693 (comment) Integer Overflow nlohmann/json#1447 UTF8, json dump out of bounds nlohmann/json#1445 Possibly influences #7532
This includes the following fixes: nlohmann/json#1436 > For a deeply-nested JSON object, the recursive implementation of json_value::destroy function causes stack overflow. nlohmann/json#1708 nlohmann/json#1722 Stack size nlohmann/json#1693 (comment) Integer Overflow nlohmann/json#1447 UTF8, json dump out of bounds nlohmann/json#1445 Possibly influences #7532
This pull request fixes #832, #1419, #1835.
For a deeply-nested JSON object, the recursive implementation of
json_value::destroy
function causes stack overflow.This is problematic for users that wish to expose a public API since an attacker can crash their systems by sending a deeply nested JSON request (e.g. a request containing 512K bytes of
[
).In this pull request:
First, a regression test has been added for this particular issue. We should probably add more regression tests for more complex scenarios.
json_value::destroy
method has been modified to deconstruct children of the JSON value iteratively to prevent triggering a recursive deconstruction. The computational complexity and memory requirement of this approach isO(n)
which is equivalent to previous recursive deconstruction. Also,std::move
has been used to minimize the overhead of flattening original JSON object.One question: The iterative deconstruction can be also implemented inside
~basic_json::basic_json
. I am not sure which place is more appropriate. I have currently implemented it insidejson_value::destory
merely because I find it easier.Pull request checklist
Read the Contribution Guidelines for detailed information.
include/nlohmann
directory, runmake amalgamate
to create the single-header filesingle_include/nlohmann/json.hpp
. The whole process is described here.Please don't
#ifdef
s or other means.