Add support for high-precision numbers in UBJSON encoding

[nlohmann]

This PR adds high-precision numbers for UBJSON.

• Entries with a 'H' prefix use the JSON number parser and store the parsed number.
• When serializing, integer values greater than int64max use the 'H' prefix, followed by a JSON text of the number.

Closes #2286.

[nlohmann]

There is a SIGSEGV when executing the tests with MSVC 2019. I cannot reproduce this issue on my machine (macOS), and since there is no issue when running with Valgrind or the sanitizers, I have no idea what to do.

[coveralls]

Coverage remained the same at 100.0% when pulling 3366241 on issue2286 into a048b72 on develop.

[t-b]

It is not 100% reproducible. Worked 5 times, once crashed with

3: Test command: E:\projekte\json\build-msvc2019\test\Debug\test-ubjson.exe "--no-skip"
43: Test timeout computed to be: 1500
43: [doctest] doctest version is "2.3.7"
43: [doctest] run with "--help" for options
43: ===============================================================================
43: E:\projekte\json\test\src\unit-ubjson.cpp(2446):
43: TEST CASE:  UBJSON roundtrips
43:   input from self-generated UBJSON files
43:
43: E:\projekte\json\test\src\unit-ubjson.cpp(2446): FATAL ERROR: test case CRASHED: SIGSEGV - Stack overflow
43:
43: ===============================================================================
43: E:\projekte\json\test\src\unit-ubjson.cpp(2446):
43: TEST CASE:  UBJSON roundtrips
43:
43: ===============================================================================
43: [doctest] test cases:      4 |      3 passed |      1 failed |      0 skipped
43: [doctest] assertions: 691826 | 691826 passed |      0 failed |
43: [doctest] Status: FAILURE!
2/2 Test #43: test-ubjson ......................***Exception: SegFault139.84 sec

If I only run that sub case with ./test/Debug/test-ubjson.exe --subcase="UBJSON roundtrips" I can't break it either.

[nlohmann]

It is not 100% reproducible. Worked 5 times, once crashed with

Can you get a stack trace? I have no idea what even goes wrong...

[fmotty]

I was able to reproduce this issue under the debugger and obtain the stack trace. This is a quite a classical stack exhaustion case due to large functions and recursion. The 'get_ubjson_value' function stack demands grew, and some of the tests started failing in debug mode. The test consumed 1 MiB of stack space and crashed. In non-debug mode, with all the optimizations, the test passes.

Basically, in debug mode, the compiler (not only MSVC compiler, others do the same) does not efficiently merge the stack of various unrelated blocks. Consider this example:
void Foo() {
{ char a[1024]; }
{ char b[1024]; }
}
One would expect that the compiler allocates only ~1 KiB of stack space for Foo, but, in fact, it uses ~2 KiB, placing 'a' and 'b' in separate locations. Multiply it by the number of times the function reenters, and you can see how stack is quickly exhausted.
The stack trace is attached.
stacktrace.txt

In order to verify the theory that the newly added code is overfilling the stack size limit, I did a quick hack and wrapped the new code inside the added "case 'H'" switch branch into the lambda object, and the test passed.

The easiest mitigation would be to extract the new "case 'H'" code into a separate method. This way, the 'get_ubjson_value' method stack demands will shrink, and the test will pass. Alternatively, the test may be altered to provide the larger stack.

[nlohmann]

@fmotty Thanks so much for the detailed analysis!

[t-b]

@fmotty Thanks for chiming in!

[nlohmann]

---

🔖 Release item

This issue/PR will be part of the next release of the library. This template helps preparing the release notes.

Type

• ✨ New Feature
• 🐛 Bug Fix
• ⚡️ Improvement
• 🔨 Further Change
• 🔥 Deprecated function

Description

• Add support for high-precision numbers in UBJSON encoding Add support for high-precision numbers in UBJSON encoding #2286 Add support for high-precision numbers in UBJSON encoding #2297

---