Test JSON Libraries

A simple project to test JSON libraries with Delphi and C++Builder.

Disclaimer

This is a simple project and not a final product. I know that there are many points that could be better structured. The initial goal was to quickly deploy a single unit "FoMain" kind of project, spending time to include and test the most popular JSON libraries for Delphi. The Delphi version has a simple structure with TLib and TTest classes. Also, it is important to mention that the size of JSON objects used in tests here doesn't represent the most common cases of data manipulation in JSON. Finally, the analysis of the results is not intended to detract from any of the tested libraries and should not be seen as any kind of criticism of the authors of the libraries. Knowledge is freedom.

Tested libraries

Library	Delphi	C++Builder
McJSON	✓	✓
uLkJSON	✓	✓
System.JSON	✓	✓
JsonDataObjects	✓	✓
SuperObject	✓
X-SuperObject	✓
JsonTools	✓	✓
Json4Delphi	✓	✓
Grijjy.Bson	✓
Neslib.Json	✓
dwsJSON	✓
chimera.json	✓
DynamicDataObjects	✓
EasyJson	✓
jsonDoc	✓

Note: in order of inclusion.

Compilers and computer

• Delphi VCL 32-bits project built with version 12.1 (Athens Community Edition).
• C++Builder VCL 32-bits project built with version 10.2 (Tokyo).
• Old 64-bits machine: AMD A12-9700P RADEON R7, 10 COMPUTE CORES 4C+6G - 2.50 GHz - 12 GB RAM.

Tests

There are three test types:

• Speed Run
  • Generate N items { ... "key_i":"value_i", ... }
  • Save to file
  • Clear
  • Load from file
  • Find M items
  • Clone/parse
• Validation
• Open File

Procedure

For each library and test type the app was ran and closed. The results of Speed Run compute the average time of 5 repetitions. The Memory consuption is logged and will be part of some results.

Speed Run test

This is a performance test with the following configuration (select Default in the Preset list).

• N = 50k items.
• M = 1k items (keep same random sequence for all tests).
• 5 repetitions.
• Memory collumn is the app memory consuption after the Genereate sub-test.
• It is expected that the Clone/Parse sub-test consumes twice as memory as the Generate or Load from file sub-test.
• See images as confirmation.

Results with Delphi

Library	Generate	Save	Load	Find	Parse	[Total]	Memory
Neslib.Json	.03 s	.03 s	.03 s	.00 s	.04 s	0.18 s	10.19 MiB
Grijjy.Bson	.03 s	.04 s	.04 s	.00 s	.06 s	0.23 s	7.52 MiB
chimera.json	.04 s	.03 s	.08 s	.01 s	.08 s	0.30 s	8.78 MiB
McJSON	.02 s	.06 s	.02 s	.18 s	.08 s	0.41 s	9.85 MiB
System.JSON	.02 s	.01 s	.06 s	.22 s	.06 s	0.43 s	11.38 MiB
EasyJson	.02 s	.01 s	.06 s	.23 s	.07 s	0.45 s	11.38 MiB
LkJson	.06 s	.05 s	.10 s	.01 s	.14 s	0.45 s	2.99 MiB
JsonDoc	.30 s	.13 s	.16 s	.01 s	.46 s	1.13 s	6.14 MiB
SuperObject	.13 s	1.21 s	.05 s	.00 s	.06 s	1.53 s	9.68 MiB
dwsJSON	.01 s	.01 s	0.92 s	.03 s	0.91 s	1.92 s	9.88 MiB
JsonDataObjects	5.61 s	.01 s	.09 s	.11 s	.10 s	5.97 s	8.98 MiB
JsonTools	10.36 s	-	-	.22 s	8.55 s	19.18 s	7.88 MiB
Json4Delphi	.02 s	.06 s	35.00 s	.40 s	35.71 s	71.23 s	11.57 MiB
DynamicDataObjects	28.73 s	.02 s	30.24 s	.58 s	29.25 s	88.88 s	11.51 MiB
X-SuperObject	5.18 min	.06 s	1.77 min	6.18 s	1.76 min	8.81 min	11.48 MiB

Notes:

• See Conclusions about the EasyJson and System.JSON.
• See Know issues about the incomplete test for JsonTools.

Results with C++Builder

Library	Generate	Save	Load	Find	Parse	Total	Memory
McJSON	.09 s	.08 s	.04 s	.39 s	.10 s	0.74 s	5.09 MiB
LkJson	.19 s	.08 s	.16 s	.01 s	.23 s	0.74 s	2.88 MiB
System.JSON	.12 s	.08 s	.24 s	2.94 s	.32 s	3.78 s	9.39 MiB
JsonDataObjects	15.23 s	.02 s	.17 s	.30 s	.19 s	15.97 s	4.30 MiB
JsonTools	24.41 s	-	-	.46 s	23.50 s	48.39 s	5.78 MiB
Json4Delphi	.10 s	.11 s	63.94 s	.93 s	64.03 s	129.17 s	6.81 MiB

Notes:

• See Know issues about the incomplete test for JsonTools.

Scaling test

Here are some results just for the Generate sub-test increasing N using C++Builder project.

Library	1k	5k	10k	25k	50k
McJSON	.01 s	.02 s	.03 s	.05 s	.10 s
LkJson	.01 s	.02 s	.04 s	.10 s	.17 s
System.JSON	.01 s	.09 s	.04 s	.07 s	.12 s
JsonDataObjects	.01 s	.08 s	.26 s	2.76 s	15.55 s
JsonTools	.02 s	.12 s	.43 s	5.51 s	23.87 s
Json4Delphi	.01 s	.02 s	.03 s	.06 s	.11 s

Validation test

This validation test should be analyzed carefully. Some libraries have violations for some sort of self-management in reading JSON data.

• .\test\valid files extracted from MJPA/SimpleJSON
• These are not valid JSON files because first line has a text as description.

Results with Delphi

Library	Expected to Fail but Passed	Expected to Pass but Failed
McJSON	-	-
LkJson	fail(01, 07, 08, 16, 18, 19, 20, 21)	-
System.JSON	fail(07)	pass(04)
JsonDataObjects	fail(01, 05, 08, 15, 18, 19)	pass(04, 05)
SuperObject	fail(01, 06, 07, 08, 10, 11, 16, 18, 19, 20, 21)	-
X-SuperObject	fail(01, 06, 08, 15, 16, 17, 18, 19, 20, 21)	pass(01, 04, 05)
JsonTools	fail(01, 16, 20, 21)	pass(04, 05)
Json4Delphi	-	pass(01, 03, 04, 05, 06)
Grijjy.Bson	fail(15, 20)	pass(01, 02, 04, 05)
Neslib.Json	fail(07, 15, 16, 18, 19, 20, 21)	pass(04, 05)
dwsJSON	fail(16, 18, 19, 21)	-
chimera.json	fail(01, 08, 10, 16, 18, 19, 20, 21, 23)	-
DynamicDataObjects	fail(05, 06, 07, 09, 16, 17, 18, 19, 20, 21)	pass(01, 05)
EasyJson	fail(07)	pass(04, 05)
JsonDoc	fail(15, 20, 21)	pass(01, 02, 04, 05)

Results with C++Builder1

Library	Expected to Fail but Passed	Expected to Pass but Failed
McJSON	-	-
LkJson	fail(01, 07, 08, 16, 18, 19, 20, 21)	-
System.JSON	fail(07)	pass(04)
JsonDataObjects	fail(01, 05, 08, 15, 18, 19)	pass(04, 05)
JsonTools	fail(01, 16, 20, 21)	pass(04, 05)
Json4Delphi	-	pass(01, 03, 04, 05, 06)

List of test files names and description

• fail01.json = \x is not a valid escape character
• fail02.json = Objects require colon between name/value1111111
• fail03.json = Objects do not have comma separators
• fail04.json = Arrays don't have colon separators
• fail05.json = Truth is not a valid boolean value
• fail06.json = Strings need double quotes, not single quotes
• fail07.json = Line break in a string value is not valid
• fail08.json = Escaped line break char is still not valid
• fail09.json = Unclosed array
• fail10.json = Numbers require exponent if 'e' is there
• fail11.json = Only 1 sign char can precede the value
• fail12.json = Commas cannot close objects
• fail13.json = Brackets must be matching
• fail14.json = Double quotes must be escaped
• fail15.json = Key string must be quoted
• fail16.json = Arrays must not have comma after last value
• fail17.json = Arrays must have values between commas
• fail18.json = Nothing but whitespace can follow the root value
• fail19.json = Each opening bracket must be closed
• fail20.json = Extra comma after object
• fail21.json = Numbers cannot have leading 0s
• fail22.json = Numbers can't be hex encoded
• fail23.json = Decimal numbers need a digit before the dot
• pass01.json = General large array testing valid values
• pass02.json = Heavily nested array
• pass03.json = Nested object
• pass04.json = Simple string value
• pass05.json = Unicode character string
• pass06.json = From https://json.org/example.html

Open File test

This is a simple test to open files with any library included into this project using a large JSON files (~ 1 MiB in size).

• Opens file with Verbose unchecked.
• Statistics for memory allocated and loading time.
• Opens file with Verbose checked.
• Checks JSON was read and processed by ToString method.
• Checks memory leaking when closing the app.

Results with Delphi

Library	Memory	Factor	Load Time	Pitfalls
JsonDataObjects	2032.2	2x	6.40	Leaks memory
JsonDoc	1672.9	2x	59.40
Grijjy.Bson	4313.1	4x	25.20	Leaks memory
dwsJSON	4776.2	5x	15.40
Neslib.Json	4676.6	5x	21.80
LkJson	6291.7	6x	72.00
SuperObject	7180.1	7x	68.60
DynamicDataObjects	7529.0	8x	28.40	ToString fails
System.JSON	9420.3	9x	40.60
EasyJson	9420.3	9x	50.20
McJSON	10385.7	10x	46.80
X-SuperObject	10388.4	10x	121.80	ToString fails
Json4Delphi	9505.3	10x	243.60
chimera.json	12086.7	12x	915.60
JsonTools	-	-	-	Load file fails

Notes:

• Ordered by: Factor, Load Time ASC.
• Memory in kiB.
• Allocation Factor = Round(Memory in kiB / 1000 kiB).
• Load time in milliseconds.

The Top-Three libraries with the lowest memory consumption are: JsonDoc, JsonDataObjects and Grijjy.Bson. These last two presented problems of Memory Leaking, but the TLib descendant classes use simple Create, LoadFromFile/Parse and Free methods.

The Top-Three fastest libraries in terms of loading time are: JsonDataObjects, dwsJSON and Neslib.Json. This last one keeps consistent with Speed tests results.

In general, JsonDoc might be considered the best library in respect of memory consumption and loading time tradeoff, highlighting its remarkable low memory and allocation factor without pitfalls. Also, JsonDataObjects has a good chance of keeping the top spot once the cause of the memory leak is found.

Conclusions

1. All Speed Run tests with Delphi version are twice as fast as with C++Builder.

2. For JSON structures with less than 5000 objects, the choice of libraries can be screened not only based on performance. Standard/Compiler compatibility and ease of use should have priority in terms of choice criterion.

3. Considering JSON files with a larger amount of data, there are significant differences in terms of memory consumption and loading time. The positive highlight is JsonDoc.

4. Neslib.Json is the fastest library tested until now, closely followed Grijjy.Bson.

5. LkJson has great performance and the lowest memory consumption among all tested libraries. Some changes are needed to use it with Delphi and C++Builder 10.2 in order to save and load UTF-8 encoded files. For some, an obstacle can be that their interfaces are more verbose for C++ usage. For example:

JsonP = dynamic_cast<TlkJSONObject*>(TlkJSON::ParseText(TlkJSON::GenerateText(Json)))

6. The Validation tests can demonstrate that even the most modern libraries can have occasional small violations against the standard.

7. The Open File tests using a file slightly larger than most common JSON uses demonstrate significant variation in terms of loading time and allocation factor.

8. For older versions of Delphi and C++Builder, the McJSON library can be a good choice in terms of compatibility, ease of use and good performance.

9. This project demonstrates some of the facilities and obstacles encountered by C++Builder developers in using libraries developed for Delphi.

10. EasyJson uses System.JSON internally, so it is expected to see similar results.

Know issues

• TgoBsonDocument.LoadFromJsonFile() failed.
• SuperObject compiles but it is not working with C++Builder. Any help getting SuperObject working with C++Builder is appreciated.
• JsonTools had problems saving to file: it was truncated at object "key25412". So, Speed Run tests were ran without the Clear and Load from file steps.
• JsonTools gave an error Root node must be an array or object trying to load form a UTF-8 file with 50k items file from other sub-tests.
• chimera.json seems to completelly break Delphi 10.4's code completion (Code insight manager = LSP).

These libraries were tested:

• XSuperObject does not compile with C++Builder 10.2.
• dwsJSON does not compile with Delphi 10.1 Starter.

Changes on original source code

There were necessary modifications in order to compile and run some libraries with C++Builder.

• uLkJSON:
  • {$DEFINE USE_D2009}
  • TlkJSONstreamed.LoadFromStream();
  • TlkJSONstreamed.SaveToStream();
• SuperObject:
  • defined(VER290) or defined(VER300) or defined(VER310) or defined(VER320) or defined(VER330) or defined(VER340) or defined(VER350) or defined(VER360)
  • procedure FromInterface;