Skip to content

pwwang/toml-bench

Repository files navigation

toml-bench

deps

Which toml package to use in python?

See also: toml-lang and PEP 680

Report

Version

The verions of the packages tested in this report.

Version
toml 0.10.2
tomli/tomli_w 2.0.1; tomli_w: 1.0.0
tomlkit 0.12.5
rtoml 0.11.0
qtoml 0.3.1
tomllib (Python 3.12.2)

Dumping None value

How the package dumps None value in python

Literally <package>.dumps(None)

Dumped value or error
toml 'NoneType' object is not iterable
tomli/tomli_w 'NoneType' object has no attribute 'items'
tomlkit Expecting Mapping or TOML Container, <class 'NoneType'> given
rtoml "null"
---
rtoml v0.11+ supports dumping None to a desired string:
rtoml.dumps(data, none_value='@None'):
"@None"
qtoml 'NoneType' object has no attribute 'items'
tomllib module 'tomllib' has no attribute 'dumps'

Dumping key-None pair

How the package dumps key-value pair with value None

Literally <package>.dumps({"key": None})

Dumped value or error
toml
tomli/tomli_w Object of type <class 'NoneType'> is not TOML serializable
tomlkit Invalid type <class 'NoneType'>
rtoml key = "null"

---
rtoml v0.11+ supports dumping None to a desired string:
rtoml.dumps(data, none_value='@None'):
key = "@None"
qtoml TOML cannot encode None
tomllib module 'tomllib' has no attribute 'dumps'

Dumping list with None value

How the package dumps a list with None value in it.

Literally <package>.dumps({"key": [1, 2, 3, None, 5]})

Dumped value or error
toml key = [ 1, 2, 3, "None", 5,]
tomli/tomli_w Object of type <class 'NoneType'> is not TOML serializable
tomlkit Invalid type <class 'NoneType'>
rtoml key = [1, 2, 3, "null", 5]

---
rtoml v0.11+ supports dumping None to a desired string:
rtoml.dumps(data, none_value='@None'):
key = [1, 2, 3, "@None", 5]
qtoml bad type '<class 'NoneType'>' for dump_value
tomllib module 'tomllib' has no attribute 'dumps'

Loading None-like values

How the package loads None-like value in string

Literally <package>.loads('v1 = "null" v2 = "None"')

Loaded as
toml {'v1': 'null', 'v2': 'None'}
tomli/tomli_w module 'tomli_w' has no attribute 'loads'
tomlkit {'v1': 'null', 'v2': 'None'}
rtoml {'v1': 'null', 'v2': 'None'}
---
rtoml v0.11+ supports loading custom None values:
rtoml.loads(data, none_value='None'):
{'v1': 'null', 'v2': None}
rtoml.loads(data, none_value='null'):
{'v1': None, 'v2': 'None'}
qtoml {'v1': 'null', 'v2': 'None'}
tomllib {'v1': 'null', 'v2': 'None'}

Dumping a heterogenous array

How the package dumps a python dictionary with a heterogenous array.

Literally <package>.dumps({"v": [1, 1.2, True, "string"]})

Dumped value or error
toml v = [ 1, 1.2, true, "string",]
tomli/tomli_w v = [
    1,
    1.2,
    true,
    "string",
]
tomlkit v = [1, 1.2, true, "string"]
rtoml v = [1, 1.2, true, "string"]
qtoml v = [1, 1.2, true, 'string']
tomllib Dumping not supported

Loading a heterogenous array

How the package loads a toml string with a heterogenous array.

Literally <package>.loads('v = [1, 1.2, True, "string"]')

Loaded as
toml Not a homogeneous array (line 2 column 1 char 1)
tomli/tomli_w {'v': [1, 1.2, True, 'string']}
tomlkit {'v': [1, 1.2, True, 'string']}
rtoml {'v': [1, 1.2, True, 'string']}
qtoml {'v': [1, 1.2, True, 'string']}
tomllib {'v': [1, 1.2, True, 'string']}

Dumping a nested array

How the package dumps a python dictionary with a nested array.

Literally <package>.dumps({"v": [[1], [1, 2]]})

Dumped value or error
toml v = [ [ 1,], [ 1, 2,],]
tomli/tomli_w v = [
    [
        1,
    ],
    [
        1,
        2,
    ],
]
tomlkit v = [[1], [1, 2]]
rtoml v = [[1], [1, 2]]
qtoml v = [[1], [1, 2]]
tomllib Dumping not supported

Loading a nested array

How the package loads a toml string with a nested array.

Literally <package>.loads('v = [[1], [1, 2]]')

Loaded as
toml {'v': [[1], [1, 2]]}
tomli/tomli_w {'v': [[1], [1, 2]]}
tomlkit {'v': [[1], [1, 2]]}
rtoml {'v': [[1], [1, 2]]}
qtoml {'v': [[1], [1, 2]]}
tomllib {'v': [[1], [1, 2]]}

Dumping keeps order of keys?

Whether the package preserves the order of the keys while dumps a python dictionary.

Thus, whether <package>.dumps({"c": 1, "a": 2, "b": 3}) yields a string like c = 1\na = 2\nb = 3\n.

Order kept?
toml Kept
tomli/tomli_w Kept
tomlkit Kept
rtoml Kept
qtoml Kept
tomllib Dumping not supported

Loading keeps order of keys?

Whether the package preserves the order of the keys while loads a TOML string.

Thus, whether <package>.loads('c = 1\na = 2\nb = 3\n') yields a dictionary with keys in the order of ['c', 'a', 'b'].

Order kept?
toml Kept
tomli/tomli_w Kept
tomlkit Kept
rtoml Kept
qtoml Kept
tomllib Kept

Dumping unicode

How the package dumps Unicode in python

Literally, <package>.dumps({"你好": "世界"})

Dumped value
toml "你好" = "世界"
tomli/tomli_w "你好" = "世界"
tomlkit "你好" = "世界"
rtoml "你好" = "世界"
qtoml '你好' = '世界'
tomllib Dumping not supported

Loaded unicode

How the package loads a file with unicode.

The file was created by:

## Create a file with unicode content
with open(self.datafile, "w", encoding="utf-8") as f:
    f.write('"你好" = "世界"\n')

## Use `<package>.load()` to load the file
with open(self.datafile, "r", encoding="utf-8") as f:
    loaded = <package>.load(f)
Loaded as
toml {'你好': '世界'}
tomli/tomli_w File must be opened in binary mode, e.g. use open('foo.toml', 'rb')
When loaded with rb:
{'你好': '世界'}
tomlkit {'你好': '世界'}
rtoml {'你好': '世界'}
qtoml {'你好': '世界'}
tomllib File must be opened in binary mode, e.g. use open('foo.toml', 'rb')
When loaded with rb:
{'你好': '世界'}

Compliance with valid tests in toml-test

Test the compliance with the standard test suites for valid toml files here:

https://github.com/BurntSushi/toml-test/

The tests come up with a JSON counterpart that can be used to valid whether loading the toml file yields the same result as the JSON counterpart.

Result (toml-test v1.5.0)
toml spec/array-0.toml Not a homogeneous array (line 8 column 1 char 261)
spec/keys-4.toml Found invalid character in key name: 'c'. Try quoting the key name. (line 2 column 8 char 57)
spec/local-time-0.toml Parsed as unexpected data.
datetime/no-seconds.toml invalid literal for int() with base 0: '13:37' (line 2 column 1 char 46)
datetime/local-time.toml Parsed as unexpected data.
datetime/datetime.toml Parsed as unexpected data.
comment/tricky.toml Parsed as unexpected data.
key/dotted-1.toml Parsed as unexpected data.
key/unicode.toml Found invalid character in key name: '‍'. Try quoting the key name. (line 5 column 2 char 67)
key/dotted-2.toml Found invalid character in key name: '"'. Try quoting the key name. (line 7 column 11 char 166)
key/quoted-unicode.toml Duplicate keys! (line 3 column 1 char 19)
key/dotted-empty.toml Duplicate keys! (line 2 column 1 char 17)
key/escapes.toml Parsed as unexpected data.
table/empty-name.toml Can't have a keygroup with an empty name (line 1 column 1 char 0)
string/raw-multiline.toml Unbalanced quotes (line 20 column 50 char 532)
string/ends-in-whitespace-escape.toml Reserved escape sequence used (line 6 column 1 char 28)
string/hex-escape.toml Reserved escape sequence used (line 3 column 1 char 35)
string/escape-esc.toml Reserved escape sequence used (line 1 column 1 char 0)
string/multiline-quotes.toml Unterminated string found. Reached end of file. (line 27 column 1 char 664)
float/zero.toml Weirdness with leading zeroes or underscores in your number. (line 4 column 1 char 47)
array/mixed-int-string.toml Not a homogeneous array (line 1 column 1 char 0)
array/nested-double.toml Not a homogeneous array (line 1 column 1 char 0)
array/string-with-comma-2.toml string index out of range
array/mixed-int-float.toml Not a homogeneous array (line 1 column 1 char 0)
array/mixed-string-table.toml list index out of range
array/mixed-int-array.toml Not a homogeneous array (line 1 column 1 char 0)
inline-table/multiline.toml Invalid inline table value encountered (line 1 column 1 char 0)
inline-table/key-dotted-1.toml Parsed as unexpected data.
inline-table/key-dotted-5.toml Not a homogeneous array (line 2 column 1 char 20)
inline-table/newline.toml Key name found without value. Reached end of line. (line 5 column 2 char 98)
157/187 (83.96%) passed
tomli/tomli_w datetime/no-seconds.toml Expected newline or end of document after a statement (at line 2, column 23)
key/unicode.toml Invalid statement (at line 3, column 1)
string/hex-escape.toml Unescaped '' in a string (at line 3, column 22)
string/escape-esc.toml Unescaped '' in a string (at line 1, column 10)
inline-table/newline.toml Invalid initial character for a key part (at line 3, column 21)
182/187 (97.33%) passed
tomlkit datetime/no-seconds.toml Invalid number at line 2 col 25
key/unicode.toml Empty key at line 3 col 0
string/hex-escape.toml Invalid character 'x' in string at line 3 col 20
string/escape-esc.toml Invalid character 'e' in string at line 1 col 8
inline-table/newline.toml Empty key at line 3 col 20
182/187 (97.33%) passed
rtoml spec/table-9.toml duplicate key: apple for key fruit at line 8 column 1
datetime/no-seconds.toml expected a colon, found a newline at line 2 column 26
key/unicode.toml unexpected character found: \u{20ac} at line 3 column 1
table/array-within-dotted.toml duplicate key: apple for key fruit at line 4 column 1
string/hex-escape.toml invalid escape character in string: x at line 3 column 21
string/escape-esc.toml invalid escape character in string: e at line 1 column 9
inline-table/newline.toml expected a table key, found a newline at line 3 column 21
180/187 (96.26%) passed
qtoml spec/string-4.toml Didn't find expected newline (line 7, column 62)
spec/string-7.toml Didn't find expected newline (line 7, column 50)
datetime/no-seconds.toml can't parse type (line 2, column 20)
datetime/milliseconds.toml Didn't find expected newline (line 2, column 27)
datetime/datetime.toml Didn't find expected newline (line 4, column 18)
comment/after-literal-no-ws.toml can't parse type (line 1, column 4)
comment/tricky.toml can't parse type (line 11, column 7)
key/unicode.toml '€' cannot begin key (line 3, column 0)
string/raw-multiline.toml Didn't find expected newline (line 22, column 3)
string/hex-escape.toml \x not a valid escape (line 3, column 43)
string/escape-esc.toml \e not a valid escape (line 1, column 33)
string/multiline-quotes.toml Didn't find expected newline (line 4, column 26)
inline-table/newline.toml ' ' cannot begin key (line 3, column 20)
174/187 (93.05%) passed
tomllib datetime/no-seconds.toml Expected newline or end of document after a statement (at line 2, column 23)
key/unicode.toml Invalid statement (at line 3, column 1)
string/hex-escape.toml Unescaped '' in a string (at line 3, column 22)
string/escape-esc.toml Unescaped '' in a string (at line 1, column 10)
inline-table/newline.toml Invalid initial character for a key part (at line 3, column 21)
182/187 (97.33%) passed

Compliance with invalid tests in toml-test

Test the compliance with the standard test suites for invalid toml files here:

https://github.com/BurntSushi/toml-test/

  • Not OK: The toml file is parsed without error, but expected to fail.
  • OK: All files are failed to parse, as expected. Showing the last parsing error.
Result (toml-test v1.5.0)
toml Not OK: integer/double-sign-plus.toml incorrectly parsed.
Not OK: integer/us-after-bin.toml incorrectly parsed.
Not OK: integer/double-sign-nex.toml incorrectly parsed.
Not OK: integer/us-after-hex.toml incorrectly parsed.
Not OK: integer/us-after-oct.toml incorrectly parsed.
Not OK: spec/inline-table-2-0.toml incorrectly parsed.
Not OK: datetime/offset-overflow-minute.toml incorrectly parsed.
Not OK: datetime/offset-overflow-hour.toml incorrectly parsed.
Not OK: control/comment-del.toml incorrectly parsed.
Not OK: control/string-del.toml incorrectly parsed.
Not OK: 63 more items incorrectly parsed.
298/371 (80.32%) passed
tomli/tomli_w OK: inline-table/linebreak-1.toml Unclosed inline table (at line 3, column 18)
371/371 (100%) passed
tomlkit Not OK: control/comment-cr.toml incorrectly parsed.
Not OK: control/multi-cr.toml incorrectly parsed.
Not OK: control/rawmulti-cd.toml incorrectly parsed.
Not OK: control/bare-cr.toml incorrectly parsed.
Not OK: table/append-with-dotted-keys-1.toml incorrectly parsed.
Not OK: table/overwrite-array-in-parent.toml incorrectly parsed.
Not OK: table/append-to-array-with-dotted-keys.toml incorrectly parsed.
Not OK: table/append-with-dotted-keys-2.toml incorrectly parsed.
Not OK: array/extend-defined-aot.toml incorrectly parsed.
Not OK: inline-table/overwrite-09.toml incorrectly parsed.
361/371 (97.30%) passed
rtoml Not OK: integer/positive-hex.toml incorrectly parsed.
Not OK: integer/positive-bin.toml incorrectly parsed.
Not OK: integer/positive-oct.toml incorrectly parsed.
Not OK: datetime/offset-overflow-minute.toml incorrectly parsed.
Not OK: datetime/offset-overflow-hour.toml incorrectly parsed.
Not OK: control/comment-del.toml incorrectly parsed.
Not OK: control/comment-cr.toml incorrectly parsed.
Not OK: control/multi-cr.toml incorrectly parsed.
Not OK: control/rawmulti-cd.toml incorrectly parsed.
Not OK: control/bare-cr.toml incorrectly parsed.
361/371 (97.30%) passed
qtoml Not OK: spec/inline-table-2-0.toml incorrectly parsed.
Not OK: spec/table-9-1.toml incorrectly parsed.
Not OK: spec/table-9-0.toml incorrectly parsed.
Not OK: datetime/offset-overflow-minute.toml incorrectly parsed.
Not OK: control/comment-del.toml incorrectly parsed.
Not OK: control/comment-lf.toml incorrectly parsed.
Not OK: control/comment-null.toml incorrectly parsed.
Not OK: control/comment-ff.toml incorrectly parsed.
Not OK: control/comment-cr.toml incorrectly parsed.
Not OK: control/multi-cr.toml incorrectly parsed.
Not OK: 14 more items incorrectly parsed.
347/371 (93.53%) passed
tomllib OK: inline-table/linebreak-1.toml Unclosed inline table (at line 3, column 18)
371/371 (100%) passed

Compliance with valid tests in python tomllib test data

Test the compliance with python tomllib test data (since python 3.11) for valid toml files here:

https://github.com/python/cpython/tree/3.11/Lib/test/test_tomllib/data/valid

The tests come up with a JSON counterpart that can be used to valid whether loading the toml file yields the same result as the JSON counterpart.

Result (cpython tag 3.12.4)
toml apostrophes-in-literal-string.toml Unbalanced quotes (line 1 column 50 char 49)
five-quotes.toml Unterminated string found. Reached end of file. (line 7 column 1 char 97)
dates-and-times/datetimes.toml Parsed as unexpected data.
multiline-basic-str/ends-in-whitespace-escape.toml Reserved escape sequence used (line 6 column 1 char 28)
8/12 (66.67%) passed
tomli/tomli_w OK, 12/12 (100%) passed
tomlkit OK, 12/12 (100%) passed
rtoml OK, 12/12 (100%) passed
qtoml apostrophes-in-literal-string.toml Didn't find expected newline (line 3, column 3)
five-quotes.toml Didn't find expected newline (line 3, column 3)
dates-and-times/datetimes.toml Didn't find expected newline (line 1, column 19)
9/12 (75.00%) passed
tomllib OK, 12/12 (100%) passed

Compliance with invalid tests in python tomllib test data

Test the compliance with python tomllib test data (since python 3.11) for invalid toml files here:

https://github.com/python/cpython/tree/main/Lib/test/test_tomllib/data/invalid

  • Not OK: The toml file is parsed without error, but expected to fail.
  • OK: All files are failed to parse, as expected. Showing the last parsing error.
Result (cpython tag 3.12.4)
toml Not OK: invalid-comment-char.toml incorrectly parsed.
Not OK: multiline-basic-str/carriage-return.toml incorrectly parsed.
Not OK: dotted-keys/extend-defined-table.toml incorrectly parsed.
Not OK: dotted-keys/extend-defined-table-with-subtable.toml incorrectly parsed.
Not OK: array/unclosed-empty.toml incorrectly parsed.
Not OK: array/file-end-after-val.toml incorrectly parsed.
Not OK: array/unclosed-after-item.toml incorrectly parsed.
Not OK: inline-table/overwrite-value-in-inner-table.toml incorrectly parsed.
Not OK: inline-table/unclosed-empty.toml incorrectly parsed.
41/50 (82.00%) passed
tomli/tomli_w OK: inline-table/overwrite-implicitly.toml Cannot overwrite a value (at line 1, column 21)
50/50 (100%) passed
tomlkit Not OK: array-of-tables/overwrite-array-in-parent.toml incorrectly parsed.
Not OK: multiline-basic-str/carriage-return.toml incorrectly parsed.
Not OK: dotted-keys/extend-defined-table.toml incorrectly parsed.
Not OK: dotted-keys/extend-defined-aot.toml incorrectly parsed.
Not OK: dotted-keys/extend-defined-table-with-subtable.toml incorrectly parsed.
Not OK: inline-table/override-val-in-table.toml incorrectly parsed.
44/50 (88.00%) passed
rtoml Not OK: multiline-basic-str/carriage-return.toml incorrectly parsed.
49/50 (98.00%) passed
qtoml Not OK: non-scalar-escaped.toml incorrectly parsed.
Not OK: invalid-comment-char.toml incorrectly parsed.
Not OK: table/redefine-2.toml incorrectly parsed.
Not OK: table/redefine-1.toml incorrectly parsed.
Not OK: multiline-basic-str/carriage-return.toml incorrectly parsed.
Not OK: dotted-keys/extend-defined-table.toml incorrectly parsed.
Not OK: dotted-keys/extend-defined-table-with-subtable.toml incorrectly parsed.
Not OK: inline-table/overwrite-value-in-inner-table.toml incorrectly parsed.
Not OK: inline-table/override-val-with-table.toml incorrectly parsed.
41/50 (82.00%) passed
tomllib OK: inline-table/overwrite-implicitly.toml Cannot overwrite a value (at line 1, column 21)
50/50 (100%) passed

Running speed with data provided by rtoml

Test the speed of loading and dumping the loaded using data provided by rtoml

https://github.com/samuelcolvin/rtoml/raw/main/tests/data.toml

Loading speed Dumping speed
toml Excluded (heterogeneous arrays not supported) Excluded (heterogeneous arrays not supported)
tomli/tomli_w 2.14s (5000 iterations) 0.73s (5000 iterations)
tomlkit 39.78s (5000 iterations) 0.98s (5000 iterations)
rtoml 0.37s (5000 iterations) 0.08s (5000 iterations)
qtoml 4.99s (5000 iterations) 1.87s (5000 iterations)
tomllib 2.04s (5000 iterations) Dumping not supported

Running speed with data provided by tomli

Test the speed of loading and dumping the loaded using data provided by tomli

https://github.com/hukkin/tomli/raw/master/benchmark/data.toml

Loading speed Dumping speed
toml Excluded (heterogeneous arrays not supported) Excluded (heterogeneous arrays not supported)
tomli/tomli_w 1.41s (5000 iterations) 0.46s (5000 iterations)
tomlkit 24.55s (5000 iterations) 0.51s (5000 iterations)
rtoml 0.32s (5000 iterations) 0.16s (5000 iterations)
qtoml 3.63s (5000 iterations) 1.25s (5000 iterations)
tomllib 1.44s (5000 iterations) Dumping not supported

Other reports

Run your own report

Install

pip install -U toml-bench

Generate your own report

toml-bench

Use a different data directory than the default one

toml-bench --datadir /tmp/toml-bench

Write the report to a markdown file

toml-bench --report ./README.md

Test with a different version of compliance set (BurntSushi/toml-test)

toml-bench --comver 1.0.0

Use a different number of iterations in speed tests

toml-bench --iter 5000

Test with different versions of packages

git clone https://github.com/pwwang/toml-bench.git
cd toml-bench
# See https://python-poetry.org/docs/cli/#add
# for how to specify a version constraint
poetry add "tomli=2.0.0"
poetry update
poetry install
poetry run toml-bench