Skip to content

pwwang/toml-bench

Repository files navigation

toml-bench

deps

Which toml package to use in python?

See also: toml-lang and PEP 680

Report

Version

The verions of the packages tested in this report.

Version
toml 0.10.2
tomli/tomli_w 2.0.1; tomli_w: 1.0.0
tomlkit 0.11.8
pytomlpp 1.0.13
rtoml 0.9.0
qtoml 0.3.1

Dumping None value

How the package dumps None value in python

Literally <package>.dumps(None)

Dumped value or error
toml 'NoneType' object is not iterable
tomli/tomli_w 'NoneType' object has no attribute 'items'
tomlkit Expecting Mapping or TOML Container, <class 'NoneType'> given
pytomlpp dumps(): incompatible function arguments. The following argument types are supported:
1. (arg0: dict) -> str

Invoked with: None
rtoml "null"
qtoml 'NoneType' object has no attribute 'items'

Dumping key-None pair

How the package dumps key-value pair with value None

Literally <package>.dumps({"key": None})

Dumped value or error
toml
tomli/tomli_w Object of type <class 'NoneType'> is not TOML serializable
tomlkit Invalid type <class 'NoneType'>
pytomlpp cannot convert value None to proper toml type
rtoml key = "null"
qtoml TOML cannot encode None

Dumping list with None value

How the package dumps a list with None value in it.

Literally <package>.dumps({"key": [1, 2, 3, None, 5]})

Dumped value or error
toml key = [ 1, 2, 3, "None", 5,]
tomli/tomli_w Object of type <class 'NoneType'> is not TOML serializable
tomlkit Invalid type <class 'NoneType'>
pytomlpp not a valid type for conversion None
rtoml key = [1, 2, 3, "null", 5]
qtoml bad type '<class 'NoneType'>' for dump_value

Loading None-like values

How the package loads None-like value in string

Literally <package>.loads('v1 = "null" v2 = "None"')

Loaded as
toml {'v1': 'null', 'v2': 'None'}
tomli/tomli_w {'v1': 'null', 'v2': 'None'}
tomlkit {'v1': 'null', 'v2': 'None'}
pytomlpp {'v1': 'null', 'v2': 'None'}
rtoml {'v1': 'null', 'v2': 'None'}
qtoml {'v1': 'null', 'v2': 'None'}

Dumping a heterogenous array

How the package dumps a python dictionary with a heterogenous array.

Literally <package>.dumps({"v": [1, 1.2, True, "string"]})

Dumped value or error
toml v = [ 1, 1.2, true, "string",]
tomli/tomli_w v = [
    1,
    1.2,
    true,
    "string",
]
tomlkit v = [1, 1.2, true, "string"]
pytomlpp v = [ 1, 1.2, 1, 'string' ]
rtoml v = [1, 1.2, true, "string"]
qtoml v = [1, 1.2, true, 'string']

Loading a heterogenous array

How the package loads a toml string with a heterogenous array.

Literally <package>.loads('v = [1, 1.2, True, "string"]')

Loaded as
toml Not a homogeneous array (line 2 column 1 char 1)
tomli/tomli_w {'v': [1, 1.2, True, 'string']}
tomlkit {'v': [1, 1.2, True, 'string']}
pytomlpp {'v': [1, 1.2, True, 'string']}
rtoml {'v': [1, 1.2, True, 'string']}
qtoml {'v': [1, 1.2, True, 'string']}

Dumping a nested array

How the package dumps a python dictionary with a nested array.

Literally <package>.dumps({"v": [[1], [1, 2]]})

Dumped value or error
toml v = [ [ 1,], [ 1, 2,],]
tomli/tomli_w v = [
    [
        1,
    ],
    [
        1,
        2,
    ],
]
tomlkit v = [[1], [1, 2]]
pytomlpp v = [ [ 1 ], [ 1, 2 ] ]
rtoml v = [[1], [1, 2]]
qtoml v = [[1], [1, 2]]

Loading a nested array

How the package loads a toml string with a nested array.

Literally <package>.loads('v = [[1], [1, 2]]')

Loaded as
toml {'v': [[1], [1, 2]]}
tomli/tomli_w {'v': [[1], [1, 2]]}
tomlkit {'v': [[1], [1, 2]]}
pytomlpp {'v': [[1], [1, 2]]}
rtoml {'v': [[1], [1, 2]]}
qtoml {'v': [[1], [1, 2]]}

Dumping keeps order of keys?

Whether the package preserves the order of the keys while dumps a python dictionary.

Thus, whether <package>.dumps({"c": 1, "a": 2, "b": 3}) yields a string like c = 1\na = 2\nb = 3\n.

Order kept?
toml Kept
tomli/tomli_w Kept
tomlkit Kept
pytomlpp Lost
rtoml Kept
qtoml Kept

Loading keeps order of keys?

Whether the package preserves the order of the keys while loads a TOML string.

Thus, whether <package>.loads('c = 1\na = 2\nb = 3\n') yields a dictionary with keys in the order of ['c', 'a', 'b'].

Order kept?
toml Kept
tomli/tomli_w Kept
tomlkit Kept
pytomlpp Lost
rtoml Kept
qtoml Kept

Dumping unicode

How the package dumps Unicode in python

Literally, <package>.dumps({"你好": "世界"})

Dumped value
toml "你好" = "世界"
tomli/tomli_w "你好" = "世界"
tomlkit "你好" = "世界"
pytomlpp '你好' = '世界'
rtoml "你好" = "世界"
qtoml '你好' = '世界'

Loaded unicode

How the package loads a file with unicode.

The file was created by:

## Create a file with unicode content
with open(self.datafile, "w", encoding="utf-8") as f:
    f.write('"你好" = "世界"\n')

## Use `<package>.load()` to load the file
with open(self.datafile, "r", encoding="utf-8") as f:
    loaded = <package>.load(f)
Loaded as
toml {'你好': '世界'}
tomli/tomli_w File must be opened in binary mode, e.g. use open('foo.toml', 'rb')
When loaded with rb:
{'你好': '世界'}
tomlkit {'你好': '世界'}
pytomlpp {'你好': '世界'}
rtoml {'你好': '世界'}
qtoml {'你好': '世界'}

Compliance with valid tests in toml-test

Test the compliance with the standard test suites for valid toml files here:

https://github.com/BurntSushi/toml-test/

The tests come up with a JSON counterpart that can be used to valid whether loading the toml file yields the same result as the JSON counterpart.

Result (toml-test v1.3.0)
toml key/escapes.toml Parsed as unexpected data.
key/dotted.toml Found invalid character in key name: '"'. Try quoting the key name. (line 12 column 11 char 245)
comment/tricky.toml Parsed as unexpected data.
inline-table/key-dotted.toml Parsed as unexpected data.
inline-table/multiline.toml Invalid inline table value encountered (line 1 column 1 char 0)
array/nested-double.toml Not a homogeneous array (line 1 column 1 char 0)
array/mixed-int-string.toml Not a homogeneous array (line 1 column 1 char 0)
array/mixed-string-table.toml list index out of range
array/mixed-int-array.toml Not a homogeneous array (line 1 column 1 char 0)
array/mixed-int-float.toml Not a homogeneous array (line 1 column 1 char 0)
float/zero.toml Weirdness with leading zeroes or underscores in your number. (line 4 column 1 char 47)
string/escape-esc.toml Reserved escape sequence used (line 1 column 1 char 0)
datetime/datetime.toml Parsed as unexpected data.
datetime/local-time.toml Parsed as unexpected data.
86/100 (86.00%) passed
tomli/tomli_w string/escape-esc.toml Unescaped '' in a string (at line 1, column 10)
99/100 (99.00%) passed
tomlkit string/escape-esc.toml Invalid character 'e' in string at line 1 col 8
99/100 (99.00%) passed
pytomlpp string/escape-esc.toml Error while parsing string: escape sequence '\e' is not supported in TOML 1.0.0 and earlier (error occurred at line 1, column 9)
99/100 (99.00%) passed
rtoml string/escape-esc.toml invalid escape character in string: e at line 1 column 9
99/100 (99.00%) passed
qtoml comment/tricky.toml can't parse type (line 11, column 7)
string/multiline-quotes.toml Didn't find expected newline (line 4, column 26)
string/escape-esc.toml \e not a valid escape (line 1, column 33)
datetime/milliseconds.toml Didn't find expected newline (line 2, column 27)
datetime/datetime.toml Didn't find expected newline (line 2, column 18)
95/100 (95.00%) passed

Compliance with invalid tests in toml-test

Test the compliance with the standard test suites for invalid toml files here:

https://github.com/BurntSushi/toml-test/

  • Not OK: The toml file is parsed without error, but expected to fail.
  • OK: All files are failed to parse, as expected. Showing the last parsing error.
Result (toml-test v1.3.0)
toml Not OK: key/start-dot.toml incorrectly parsed.
Not OK: key/escape.toml incorrectly parsed.
Not OK: key/special-character.toml incorrectly parsed.
Not OK: key/two-equals3.toml incorrectly parsed.
Not OK: key/two-equals2.toml incorrectly parsed.
Not OK: inline-table/add.toml incorrectly parsed.
Not OK: array/no-close-2.toml incorrectly parsed.
Not OK: array/tables-1.toml incorrectly parsed.
Not OK: array/no-close.toml incorrectly parsed.
Not OK: array/extending-table.toml incorrectly parsed.
Not OK: 39 more items incorrectly parsed.
188/237 (79.32%) passed
tomli/tomli_w OK: datetime/mday-under.toml Expected newline or end of document after a statement (at line 3, column 9)
237/237 (100%) passed
tomlkit Not OK: control/bare-cr.toml incorrectly parsed.
Not OK: control/comment-cr.toml incorrectly parsed.
Not OK: table/append-with-dotted-keys-2.toml incorrectly parsed.
Not OK: table/append-with-dotted-keys-1.toml incorrectly parsed.
233/237 (98.31%) passed
pytomlpp Not OK: control/bare-cr.toml incorrectly parsed.
Not OK: control/comment-cr.toml incorrectly parsed.
235/237 (99.16%) passed
rtoml Not OK: control/bare-cr.toml incorrectly parsed.
Not OK: control/comment-cr.toml incorrectly parsed.
Not OK: control/comment-del.toml incorrectly parsed.
Not OK: integer/positive-hex.toml incorrectly parsed.
Not OK: integer/positive-bin.toml incorrectly parsed.
232/237 (97.89%) passed
qtoml Not OK: inline-table/trailing-comma.toml incorrectly parsed.
Not OK: inline-table/add.toml incorrectly parsed.
Not OK: control/bare-cr.toml incorrectly parsed.
Not OK: control/comment-us.toml incorrectly parsed.
Not OK: control/comment-cr.toml incorrectly parsed.
Not OK: control/comment-lf.toml incorrectly parsed.
Not OK: control/comment-del.toml incorrectly parsed.
Not OK: control/comment-null.toml incorrectly parsed.
Not OK: table/duplicate-key-dotted-table.toml incorrectly parsed.
Not OK: table/duplicate-key-dotted-table2.toml incorrectly parsed.
Not OK: 2 more items incorrectly parsed.
225/237 (94.94%) passed

Compliance with valid tests in python tomllib test data

Test the compliance with python tomllib test data (since python 3.11) for valid toml files here:

https://github.com/python/cpython/tree/3.11/Lib/test/test_tomllib/data/valid

The tests come up with a JSON counterpart that can be used to valid whether loading the toml file yields the same result as the JSON counterpart.

Result (cpython tag 3.11.0)
toml apostrophes-in-literal-string.toml Unbalanced quotes (line 1 column 50 char 49)
five-quotes.toml Unterminated string found. Reached end of file. (line 7 column 1 char 97)
dates-and-times/datetimes.toml Parsed as unexpected data.
multiline-basic-str/ends-in-whitespace-escape.toml Reserved escape sequence used (line 6 column 1 char 28)
8/12 (66.67%) passed
tomli/tomli_w OK, 12/12 (100%) passed
tomlkit OK, 12/12 (100%) passed
pytomlpp OK, 12/12 (100%) passed
rtoml OK, 12/12 (100%) passed
qtoml apostrophes-in-literal-string.toml Didn't find expected newline (line 3, column 3)
five-quotes.toml Didn't find expected newline (line 3, column 3)
dates-and-times/datetimes.toml Didn't find expected newline (line 1, column 19)
9/12 (75.00%) passed

Compliance with invalid tests in python tomllib test data

Test the compliance with python tomllib test data (since python 3.11) for invalid toml files here:

https://github.com/python/cpython/tree/main/Lib/test/test_tomllib/data/invalid

  • Not OK: The toml file is parsed without error, but expected to fail.
  • OK: All files are failed to parse, as expected. Showing the last parsing error.
Result (cpython tag 3.11.0)
toml Not OK: invalid-comment-char.toml incorrectly parsed.
Not OK: inline-table/unclosed-empty.toml incorrectly parsed.
Not OK: inline-table/overwrite-value-in-inner-table.toml incorrectly parsed.
Not OK: array/unclosed-empty.toml incorrectly parsed.
Not OK: array/unclosed-after-item.toml incorrectly parsed.
Not OK: array/file-end-after-val.toml incorrectly parsed.
Not OK: dotted-keys/extend-defined-table-with-subtable.toml incorrectly parsed.
Not OK: dotted-keys/extend-defined-table.toml incorrectly parsed.
Not OK: multiline-basic-str/carriage-return.toml incorrectly parsed.
41/50 (82.00%) passed
tomli/tomli_w OK: multiline-basic-str/escape-only.toml Unterminated string (at end of document)
50/50 (100%) passed
tomlkit Not OK: inline-table/override-val-in-table.toml incorrectly parsed.
Not OK: dotted-keys/extend-defined-table-with-subtable.toml incorrectly parsed.
Not OK: dotted-keys/extend-defined-aot.toml incorrectly parsed.
Not OK: dotted-keys/extend-defined-table.toml incorrectly parsed.
Not OK: array-of-tables/overwrite-array-in-parent.toml incorrectly parsed.
Not OK: multiline-basic-str/carriage-return.toml incorrectly parsed.
44/50 (88.00%) passed
pytomlpp Not OK: multiline-basic-str/carriage-return.toml incorrectly parsed.
49/50 (98.00%) passed
rtoml Not OK: multiline-basic-str/carriage-return.toml incorrectly parsed.
49/50 (98.00%) passed
qtoml Not OK: non-scalar-escaped.toml incorrectly parsed.
Not OK: invalid-comment-char.toml incorrectly parsed.
Not OK: inline-table/override-val-with-table.toml incorrectly parsed.
Not OK: inline-table/overwrite-value-in-inner-table.toml incorrectly parsed.
Not OK: dotted-keys/extend-defined-table-with-subtable.toml incorrectly parsed.
Not OK: dotted-keys/extend-defined-table.toml incorrectly parsed.
Not OK: table/redefine-2.toml incorrectly parsed.
Not OK: table/redefine-1.toml incorrectly parsed.
Not OK: multiline-basic-str/carriage-return.toml incorrectly parsed.
41/50 (82.00%) passed

Running speed with data provided by pytomlpp

Test the speed of loading and dumping the loaded using data provided by pytomlpp

https://github.com/bobfang1992/pytomlpp/raw/master/benchmark/data.toml

Loading speed Dumping speed
toml Excluded (heterogeneous arrays not supported) Excluded (heterogeneous arrays not supported)
tomli/tomli_w 7.55s (1000 iterations) 3.11s (1000 iterations)
tomlkit 150.98s (1000 iterations) 2.14s (1000 iterations)
pytomlpp 1.41s (1000 iterations) 0.98s (1000 iterations)
rtoml 0.98s (1000 iterations) values must be emitted before tables
qtoml 18.39s (1000 iterations) 4.75s (1000 iterations)

Running speed with data provided by rtoml

Test the speed of loading and dumping the loaded using data provided by rtoml

https://github.com/samuelcolvin/rtoml/raw/main/tests/data.toml

Loading speed Dumping speed
toml Excluded (heterogeneous arrays not supported) Excluded (heterogeneous arrays not supported)
tomli/tomli_w 1.56s (1000 iterations) 0.55s (1000 iterations)
tomlkit 29.90s (1000 iterations) 0.75s (1000 iterations)
pytomlpp 0.24s (1000 iterations) 0.19s (1000 iterations)
rtoml 0.19s (1000 iterations) 0.05s (1000 iterations)
qtoml 4.57s (1000 iterations) 1.26s (1000 iterations)

Running speed with data provided by tomli

Test the speed of loading and dumping the loaded using data provided by tomli

https://github.com/hukkin/tomli/raw/master/benchmark/data.toml

Loading speed Dumping speed
toml Excluded (heterogeneous arrays not supported) Excluded (heterogeneous arrays not supported)
tomli/tomli_w 1.11s (1000 iterations) 0.38s (1000 iterations)
tomlkit 19.39s (1000 iterations) 0.37s (1000 iterations)
pytomlpp 0.25s (1000 iterations) 0.18s (1000 iterations)
rtoml 0.20s (1000 iterations) 0.11s (1000 iterations)
qtoml 3.61s (1000 iterations) 0.94s (1000 iterations)

Other reports

Run your own report

Install

pip install -U toml-bench

Generate your own report

toml-bench

Use a different data directory than the default one

toml-bench --datadir /tmp/toml-bench

Write the report to a markdown file

toml-bench --report ./README.md

Test with a different version of compliance set (BurntSushi/toml-test)

toml-bench --comver 1.0.0

Use a different number of iterations in speed tests

toml-bench --iter 5000

Test with different versions of packages

git clone https://github.com/pwwang/toml-bench.git
cd toml-bench
# See https://python-poetry.org/docs/cli/#add
# for how to specify a version constraint
poetry add "tomli=2.0.0"
poetry update
poetry install
poetry run toml-bench