Faster marshalling

[serhiy-storchaka]

BPO	23344
Nosy	@kristjanvalur, @serhiy-storchaka
Dependencies	bpo-23392: Add tests for marshal FILE* API
Files	marshal_faster_write.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2015-02-11.15:51:27.406>
created_at = <Date 2015-01-28.22:22:06.820>
labels = ['interpreter-core', 'performance']
title = 'Faster marshalling'
updated_at = <Date 2015-02-11.15:51:27.405>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2015-02-11.15:51:27.405>
actor = 'serhiy.storchaka'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2015-02-11.15:51:27.406>
closer = 'serhiy.storchaka'
components = ['Interpreter Core']
creation = <Date 2015-01-28.22:22:06.820>
creator = 'serhiy.storchaka'
dependencies = ['23392']
files = ['37898']
hgrepos = []
issue_num = 23344
keywords = ['patch']
message_count = 4.0
messages = ['234920', '235347', '235747', '235752']
nosy_count = 3.0
nosy_names = ['kristjan.jonsson', 'python-dev', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue23344'
versions = ['Python 3.5']

[serhiy-storchaka]

Currently writing marshalled data to buffer is not very efficient. Data is written byte by byte with testing conditions p->fp != NULL and p->ptr != p->end for every byte. Proposed patch makes writing to buffer faster.

Benchmark results:

$ ./python -m timeit -s "import marshal; d = compile(open('Lib/_pydecimal.py').read(), '_pydecimal.py', 'exec')" -- "marshal.dumps(d)"
Unpatched: 100 loops, best of 3: 4.64 msec per loop
Patched: 100 loops, best of 3: 3.39 msec per loop

$ ./python -m timeit -s "import marshal; a = ['%010x' % i for i in range(10**4)]" -- "marshal.dumps(a)"
Unpatched: 1000 loops, best of 3: 1.96 msec per loop
Patched: 1000 loops, best of 3: 1.32 msec per loop

$ ./python -m timeit -s "import marshal; a = ['%0100x' % i for i in range(10**4)]" -- "marshal.dumps(a)"
Unpatched: 100 loops, best of 3: 10.3 msec per loop
Patched: 100 loops, best of 3: 3.39 msec per loop

[kristjanvalur]

looks good to me, although it has been pointed out that marshal _write_ speed is less critical than read speed :)

[python-dev]

New changeset bb05f845e7dc by Serhiy Storchaka in branch 'default':
Issue bpo-23344: marshal.dumps() is now 20-25% faster on average.
https://hg.python.org/cpython/rev/bb05f845e7dc

[serhiy-storchaka]

Thank you for your review Kristján. Together with bpo-20416 this increases dumping speed almost twice for typical module data and up to 5x for some data.