improve json decoding performance

[pitrou]

BPO	7451
Nosy	@rhettinger, @etrepum, @pitrou, @benjaminp
Files	json-opts2.patch json-opts3.patch json-opts4.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2010-09-04.20:17:11.093>
created_at = <Date 2009-12-07.23:08:18.891>
labels = ['library', 'performance']
title = 'improve json decoding performance'
updated_at = <Date 2010-09-04.20:17:11.091>
user = 'https://github.com/pitrou'

bugs.python.org fields:

activity = <Date 2010-09-04.20:17:11.091>
actor = 'pitrou'
assignee = 'none'
closed = True
closed_date = <Date 2010-09-04.20:17:11.093>
closer = 'pitrou'
components = ['Library (Lib)']
creation = <Date 2009-12-07.23:08:18.891>
creator = 'pitrou'
dependencies = []
files = ['15478', '15484', '18755']
hgrepos = []
issue_num = 7451
keywords = ['patch']
message_count = 11.0
messages = ['96082', '96093', '96097', '96098', '96099', '96169', '96171', '96903', '96908', '115611', '115612']
nosy_count = 5.0
nosy_names = ['rhettinger', 'bob.ippolito', 'pitrou', 'benjamin.peterson', 'vkuznet']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue7451'
versions = ['Python 3.2']

[pitrou]

This patch does two things:

• it speeds up decoding of JSON objects by doing fewer temporary allocations
• it reduces memory use of decoded JSON objects by reusing key strings
  when they appear several times in a JSON string

[benjaminp]

The problem with this is that it makes json in Python diverge even more
from the simplejson trunk.

[pitrou]

Unless Bob applies it to simplejson as well, that is.

[pitrou]

I've just noticed that there is a persistent decoder instance in
json._default_decoder. To avoid keeping references to past keys
forever, this updated patch clears the memo dict when a string has
finished decoding.

[pitrou]

Here is the patch.

[vkuznet]

I wonder if you can make a patch for 2.6 python branch.

[pitrou]

I wonder if you can make a patch for 2.6 python branch.

No, 2.6 doesn't have the same C accelerator in the first place.

[etrepum]

I applied most of this patch to r206 of simplejson trunk

[pitrou]

Le dimanche 27 décembre 2009 à 11:10 +0000, Bob Ippolito a écrit :

Bob Ippolito <bob@redivi.com> added the comment:

I applied most of this patch to r206 of simplejson trunk

Thank you. Will you port it to CPython yourself or would you prefer
someone else to do it?

[pitrou]

Updated patch against py3k.

[pitrou]

Committed in r84505.