Add OrderedDict written in C

[ericsnowcurrently]

BPO	16991
Nosy	@rhettinger, @gpshead, @ncoghlan, @pitrou, @scoder, @ericvsmith, @tiran, @benjaminp, @ned-deily, @ezio-melotti, @merwok, @alex, @asvetlov, @skrah, @florentx, @markshannon, @ericsnowcurrently, @JimJJewett, @serhiy-storchaka, @1st1, @westurner, @refi64, @MojoVampire
Files	odict-speed.diff: pybench addition for OrderedDict cOrderedDict.diff cOrderedDict-2.diff cOrderedDict-3.diff cOrderedDict-4.diff 3b2a9026d48e.diff c3fab329aa7f.diff ba1c6d40ca63.diff 0813b1a88171.diff odict-windows.diff crash-1.py crash-2.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ericsnowcurrently'
closed_at = <Date 2015-06-01.16:26:38.776>
created_at = <Date 2013-01-18.04:08:30.389>
labels = ['interpreter-core', 'type-feature', 'release-blocker']
title = 'Add OrderedDict written in C'
updated_at = <Date 2015-06-02.14:01:03.238>
user = 'https://github.com/ericsnowcurrently'

bugs.python.org fields:

activity = <Date 2015-06-02.14:01:03.238>
actor = 'eric.snow'
assignee = 'eric.snow'
closed = True
closed_date = <Date 2015-06-01.16:26:38.776>
closer = 'eric.snow'
components = ['Interpreter Core']
creation = <Date 2013-01-18.04:08:30.389>
creator = 'eric.snow'
dependencies = []
files = ['30461', '30661', '32262', '38753', '39445', '39496', '39499', '39500', '39566', '39567', '39581', '39582']
hgrepos = ['309']
issue_num = 16991
keywords = ['patch']
message_count = 134.0
messages = ['180170', '180176', '180202', '180236', '180640', '180682', '180688', '180690', '181421', '181422', '182132', '182134', '182145', '182147', '182214', '184801', '184835', '189891', '189893', '189895', '189897', '190589', '190590', '190639', '190640', '190643', '191557', '200607', '209700', '209744', '209831', '215971', '231087', '232412', '232803', '232825', '232828', '239665', '239666', '239740', '239761', '239867', '239871', '243096', '243121', '243123', '243343', '243358', '243371', '243379', '243492', '243699', '243702', '243705', '243706', '243714', '243715', '243716', '243726', '243733', '243749', '243753', '243799', '243801', '243802', '243803', '243806', '243807', '243843', '243849', '243863', '243868', '243903', '243915', '243959', '243960', '243964', '243965', '243966', '244005', '244039', '244040', '244044', '244046', '244047', '244048', '244053', '244054', '244067', '244070', '244071', '244073', '244074', '244076', '244122', '244380', '244401', '244402', '244421', '244436', '244437', '244438', '244439', '244440', '244441', '244442', '244462', '244463', '244464', '244465', '244466', '244467', '244468', '244469', '244475', '244476', '244478', '244480', '244481', '244484', '244486', '244491', '244492', '244493', '244496', '244499', '244535', '244574', '244575', '244586', '244587', '244600', '244664', '244668']
nosy_count = 29.0
nosy_names = ['rhettinger', 'gregory.p.smith', 'ncoghlan', 'pitrou', 'scoder', 'eric.smith', 'christian.heimes', 'benjamin.peterson', 'ned.deily', 'ezio.melotti', 'eric.araujo', 'mrabarnett', 'Arfrever', 'alex', 'asvetlov', 'skrah', 'flox', 'BreamoreBoy', 'Mark.Shannon', 'python-dev', 'eric.snow', 'Jim.Jewett', 'serhiy.storchaka', 'yselivanov', 'westurner', 'refi64', 'josh.r', 'tonn81', 'introom']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue16991'
versions = ['Python 3.5']

[ericsnowcurrently]

Here's an initial stab at writing OrderedDict in C. Though, the implementation is not heavily optimized and isn't super subclass-friendly, my hope is that it's relatively close to the final version. I'm getting this up now to get some eyes on it.

The spot in the builtins is mostly for convenience, but I expect it will need to be exposed somewhere (perhaps _collections?).

My experience with the C-API is relatively limited and my C-fu is not at a professional level. However, I'm pretty sure that I have most everything correct.

The ultimate goal for this type is to use it for **kwargs.

Note: this first patch has some reference leaks that I'm tracking down.

[benjaminp]

Nit: This really not be exposed through _collections rather than hacked into builtins.

[ezio-melotti]

The tests should probably be updated to use the PEP-399 idiom in order to test both the implementations.

[ericsnowcurrently]

@benjamin: Yeah, I've fixed that.

@ezio: Good point. I've touched that up.

Once I have cleared up reference counting issues I'll put up a new patch.

[ericsnowcurrently]

Here's a cleanup of test.test_collections that helps keep the subsequent patch (still forthcoming) cleaner.

[ezio-melotti]

What's the reason for moving the OrderedDict tests in a separate file?

[ericsnowcurrently]

What's the reason for moving the OrderedDict tests in a separate file?

Following the precedent of collections.deque and collections.defaultdict:

• a big chunk of code
• the default implementation will be coming via _collections.

[ezio-melotti]

These are indeed good reasons.

[ericsnowcurrently]

Here's an updated patch. I still have some ref-counting issues, but the patch is much closer to what I expect will be the final version. At this point it passes all the main unit tests (segfaults in some of the supplemental Mapping tests).

One key thing is that for now the various iterators are faked using lists. Once everything is sorted out I'll plug my implementation of the iterators back in (which I'd already mostly finished).

I see room for efficiency improvements in a couple spots. As well, I plan on improving the subclass-ability of the type once it's otherwise happy.

Any feedback at the point would be helpful. Regardless, I'm being careful to get this right and I'm no expert with the C-API, so it's taking a while. :)

Some other considerations:

• ensure that __init__() can reset and populate an existing dictionary:
  d=OrderedDict(); d[0]=1; d.__init__() # resets
• avoid any reference cycles so that dicts can clean-up right-away when their ref-count drops to zero
  rather than waiting on GC.
• possibly use the GIL to make the link updates atomic, trying to make the overall class thread safe.
• methods like get() pop() or setdefault() shouldn't rely on __getitem__ raising KeyError
  because __missing__ may be present in a subclass

[ericsnowcurrently]

Looks like I didn't get the patch lined up to tip so the review link isn't showing up. I'll have to fix that tomorrow.

[ericsnowcurrently]

Are there any objections to this unittest cleanup patch? I'd like to commit it separately prior to anything that will go in for the C OrderedDict.

[ericsnowcurrently]

Incidently, I'm just about done with the OrdereDict patch. I'm attaching the current one. At present it passes the unit tests, but I have two memory-related issues and a number of open questions (that I don't think are critical).

The patch has the unittest cleanup merged in just so it will show up in reviewboard.