Add a case-insensitive case-preserving dict

[pitrou]

BPO	18986
Nosy	@tim-one, @warsaw, @theller, @birkenfeld, @rhettinger, @mdickinson, @jaraco, @pitrou, @vstinner, @ericvsmith, @merwok, @bitdancer, @ethanfurman, @vadmium, @serhiy-storchaka, @demianbrecht
Files	transform.patch transformdict.patch ctransformdict.patch dict__transform__.patch: Add dict.transform transformdict2.patch transformdict3.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/rhettinger'
closed_at = <Date 2015-05-16.22:07:19.854>
created_at = <Date 2013-09-09.11:22:01.918>
labels = ['type-feature', 'library']
title = 'Add a case-insensitive case-preserving dict'
updated_at = <Date 2015-05-16.22:07:19.852>
user = 'https://github.com/pitrou'

bugs.python.org fields:

activity = <Date 2015-05-16.22:07:19.852>
actor = 'ethan.furman'
assignee = 'rhettinger'
closed = True
closed_date = <Date 2015-05-16.22:07:19.854>
closer = 'ethan.furman'
components = ['Library (Lib)']
creation = <Date 2013-09-09.11:22:01.918>
creator = 'pitrou'
dependencies = []
files = ['31713', '31727', '31729', '31749', '31757', '31761']
hgrepos = []
issue_num = 18986
keywords = ['patch']
message_count = 86.0
messages = ['197359', '197360', '197361', '197362', '197366', '197367', '197368', '197369', '197370', '197376', '197377', '197379', '197380', '197381', '197387', '197389', '197390', '197391', '197392', '197393', '197398', '197399', '197400', '197401', '197402', '197403', '197405', '197406', '197410', '197411', '197412', '197430', '197434', '197445', '197446', '197457', '197464', '197469', '197479', '197516', '197525', '197526', '197527', '197528', '197529', '197531', '197533', '197635', '197637', '197644', '197648', '197710', '197711', '197733', '197969', '197970', '197973', '197975', '197980', '197981', '197982', '197983', '197984', '197986', '197987', '197989', '197990', '198281', '198912', '199652', '199708', '205979', '205995', '206027', '206195', '206196', '234050', '234062', '234077', '234087', '236105', '236161', '236163', '236178', '236909', '243370']
nosy_count = 19.0
nosy_names = ['tim.peters', 'barry', 'theller', 'georg.brandl', 'rhettinger', 'mark.dickinson', 'jaraco', 'pitrou', 'vstinner', 'eric.smith', 'eric.araujo', 'mrabarnett', 'Arfrever', 'r.david.murray', 'ethan.furman', 'sbt', 'martin.panter', 'serhiy.storchaka', 'demian.brecht']
pr_nums = []
priority = 'normal'
resolution = 'rejected'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue18986'
versions = ['Python 3.5']

[pitrou]

This is a very common need when implementing network protocols. You want to match keys case-insensitively but also preserve the original casing (e.g. for presentation).

When searching on the Web, you see many people reimplementing their own variant (often incomplete, or buggy). For example, Twisted has its own, the email package has something resembling it, WebOb also.

Having an implementation in the stdlib would spare many people the effort, ensure the implementation is complete and well-tested, and perhaps also add some optimizations to mitigate the overhead compared to a plain dict.

Note this is an instance of a more general pattern, where they key used for matching is derived from the lookup key using a constant derivation function. So maybe we want to implement the more general pattern and let users specify str.lower as the key derivation function.

[mrabarnett]

Surely a case-insensitive dict should use str.casefold, not str.lower?

[pitrou]

Surely a case-insensitive dict should use str.casefold, not
str.lower?

Perhaps. Network protocols will usually only allow ASCII in parts
where case is insensitive (e.g. header names), so it shouldn't make a
difference.

Implementing the generic pattern means this is left at the user's
discretion, though.

[pitrou]

For the record, I have my own implementation here:
https://bitbucket.org/optiflowsrd/obelus/src/tip/obelus/casedict.py?at=default
https://bitbucket.org/optiflowsrd/obelus/src/tip/obelus/test/test_casedict.py?at=default

[bitdancer]

For the record, email is not a good argument for this, since email could not use this data structure (its data structure is *not* a dict, but a list with dict-like features grafted on).

I do think this would be useful, and the generic version (analogous to defaultdict) would seem to make sense.

[pitrou]

Ok, let's bikeshed this a bit. What should be the name?

• projectdict?
• normalizedict?
• normdict?
• derivedict?
• transformdict?
• any ideas?

[ethanfurman]

I would say

• transformkeydict

Too bad we can't just add an extra 'transform_key' keyword to defaultdict.

[ericvsmith]

It would be nice to combine the behaviors that defaultdict and the case-insensitive comparisons.

[pitrou]

It would be nice to combine the behaviors that defaultdict and the case-insensitive comparisons.

Any use case? In my experience they are used in completely different
situations. defaultdict mostly to use the writing of some (internal)
algorithms, a case-insensitive dict to store user-visible data.

[ericvsmith]

Just today I was using a defaultdict where the keys are stock symbols. They're case insensitive (at least for this particular application).

In this case I just str.upper everything, but it would be a nice feature to case-preserve the keys that I pre-populate. I care less about the keys from user data.

[pitrou]

In this case I just str.upper everything, but it would be a nice
feature to case-preserve the keys that I pre-populate. I care less
about the keys from user data.

Well, stock symbols are what I would call user data :-)

[ericvsmith]

True enough!

I was trying to distinguish keys that I populate with initial values (mostly stock indexes) versus those where I just read values from a user-supplied file. When I populate the index values, I'd like to preserve the case I initially used. When I use user-supplied values, I don't know that the first value I use to populate the defaultdict has any more meaning that the last one I see.

It would just be a nice-to-have feature for which I have a real use case. It's not so critical that I can't work around it.

[ethanfurman]

To the point, however, Eric's example would make use of both the defaultdict portion and the transformkey portion in a single dict.