tuple function gives wrong answer when called on list subclass with custom __iter__

[DavidMacIver]

BPO	23757
Nosy	@rhettinger, @bitdancer, @serhiy-storchaka
Superseder	bpo-10977: Concrete object C API considered harmful to subclasses of builtin types
Files	listwithiter.py: Reproduction test case fix_list_to_tuple.diff: Patch without the OP's test fix_list_to_tuple_2.diff: + tests

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/rhettinger'
closed_at = <Date 2015-05-17.21:47:59.965>
created_at = <Date 2015-03-24.08:42:29.638>
labels = ['easy', 'type-bug', 'library']
title = 'tuple function gives wrong answer when called on list subclass with custom __iter__'
updated_at = <Date 2015-05-17.21:47:59.964>
user = 'https://bugs.python.org/DavidMacIver'

bugs.python.org fields:

activity = <Date 2015-05-17.21:47:59.964>
actor = 'rhettinger'
assignee = 'rhettinger'
closed = True
closed_date = <Date 2015-05-17.21:47:59.965>
closer = 'rhettinger'
components = ['Library (Lib)']
creation = <Date 2015-03-24.08:42:29.638>
creator = 'David MacIver'
dependencies = []
files = ['38663', '39365', '39404']
hgrepos = []
issue_num = 23757
keywords = ['patch', 'easy']
message_count = 11.0
messages = ['239098', '239138', '239152', '239154', '239161', '243073', '243158', '243159', '243392', '243428', '243429']
nosy_count = 6.0
nosy_names = ['rhettinger', 'stub', 'r.david.murray', 'python-dev', 'serhiy.storchaka', 'David MacIver']
pr_nums = []
priority = 'high'
resolution = 'fixed'
stage = 'commit review'
status = 'closed'
superseder = '10977'
type = 'behavior'
url = 'https://bugs.python.org/issue23757'
versions = ['Python 2.7', 'Python 3.4', 'Python 3.5']

[DavidMacIver]

Converting a list to a tuple appears to have an optimisation that is wrong in the presence of subclassing to override __iter__. It ignores the user defined iter and uses the normal list one. I've attached a file with a test case to demonstrate this.

I've verified this on both python 2.7 and python 3.4. It's presumably also the case on everything in between.

This was found because it caused a bug with a type that pytz uses, which lazily populates the list on iteration: https://bugs.launchpad.net/pytz/+bug/1435617

[bitdancer]

This is a specific instance of the general problem covered by bpo-10977.

[DavidMacIver]

Ah, I hadn't seen that. Thanks for the link.

But... is it really? They have basically the same root cause, but the general problem seems to be hard to fix, while the specific problem here seems to be basically "don't use the concrete API here because it breaks things".

[bitdancer]

Right, but unless I miss my guess (I haven't looked at the code) it isn't that the concrete api is being used by the constructor, it's that the concrete API is being used by the iterator protocol called by the constructor. If I'm wrong it would be sensible to fix the constructor and we can reopen this. Even if I'm right perhaps there's a way to fix the constructor, but in that case it would be best addressed in the context of that issue.

[DavidMacIver]

So as a data point, this problem seems to be unique to tuple. set(x), list(x), tuple(iter(x)) all seem to work as expected and respect the overridden __iter__ (set and list were both included in the test case I attached to demonstrated this, iter I just checked right now). This suggests that the problem is in what tuple is doing, not in some general iterator protocol,

[stub]

Can we get this reopened? As David MacIver points out, this seems entirely a wart in tuple's constructor (compared to all the other builtin types), whereas 10977 is worrying about how 3rd party code using the C API can corrupt subclasses of builtin types (a much larger scope, and much less likely to be resolved in a good way).

Does it make sense to require python code wishing to case a tuple or tuple subclass do so using tuple(list(o)), or should tuple(o) work as expected? The primary use is of course converting a mutable sequence to an immutable representation to use as a dict key.

[rhettinger]

Added a patch. Needs to have the OP's test case added.

[serhiy-storchaka]

And it would be nice to add the same test for list, set, etc (if they don't exist).

[serhiy-storchaka]

Raymond's patch LGTM. Here is updated patch with tests.