__getitem__ and __setitem__ try to be smart when invoked with negative slice indices

[kt]

BPO	21785
Nosy	@rhettinger, @serhiy-storchaka, @eryksun

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-05-31.14:43:04.028>
created_at = <Date 2014-06-17.05:16:03.404>
labels = ['interpreter-core', 'type-bug']
title = '__getitem__ and __setitem__ try to be smart when invoked with negative slice indices'
updated_at = <Date 2020-05-31.14:43:04.027>
user = 'https://bugs.python.org/kt'

bugs.python.org fields:

activity = <Date 2020-05-31.14:43:04.027>
actor = 'serhiy.storchaka'
assignee = 'docs@python'
closed = True
closed_date = <Date 2020-05-31.14:43:04.028>
closer = 'serhiy.storchaka'
components = ['Interpreter Core']
creation = <Date 2014-06-17.05:16:03.404>
creator = 'kt'
dependencies = []
files = []
hgrepos = []
issue_num = 21785
keywords = []
message_count = 6.0
messages = ['220792', '220795', '220806', '220808', '220809', '370468']
nosy_count = 5.0
nosy_names = ['rhettinger', 'docs@python', 'serhiy.storchaka', 'eryksun', 'kt']
pr_nums = []
priority = 'low'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue21785'
versions = ['Python 2.7']

[kt]

Consider the following example:

class A:
     def __getitem__(self, index):
         return True

If you invoke A()[-1], everything is fine. However, if you invoke A()[-1:2], you get an "AttributeError: A instance has no attribute '__len__'".

Moreover, if you define __len__ for your class, you will discover that __getitem__ will act "smart" and modify slice you are passing into the function. Check this out:

class A:
    def __getitem__(self, index):
        return index.start
    def __len__(self):
        return 10

Now A()[-1:10] outputs "9". The same kind of argument-mangling happens within __setitem__.

This is completely unintuitive and contrary to what I read in the docs (https://docs.python.org/2/reference/datamodel.html#object.\_\_getitem__):
"Note that the special interpretation of negative indexes (if the class wishes to emulate a sequence type) is up to the __getitem__() method.".

Especially intuitive is the fact that if you do A()[slice(-1,10)] or A().__getitem__(slice(-1,10)), no special treatment is done for the -1, everything works fine and the __len__ method is not invoked.

As far as I understand, the root cause is the behaviour of STORE_SLICE+3 command, which tries to be too smart.

I have discovered this within code where slice indexing was used to insert arbitrary intervals into a data structure (hence using negative numbers would be totally fine), and obviuosly such behaviour broke the whole idea, albeit it was nontrivial to debug and discover.

This does not seem to be a problem for Python 3.3, however I believe fixing this in Python 2.7 is important as well.

[rhettinger]

It's too late for a behavior change in 2.7. That risks breaking code that relies on the current behavior.

However, the docs could be amended to indicate that slices with negative indicies are treated differently from scalar negative indicies, the former get length adjusted automatically and the latter don't.

[kt]

Do note that things are not as simple as "slices with negative indices are treated differently from scalar negative indicies".

Namely, behaviour differs whether you use [] or .__getitem__, and whether you use [a:b] or [slice(a,b)]. This does not make sense from a specification perspective, but has to be made clear in the docs then.

Besides, Jython does not have this problem and I presume other Python implementations might also be fine (e.g. PyPy or whatever else there exists, couldn't test now).

Hence, although fixing the docs does seem like a simple solution, if you want to regard the docs as a "specification of the Python language" rather than a list of particular CPython features, this won't be reasonable.

[eryksun]

Refer to the documentation for deprecated __getslice__ when slicing an instance of a classic class:

https://docs.python.org/2/reference/datamodel.html#object.\_\_getslice__

The SLICE+3 implementation (apply_slice) calls PySequence_GetSlice if both index values can be converted to Py_ssize_t integers and if the type defines sq_slice (instance_slice for the "instance" type). The "instance" type is used for an instance of a classic class. This predates unification of Python classes and types.

apply_slice
http://hg.python.org/cpython/file/f89216059edf/Python/ceval.c#l4383

PySequence_GetSlice
http://hg.python.org/cpython/file/f89216059edf/Objects/abstract.c#l1995

instance_slice
http://hg.python.org/cpython/file/f89216059edf/Objects/classobject.c#l1177

A new-style class, i.e. a class that subclasses object, would have to define or inherit __getslice__ in order for the C sq_slice slot to be defined. But __getslice__ is deprecated and shouldn't be implemented unless you have to override it in a subclass of a built-in type.

When sq_slice doesn't exist, apply_slice instead calls PyObject_GetItem with a slice object:

    class A(object):
        def __getitem__(self, index):
            return index.start
        def __len__(self):
            return 10

    >>> A()[-1:10]
    -1

By the way, you don't observe the behavior in Python 3 because it doesn't have classic classes, and the __getslice__, __setslice__, and __delslice__ methods are not in its data model.

[kt]

Aha, I see. I knew I'd get bitten by not explicitly subclassing (object) one day.

In any case, adding a reference to this issue into the docs of __getitem__ and __setitem__ would probably save someone some hours of utter confusion in the future.

[serhiy-storchaka]

Python 2.7 is no longer supported.