Fix range slicing and indexing to handle lengths > sys.maxsize

[ncoghlan]

BPO	10889
Nosy	@birkenfeld, @mdickinson, @ncoghlan
Files	issue10889_range_subscripts.diff: Indexing and slicing for large ranges

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/birkenfeld'
closed_at = <Date 2011-01-12.03:22:45.097>
created_at = <Date 2011-01-11.15:59:27.770>
labels = []
title = 'Fix range slicing and indexing to handle lengths > sys.maxsize'
updated_at = <Date 2011-01-12.07:53:56.689>
user = 'https://github.com/ncoghlan'

bugs.python.org fields:

activity = <Date 2011-01-12.07:53:56.689>
actor = 'mark.dickinson'
assignee = 'georg.brandl'
closed = True
closed_date = <Date 2011-01-12.03:22:45.097>
closer = 'ncoghlan'
components = []
creation = <Date 2011-01-11.15:59:27.770>
creator = 'ncoghlan'
dependencies = []
files = ['20355']
hgrepos = []
issue_num = 10889
keywords = ['patch']
message_count = 6.0
messages = ['126017', '126023', '126027', '126029', '126030', '126069']
nosy_count = 3.0
nosy_names = ['georg.brandl', 'mark.dickinson', 'ncoghlan']
pr_nums = []
priority = 'normal'
resolution = 'accepted'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue10889'
versions = ['Python 3.2']

[ncoghlan]

Enhancement to range to correctly handle indexing and slicing when len(x) raises OverflowError.

Note that this enables correct calculation of the length of such ranges via:

        def _range_len(x):
            try:
                length = len(x)
            except OverflowError:
                step = x[1] - x[0]
                length = 1 + ((x[-1] - x[0]) // step)
            return length

[ncoghlan]

Having started work on this, the code changes are probably too significant to consider adding it to 3.2 at this late stage.

Writing my own slice interpretation support which avoids the ssize_t limit is an interesting exercise :)

[ncoghlan]

Attached patch moves range indexing and slicing over to PyLong and updates the tests accordingly.

Georg, I think this really makes the large range story far more usable - if you're OK with it, I would like to check it in this week so it lands in 3.2.

[ncoghlan]

Oh, and to explain my negative comment from earlier: that was my reaction when I realised I also needed to write PyLong versions of _PyEval_SliceIndex and PySlice_GetIndicesEx to make range slicing with large integers work properly.

As it turned out, the end result wasn't as scary as I initially feared (while compute_slice_indices is quite long, most of that is just the verbosity of PyLong arithmetic).

[birkenfeld]

It's a moderate chunk of code, but lots of new tests... I'd say go for it.

[ncoghlan]

Committed as r87948.

I added a few large_range tests to those in the patch. I checked that IndexError is raised when appropriate, as well as a specific test for the combination of a large range with a large negative step.