Make itertools.islice supports negative values for start and stop arguments for sized iterable object

[TitanSnow]

BPO	33040
Nosy	@rhettinger, @serhiy-storchaka, @titansnow, @ynikitenko
Files	islice.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-03-10.16:58:30.116>
created_at = <Date 2018-03-10.09:18:46.898>
labels = ['type-feature', 'library']
title = 'Make itertools.islice supports negative values for start and stop arguments for sized iterable object'
updated_at = <Date 2021-04-28.19:16:14.128>
user = 'https://github.com/TitanSnow'

bugs.python.org fields:

activity = <Date 2021-04-28.19:16:14.128>
actor = 'ynikitenko'
assignee = 'none'
closed = True
closed_date = <Date 2018-03-10.16:58:30.116>
closer = 'rhettinger'
components = ['Library (Lib)']
creation = <Date 2018-03-10.09:18:46.898>
creator = 'tttnns'
dependencies = []
files = ['47475']
hgrepos = []
issue_num = 33040
keywords = ['patch']
message_count = 6.0
messages = ['313517', '313519', '313545', '313636', '392246', '392249']
nosy_count = 4.0
nosy_names = ['rhettinger', 'serhiy.storchaka', 'tttnns', 'ynikitenko']
pr_nums = []
priority = 'normal'
resolution = 'rejected'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue33040'
versions = []

[TitanSnow]

islice() does not support negative values for start or stop,
which does not matter for plain iterators.
However, for some cases, we have a sized iterable object
which is not subscriptable, using islice() makes
code ugly::

    d = OrderedDict()
    for i in range(10): d[i] = i
    dv = d.keys()
    # dv is a KeysView which is a sized iterable

# now I wanna get a slice of dv which does not contain the last element
islice(dv, len(dv) - 1)

As it shows, I have to use len(dv) to get its length.

For sized iterable objects, islice() could
support negative values for start or stop.
In this way, the above code can be written like this::

    islice(dv, -1)

For iterable objects which is not sized,
it could still be not supported::

islice(iter(range(10)), -1)

raises a ValueError as its original behavior.

[serhiy-storchaka]

I do not think this is suitable for the itertools module. It is designed to work with iterators, i.e. with objects that support the iterator protocol. What if the iterable change the size during iteration? Should the islice iterator produce less or larger number of items than the initial estimation? Or detect this change and raise an error? Should it revive the iteration if new items were added after consuming the last item? These design questions should have thoughtful answers. And handling all corner cases will complicate the code. I afraid that after adding support of negative start and stop we will get a request for supporting negative step.

It is possible to implement support of negative indices for arbitrary iterators using tee() or deque, but this will complicate the islice() code even more for a small niche case.

This may be a use case for the new module that works specially with sequences. It will be added once we will have enough use cases.

[rhettinger]

I concur with Serhiy. This doesn't make sense for itertools.

Marking this as closed. Thank you for the suggestion.

[TitanSnow]

Now I have thought about it and realized that
it's not suitable for islice.
But there's still a common use case to
drop some elements from the beginning or
ending of a iterable, which is also a main
reason why I wanted islice to support
negative values of start and stop.
So how about adding two functions
drop_first and drop_last
to do such things. They can be implemented
like this, in which way they support
arbitrary iterators::

    def drop_first(iterable, n=1):
        for _ in range(n):
            next(iterable)
        for e in iterable:
            yield e

    def drop_last(iterable, n=1):
        dq = deque()
        for _ in range(n):
            dq.append(next(iterable))
        for e in iterable:
            dq.append(e)
            yield dq.popleft()

[ynikitenko]

I hope it's fine to add to closed topics here.

I agree with the decision that islice should not handle a special case of sized containers vs iterables.

However, I think that the support of negative indices in islice would be nice. A simple use case would be to get the last element of an iterable.
I had to write my own code for that (you may read and/or use it if needed), because I needed this algorithm in my analysis. Here is the link to my function ISlice._run_negative_islice in my architectural framework for data analysis: https://github.com/ynikitenko/lena/blob/master/lena/flow/iterators.py#L150 or on https://lena.readthedocs.io/en/latest/flow.html#lena.flow.iterators.ISlice (I'm afraid the correct link might change later).

I also agree that to support negative indices would require more work. Indeed, it seems that for each combination of positive/negative start/stop it required a new algorithm! I didn't implement negative steps.

However, I think that because it's not easy to implement, it would be even better to include that in the standard library (because it would save other people from writing that).

If we care about code reuse: I think that negative indices make the algorithm more useful, while non-trivial steps are redundant, because they can be implemented by a composition of a slice with step=1 with a simple slice with start, stop=None, None with step!=1.

Negative slice fundamentally changes the algorithm: if one wants to have the flow inverted, fflow[0, None, -1], one would have to store it all in memory! Anyway it's easy to make this in a separate function. So I think it's more functional to implement negative indices and discard negative steps (or non-trivial steps at all).

About drop_first, drop_last suggested above: I also think that they are redundant and would be better incorporated into islice.

[ynikitenko]

Sorry for a typo. The paragraph before the last should read

Negative *step* fundamentally changes the algorithm:... flow[-1:None:-1].