experimental support for extended slicing on lists

[mwhudson]

BPO	400998
Nosy	@mwhudson, @gvanrossum, @tim-one, @arigo
Files	ext-slice-23.diff: minimally updated patch for 2.3 ext-slice-23-2.diff: hopefully final patch None: None

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/mwhudson'
closed_at = <Date 2003-02-27.15:00:40.000>
created_at = <Date 2000-07-27.21:27:53.000>
labels = ['interpreter-core']
title = 'experimental support for extended slicing on lists'
updated_at = <Date 2003-02-27.15:00:40.000>
user = 'https://github.com/mwhudson'

bugs.python.org fields:

activity = <Date 2003-02-27.15:00:40.000>
actor = 'mwh'
assignee = 'mwh'
closed = True
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2000-07-27.21:27:53.000>
creator = 'mwh'
dependencies = []
files = ['2661', '2662', '2663']
hgrepos = []
issue_num = 400998
keywords = ['patch']
message_count = 32.0
messages = ['33554', '33555', '33556', '33557', '33558', '33559', '33560', '33561', '33562', '33563', '33564', '33565', '33566', '33567', '33568', '33569', '33570', '33571', '33572', '33573', '33574', '33575', '33576', '33577', '33578', '33579', '33580', '33581', '33582', '33583', '33584', '33585']
nosy_count = 4.0
nosy_names = ['mwh', 'gvanrossum', 'tim.peters', 'arigo']
pr_nums = []
priority = 'normal'
resolution = 'accepted'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue400998'
versions = []

[mwhudson]

No description provided.

[mwhudson]

this 10 minute hack adds support for "extended slicing" to lists, by which I mean things like:

>>> range(100)[23:60:12]
[23, 35, 47, 59]

todo:
find out if this is the approved approach
add support for tuples
write docs, testsuite (make test passes as is, but should be extended).

[mwhudson]

c'mon michael! at least get *some* of the boundary cases...

[mwhudson]

negative indices!

for i in listvar[::-1]:
     pass

now iterates over the list backwards (rather than coredumping...)

[mwhudson]

updated patch to support slice assignements, deletions
(needs testing still)

[mwhudson]

bugfixes (refounting in assignment, logic in deletion) & a few tweaks.

[mwhudson]

meep! naughty Michael hadn't been running "make test" often enough.
this update fixes that.

[tim-one]

Note that without doc and test patches too, and Real Soon, I'll have to Postpone this. Assigned to me to remind me of that.

[mwhudson]

OK, I'll get to that soon (ie. <24hours, hopefully <4).
But bear in mind I'm now going to the pub...

I may need some advice for the docs...

[mwhudson]

So, I missed my deadline by twenty five minutes. Sorry :-)

This latest patch adds a fairly basic test suite, a stab at some docs, and jumps up and down on PySlice_GetIndices.

Also (wrt. stringobject.c & unicodeobject.c) it's amazing what you can break, isn't it? Thank God for test suites...

[mwhudson]

So, I missed my deadline by twenty five minutes. Sorry :-)

This latest patch adds a fairly basic test suite, a stab at some docs, and jumps up and down on PySlice_GetIndices.

Also (wrt. stringobject.c & unicodeobject.c) it's amazing what you can break, isn't it? Thank God for test suites...

[gvanrossum]

Too late for the release, sorry. And Tim hates negative strides.

We'll get back to this for 2.1.

[gvanrossum]

Is reviewing and applying this patch the best use of my time? I note that this is okay, but low priority -- I doubt that it will get used much. Plus, what about the array module, strings, UserList?

[gvanrossum]

Rejected for lack of interest.

Sorry, Michael!

[gvanrossum]

Logged In: YES
user_id=6380

Michael, the tide has changed. Would you be interested in
reviving this patch and finishing it?

It could possibly help to address bug 473985 and maybe also
459235.

If you have no time, let me know and I'll see if I can find
time myself.

[mwhudson]

Logged In: YES
user_id=6656

Whoosh, this is a blast from the past.

I've minimally updated the patch (it only failed to apply
for the trivialest of reasons). All tests pass (I had to
tweak test_bool -- operator.isMappingType([]) is True now).

Suffice to say I don't really remember how this works :-/

What more needs to be done? strings, arrays (shouldn't be
too hard).

[gvanrossum]

Logged In: YES
user_id=6380

Go ahead and check it in.

Strings and unicode also need to support slices. While
"hello"[::2] is silly, it's useful to be able to write
"hello"[slice(1,-1)].

I think array needs more work -- first 'array.array' should
be made into a type object that can be invoked as a
constructor and subclassed. That would be nice, yes, but is
a lower priority for me ATM.

Off-topic aside: the change you had to make to
PyString_Format() and its Unicode cousin suggests that we
need a better way to decide if something is really a
mapping... E.g. this now gives a confusing error message:
"%(foo)s %(bar)s" % [1,2,3]
I wonder how many other places in the code make naive
mapping tests (and of course operator.isMappingType becomes
totally unreliable). Maybe the presence of a keys() method
together with a __getitem__ method should be used as a
standard test for mapping-ness?

[gvanrossum]

Logged In: YES
user_id=6380

(I lied about array -- it is already a subclassable
new-style type. So go ahead, but do that one last.)

[mwhudson]

Logged In: YES
user_id=6656

Hmm, I'm now older and more cowardly than when I wrote this,
and I have a couple of questions.

This patch changes the behaviour of PySlice_GetIndices,
which must count as a public API function (albeit a pretty
useless one as it is now). Is that OK? Or would it be
better to have a new function PySlice_GetIndicesEx or
something. Should check whether NumPy uses this function.

Do you really want assignments to extended slices in lists?
They were never that inuitive to me.

[gvanrossum]

Logged In: YES
user_id=6380

This patch changes the behaviour of PySlice_GetIndices,
which must count as a public API function (albeit a pretty
useless one as it is now). Is that OK? Or would it be
better to have a new function PySlice_GetIndicesEx or
something. Should check whether NumPy uses this function.

The big change is that it now can set an exception. I think
that's too big a change without changing the name.
PySlice_GetIndicesEx is fine.

You can call PyInt_AsLong on any object, and if it has a
tp_int or __int__ it will do the right thing, so it's better
not to call PyInt_Check or PyLong_Check.

The check for *step==0 could come earlier.

Do you really want assignments to extended slices in
lists?
They were never that inuitive to me.

They're about as useful as getting an extended slice from a
list, so I'd say yes. They're soooooo cute! :-)

[gvanrossum]

Logged In: YES
user_id=6380

I checked the Numeric-21.0 source, and they don't use
PySlice_GetIndices. They have their own function,
slice_GetIndices, which appears to be a reformatted copy of
PySlice_GetIndices. except that the fiddling at the end is
different: it truncates out-of-bounds indices rather than
calling them errors, and it doesn't call step==0 an error
(why?).

I also checked the numarray-0.3.3 source, and they don't do
slices yet.

So I think we're safe fixing PySlice_GetIndices.

However, I think there's a bug in your code. Suppose a is
range(10). Then a[1000:] returns []. But a[1000::] returns
[9]! Similarly, a[-1000::-1] return [0] where I would expect
[]. I think the start/stop truncation needs to depend on the
step, too.

[mwhudson]

Logged In: YES
user_id=6656

Hmm. In my working copy, PySlice_GetIndicesEx now has a
different interface (it computes the length of the slice, as
that's fiddly and almost always required). So I may remain
cowardly here. As a bonus, I'll document both functions in
Doc/api/.

Yes, I was staring at my code and thinking "that can't get
all the end cases, can it?". Almost a relief to hear it
doesn't :) I'll get pen and paper out.

[gvanrossum]

Logged In: YES
user_id=6380

OK, sounds good. (BTW, a Google search for
PySlice_GetIndices and Numeric turned up a message
inpython-dev from you asking about a broken endcase in
Numeric slices, with a question about who uses it in a PS.
Of course there was no answer. The broken endcase made me
try a few things with your patch. :-)

[mwhudson]

Logged In: YES
user_id=6656

That message was from me :)

[gvanrossum]

Logged In: YES
user_id=6380

Yes, I said so ("from you"). :) That's why it was so ironic
that you had a similar endcase bug. :-)

[mwhudson]

Logged In: YES
user_id=6656

Doh, it's me that can't read, not you...

[mwhudson]

Logged In: YES
user_id=6656

OK, I think the attached gets all the cases right.

I've also implemented extended slicing for strings and
unicode strings & documented the C API functions.

I think this is ready to go, at last.

[gvanrossum]

Logged In: YES
user_id=6380

Wow. Good job on the docs! (Although personally I would have
added the footnote at the end to keep the existing note
numbering :-).

In test_bool.py,
#veris(operator.isMappingType([]), False)
should either be deleted or a different sample used, e.g.
veris(operator.isMappingType(1), False)

Some lines added to listobject.c (e.g. in
list_[ass_]subscript()) are longer than 79 characters.

But please do check it in, correcting the nits as you go!

[mwhudson]

Logged In: YES
user_id=6656

Re footnotes: wimp!

Re long lines: there's still one longish line in
listobject.c that I really couldn't see how to wrap nicely.
All the others got trimmed.

But this is now checked in (with mild modfications from the
posted patch) as (deep breath):

Doc/api/concrete.tex revision 1.16
Doc/lib/libstdtypes.tex revision 1.96
Doc/ref/ref3.tex revision 1.91
Doc/whatsnew/whatsnew23.tex revision 1.24
Include/sliceobject.h revision 2.6
Lib/test/test_bool.py revision 1.6
Lib/test/test_types.py revision 1.30
Misc/NEWS revision 1.423
Objects/listobject.c revision 2.109
Objects/sliceobject.c revision 2.13
Objects/stringobject.c revision 2.166
Objects/tupleobject.c revision 2.65
Objects/unicodeobject.c revision 2.152

[arigo]

Logged In: YES
user_id=4771

a very far-fetched bug...

>>> slice(-3058, 0, -1).indices(20)
(-1, 5, -1)
>>> slice(-30589138798749473891743819L, 0, -1).indices(20)
(0, 5, -1)

this is due to _PyEval_SliceIndex() which "rounds" very large negative
values up to 0. Of course I don't know who would care, because in both
cases 'range(*slice(...).indices(20))' returns an empty list.

[mwhudson]

Logged In: YES
user_id=6656

a comedy of errors!

i'll fix this soon.

[mwhudson]

Logged In: YES
user_id=6656

Heh. I'd actually intended to change _PyEval_SliceIndex to
round up to -INT_MAX but I'd botched it.

Fixed in Python/ceval.c revision 2.352.