Add list.clear() and list.copy()

[terryjreedy]

BPO	10516
Nosy	@birkenfeld, @rhettinger, @terryjreedy, @ncoghlan, @ericvsmith, @giampaolo, @merwok, @briancurtin, @IIIIllllIIIIllllIIIIllllIIIIllllIIIIll
Files	issue10516.4.patch issue10516.5.patch unnamed unnamed issue10516.bytearray.1.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/rhettinger'
closed_at = <Date 2011-03-03.18:22:21.841>
created_at = <Date 2010-11-23.23:37:44.335>
labels = ['interpreter-core', 'easy', 'type-feature']
title = 'Add list.clear() and list.copy()'
updated_at = <Date 2012-03-17.13:16:28.083>
user = 'https://github.com/terryjreedy'

bugs.python.org fields:

activity = <Date 2012-03-17.13:16:28.083>
actor = 'python-dev'
assignee = 'rhettinger'
closed = True
closed_date = <Date 2011-03-03.18:22:21.841>
closer = 'eli.bendersky'
components = ['Interpreter Core']
creation = <Date 2010-11-23.23:37:44.335>
creator = 'terry.reedy'
dependencies = []
files = ['19945', '20056', '20886', '20887', '20905']
hgrepos = []
issue_num = 10516
keywords = ['patch', 'easy']
message_count = 56.0
messages = ['122251', '122252', '122256', '122513', '122515', '122516', '122517', '122520', '122524', '123321', '123322', '123325', '123326', '123353', '123354', '123400', '123401', '123402', '123411', '123416', '123417', '123431', '123453', '123469', '123470', '123480', '123481', '123491', '123494', '123682', '123723', '124037', '129237', '129243', '129246', '129249', '129250', '129251', '129332', '129336', '129342', '129344', '129345', '129352', '129358', '129359', '129365', '129406', '129408', '129440', '129522', '129739', '129740', '129746', '129987', '156155']
nosy_count = 13.0
nosy_names = ['georg.brandl', 'rhettinger', 'terry.reedy', 'ncoghlan', 'eric.smith', 'giampaolo.rodola', 'eric.araujo', 'Retro', 'eli.bendersky', 'brian.curtin', 'ysj.ray', 'xuanji', 'python-dev']
pr_nums = []
priority = 'normal'
resolution = 'accepted'
stage = 'patch review'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue10516'
versions = ['Python 3.3']

[terryjreedy]

Add list.clear() method with obvious semantics.

Pro:

1. parallel to set/dict/defaultdict/deque.clear(),
   usable in generic mutable collection function;
2. makes it easier to switch between list and other collection class;
3. current alternatives are not as obvious;
4. some people seem to expect it.

Anti:

1. unneeded; del l[:] or l[:]=[] do same already.

Guido: (python-ideas list, 'Set Syntax' thread, today)
"FWIW I'm fine with adding list.clear() to 3.3."

[ericvsmith]

Guido's email is archived at:
http://mail.python.org/pipermail/python-ideas/2010-November/008732.html

[rhettinger]

Guido approved these both in a thread earlier this year.

The reasoning for copy() was the same as for clear(), some folks couldn't cope with:

   b = a[:]

[elibendersky]

Objects/listobject.c has a static function named list_clear used internally. Is it possible to just expose this function as a clear() method?

One problem is that it has this strange comment in the end:

/* Never fails; the return value can be ignored.
   Note that there is no guarantee that the list is actually empty
   at this point, because XDECREF may have populated it again! */

However, looking at the code I'm not sure the list can be cleared any more than the function does, and it actually deallocates the ob_item field of the list.

[IIIIllllIIIIllllIIIIllllIIIIllllIIIIll]

Hi, I'm also looking at listobject.c also... if we want list.clear() to behave exactly like del list[], we may be able to just call list_ass_slice on the list. Similarly for list.copy which should behave like a=l[:]

[elibendersky]

Hi, I'm also looking at listobject.c also... if we want list.clear() to
behave exactly like del list[], we may be able to just call list_ass_slice
on the list. Similarly for list.copy which should behave like a=l[:]

Note that when executed to do 'del lst[:]' (i.e. with argument v set to 0
and ilow/ihigh to the maximal range of the list), list_ass_slice will just
call list_clear anyway, which is a cue that this indeed is the right way to
do it, despite the strange comment I mentioned in my message above.

[birkenfeld]

Yes, list_clear should be called, but no, it cannot be used directly because a method needs a PyObject* return value. So a wrapper method is needed that looks like listappend() does for list.append(). list_copy() will just look like list_slice() with the index fiddling removed.

[IIIIllllIIIIllllIIIIllllIIIIllllIIIIll]

That's good if it's so... can you explain why list_clear doesn't guarantee that the list is empty? Why would XDECREF populate the list? I don't quite understand it.

eli: are you writing a patch for this?

[elibendersky]

Attaching a patch for list.clear():

1. Implements a new function in Objects/listobject.c named listclear() (to be consistent with the other "method functions")
2. listclear() is registered in list_methods and just calls list_clear(), returning None
3. A documentation string is modeled after dict.clear(), but shaped a bit differently to follow the conventions of other list method docs.

If this look fine to the more experienced devs, things left to do are:

1. Add tests
2. Implement the .copy() method in a similar manner + tests for it

Some random observations:

1. The naming of functions/methods could be made more consistent. See, for example, list_reversed vs. listreverse.
2. The documentation style of list and dict methods is different for no apparent reason:

help({}.pop) gives:

pop(...)
    D.pop(k[,d]) -> v, remove specified key and return the corresponding value.
    If key is not found, d is returned if given, otherwise KeyError is raised

While help([].pop) gives:

pop(...)
    L.pop([index]) -> item -- remove and return item at index (default last).
    Raises IndexError if list is empty or index is out of range.

Note the '--' which separates the signature from description in the list version.

[elibendersky]

Was list.copy() also approved? After all, there are many ways to achieve the same even now:

1. L[:]
2. list(L)
3. import copy and then copy.copy

Especially re the last one: list.copy() can be deep or shallow, which one should it be?

[elibendersky]

Also, where is the *official* place to document list objects and their methods?

[birkenfeld]

Yes, list.copy was also approved IIRC. And it should be a shallow copy, like all other copy methods on builtins.

[Retro]

This is really welcome. It makes Python even more readable.

If 'a' is a list object, a[:] is not so obvious at first to a newcomer, but
a.copy() is.

Also, a.clear() is so perfect and understandable. I wish you could decorate Python versions prior to 3.3 with this two new list methods.

[elibendersky]

Attaching a patch with the following:

1. list.copy() and list.clear() implemented in Objects/listobject.c, with appropriate doc strings (modeled after dict's docstrings)
2. Same methods implemented in collections.UserList
3. Tests added that exercise the methods in both list and UserList

Re the documentation, it's currently unclear where it should go. I asked on docs@python.org.

[merwok]

Hi Eli,

I think the right place is 4.6.4,
http://docs.python.org/dev/library/stdtypes#mutable-sequence-types

It starts with “List and bytearray objects support additional operations
that allow in-place modification of the object”.

For methods not supported by bytearray, you can use the fake footnote
(8) and edit its texte (“sort() is not supported by bytearray objects”).

Regards

[elibendersky]

Following Éric's suggestion, I'm attaching an updated patch with with the documentation in Doc/library/stdtypes.rst updated with the new methods.

There seems to be a slight Sphinx markup problem with this addition. I rewrote note (8) as:

:meth:`clear`, :meth:`copy` and :meth:`sort` are not supported by
:class:`bytearray` objects.

Unfortunately, :meth:`copy` is getting linked to the copy module.

[rhettinger]

Nothing will happen on this until 3.2 is done and the py3k branch starts with 3.3 submissions.

[merwok]

Eli, I learned this trick recently: :meth:`!copy`.

[elibendersky]

Éric - thanks, it works [attaching updated patch]. However, don't you think the core problem is a Sphinx bug we should report?

Raymond - this happens after final 3.2 release (on Feb 05 2011 if it's on schedule), right?

[Retro]

I'm troubled with one little letter:

"L.copy() -> list -- a shallow copy of L"); should be

"L.copy() -> list -- shallow copy of L"); without the letter 'a',
because other sentences also don't say "L.__sizeof__() -- *A* size of
L in memory, in bytes");

Please fix this.

[Retro]

Can you please help me find the definition of the copy() method of dict in
the Python sources? I want to see how that method is defined and compare the
definition to the one in Eli's patch.

[terryjreedy]

Objects/dictobject.c

[elibendersky]

Boštjan,

"a shallow copy": I took this directly from the documentation of dicts, which says:

"D.copy() -> a shallow copy of D")

As I mentioned in an earlier message, the doc-strings of list and dict methods are inconsistent in more than one way, so I'm going to leave this decision to the committer. I'll be happy to help with fixes too.

Re your other question, in the Python source root, dictionaries are mostly implemented in Objects/dictobject.c - there's an array called mapp_methods that lists the functions used to implement relevant methods. For copy() it lists:

{"copy",            (PyCFunction)dict_copy,         METH_NOARGS,

So you need dict_copy. Note that it's just a wrapper (of another wrapper, by the way) bit it's a good place to start. Arm yourself with an editor or IDE with some code-searching capabilities.

[Retro]

mapp_methods ? Don't you mean map_methods ?

[birkenfeld]

No, he means mapp_methods. Why don't you simply look at the file?

[ysjray]

Please modify the patch so that it can be applied to current py3k trunk cleanly. (Notice that Lib/collections.py has been changed to a package in bpo-11085).

[merwok]

Ray: Eli can just refresh his patch and commit. Note that the patch program will prompt you for a file name if it can’t find the file for a diff hunk, so it should be trivial to apply a patch across a rename.

Eli: Would you mind changing two nits before committing the nice patch?

+"L.clear() -> None -- remove all items from L");
It looks like other methods that return None just omit the “-> type” part.

+

+
That’s one new line too many. Some trailing spaces are also included.

[rhettinger]

Eli doesn't need to post a new patch. I'm sure he will fix any nits in his commit.

[elibendersky]

On Thu, Feb 24, 2011 at 05:26, Éric Araujo <report@bugs.python.org> wrote:

Éric Araujo <merwok@netwok.org> added the comment:

Ray: Eli can just refresh his patch and commit.  Note that the patch program will prompt you for a file name if it can’t find the file for a diff hunk, so it should be trivial to apply a patch across a rename.

Eli: Would you mind changing two nits before committing the nice patch?

+"L.clear() -> None -- remove all items from L");
It looks like other methods that return None just omit the “-> type” part.

That’s one new line too many.  Some trailing spaces are also included.

I will fix and commit over the weekend.

[rhettinger]

[Éric Araujo]

+"L.clear() -> None -- remove all items from L");
It looks like other methods that return None
just omit the “-> type” part.

These kind of nitty comments really aren't helpful.
It consumes more time to talk about them than they're worth.
In this case, Eli was modeling after the docstring in dictobject.c:

PyDoc_STRVAR(clear__doc__,
"D.clear() -> None. Remove all items from D.");

Just because list.remove.__doc__ failed to consistently follow that convention doesn't make Eli's patch incorrect.

[elibendersky]

A slightly revised patch committed in revision 88554:

1. Fixed Éric's whitespace comment
2. Fixed a test in test_descrtut.py which was listing list's methods
3. Moved the change to collections.py onto Lib/collections/init.py
4. Added NEWS entry

Éric - as I mentioned earlier in this issue, I chose to leave the syntax of the docstring for the new methods similar to the same methods in dict (dict docstring look better and more internally consistent anyhow). I propose to move further discussion of this matter into a separate issue which will deal with the overall (in)consistency in the docstrings of list and dict.

[birkenfeld]

Reading "clear and copy are not supported by bytearray": shouldn't they be? ("sort" probably really makes no sense on bytearrays.)

[elibendersky]

On Fri, Feb 25, 2011 at 10:11, Georg Brandl <report@bugs.python.org> wrote:

Georg Brandl <georg@python.org> added the comment:

Reading "clear and copy are not supported by bytearray": shouldn't they be?

Perhaps they should, and it's not a big deal to implement. But I'm not
100% clear on the policy of how such changes are approved. Should this
be discussed in the list?

[rhettinger]

Unless someone raises a controversial and non-trivial issue about adding clear() and copy() to bytearray, there is no need for a python-dev discussion on the subject. Just post a patch to the tracker.

[birkenfeld]

Yes, it should be discussed on python-dev.

[birkenfeld]

In any case, this issue can be closed.

[Retro]

I have installed Python 3.2 final on my Windows machine and I get an
exception when doing list.copy or list.clear in the interpreter. Why is that
so?

[birkenfeld]

Because they got added *after* 3.2 was released?

[Retro]

Right, right. My bad. Can't wait for Python 3.3! ;)

[rhettinger]

Georg, what is the issue? Is there some reason that bytearrays should not be copied or cleared? Is there some reason to prefer the current:

dup = b[:] # copy
del b[:] # clear

[birkenfeld]

No -- but the question is if copy() and clear() mightn't be added to the (mutable) sequence ABC if we make all builtin such sequences implement them.

[rhettinger]

The ABCs are subset of the methods for the concrete APIs. We've avoided the likes of copy() because it requires knowledge of the constructor's signature -- for example, MutableMapping does not cover copy().

It is okay for Eli to add MutableSequence.clear() because it can be implemented in terms of pop(), much like we do for MutableMapping.clear().

Eli, feel free to create a patch to add clear() and copy() to bytearray and to add clear() to MutableSequence. Assign the patch to me for review.

[elibendersky]

Attaching a patch adding copy() and clear() to bytearrays, with tests and doc.

I didn't add the methods to MutableSequence because I have a doubt about it - in particular which exception get raised by .pop if it's empty. Curiously, lists and bytearrays raise different exceptions in this case - IndexError and OverflowError, respectively.

[rhettinger]

The patch is fine. Do consider using assertIsNot() in the tests. Then go ahead and apply it.

The OverflowError in bytearray.pop() is a bug, please open a separate report for it and make a patch changing it to IndexError. Assign that bug report to me.

Go ahead and propose a patch for MutableSequence.clear() implemented with MutableSequence.pop() and catching an IndexError when empty.

Thanks for your efforts.

[birkenfeld]

Hmm, shouldn't self.__class__(self) be a good default implementation of copy()?

I'd expect any sequence to support this way of creation from another sequence, even if it's inefficient.

[rhettinger]

Hmm, shouldn't self.__class__(self) be a
good default implementation of copy()?

I'd expect any sequence to support this way
of creation from another sequence, even if it's inefficient.

The copy() method isn't being proposed for MutableSequence because it presumes that we know something about the constructor's signature. For example, the constructor of array() needs the element storage type as an argument. We refuse the temptation to guess.

In the Set api, we had no choice because many set-methods necessarily create a new set. To handle the constructor signature problem, the creation step was factored-out into the from_iterable() method so that a user could override it if necessary.

Also copy() is handled oddly in the builtin types. To handle the constructor signature issue for subclasses, they ignore the subclass and return a instance of the base class. For example, the inherited copy() method on a subclass of list or dict will create an instance of list or dict, not of the subclass itself. Accordingly, subclasses that want instances of themselves have to override the inherited copy() method. They would have to do this anyway if they subclass contained any other data in the class dictionary that would need to be passed along to a copy.

In short, we're better-off not supplying copy() as part of the MutableSequence ABC.

[elibendersky]

Committed the bytearray methods in revision 88733.

Closing this issue. Will handle wrong exception and MutableSequence ABC method addition in separate issues.

[python-dev]

New changeset 958a98bf924e by Eli Bendersky in branch 'default':
updated whatsnew/3.3.rst with the new methods added to list and bytearray (bpo-10516)
http://hg.python.org/cpython/rev/958a98bf924e