Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add list.clear() and list.copy() #54725

Closed
terryjreedy opened this issue Nov 23, 2010 · 56 comments
Closed

Add list.clear() and list.copy() #54725

terryjreedy opened this issue Nov 23, 2010 · 56 comments
Assignees
Labels
easy interpreter-core Interpreter core (Objects, Python, Grammar, and Parser dirs) type-feature A feature request or enhancement

Comments

@terryjreedy
Copy link
Member

terryjreedy commented Nov 23, 2010

BPO 10516
Nosy @birkenfeld, @rhettinger, @terryjreedy, @ncoghlan, @ericvsmith, @giampaolo, @merwok, @briancurtin, @IIIIllllIIIIllllIIIIllllIIIIllllIIIIll
Files
  • issue10516.4.patch
  • issue10516.5.patch
  • unnamed
  • unnamed
  • issue10516.bytearray.1.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/rhettinger'
    closed_at = <Date 2011-03-03.18:22:21.841>
    created_at = <Date 2010-11-23.23:37:44.335>
    labels = ['interpreter-core', 'easy', 'type-feature']
    title = 'Add list.clear() and list.copy()'
    updated_at = <Date 2012-03-17.13:16:28.083>
    user = 'https://github.com/terryjreedy'

    bugs.python.org fields:

    activity = <Date 2012-03-17.13:16:28.083>
    actor = 'python-dev'
    assignee = 'rhettinger'
    closed = True
    closed_date = <Date 2011-03-03.18:22:21.841>
    closer = 'eli.bendersky'
    components = ['Interpreter Core']
    creation = <Date 2010-11-23.23:37:44.335>
    creator = 'terry.reedy'
    dependencies = []
    files = ['19945', '20056', '20886', '20887', '20905']
    hgrepos = []
    issue_num = 10516
    keywords = ['patch', 'easy']
    message_count = 56.0
    messages = ['122251', '122252', '122256', '122513', '122515', '122516', '122517', '122520', '122524', '123321', '123322', '123325', '123326', '123353', '123354', '123400', '123401', '123402', '123411', '123416', '123417', '123431', '123453', '123469', '123470', '123480', '123481', '123491', '123494', '123682', '123723', '124037', '129237', '129243', '129246', '129249', '129250', '129251', '129332', '129336', '129342', '129344', '129345', '129352', '129358', '129359', '129365', '129406', '129408', '129440', '129522', '129739', '129740', '129746', '129987', '156155']
    nosy_count = 13.0
    nosy_names = ['georg.brandl', 'rhettinger', 'terry.reedy', 'ncoghlan', 'eric.smith', 'giampaolo.rodola', 'eric.araujo', 'Retro', 'eli.bendersky', 'brian.curtin', 'ysj.ray', 'xuanji', 'python-dev']
    pr_nums = []
    priority = 'normal'
    resolution = 'accepted'
    stage = 'patch review'
    status = 'closed'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue10516'
    versions = ['Python 3.3']

    @terryjreedy
    Copy link
    Member Author

    terryjreedy commented Nov 23, 2010

    Add list.clear() method with obvious semantics.

    Pro:

    1. parallel to set/dict/defaultdict/deque.clear(),
      usable in generic mutable collection function;
    2. makes it easier to switch between list and other collection class;
    3. current alternatives are not as obvious;
    4. some people seem to expect it.

    Anti:

    1. unneeded; del l[:] or l[:]=[] do same already.

    Guido: (python-ideas list, 'Set Syntax' thread, today)
    "FWIW I'm fine with adding list.clear() to 3.3."

    @terryjreedy terryjreedy added interpreter-core Interpreter core (Objects, Python, Grammar, and Parser dirs) type-feature A feature request or enhancement labels Nov 23, 2010
    @ericvsmith
    Copy link
    Member

    ericvsmith commented Nov 23, 2010

    @rhettinger
    Copy link
    Contributor

    rhettinger commented Nov 24, 2010

    Guido approved these both in a thread earlier this year.

    The reasoning for copy() was the same as for clear(), some folks couldn't cope with:

       b = a[:]

    @rhettinger rhettinger changed the title Add list.clear() Add list.clear() and list.copy() Nov 24, 2010
    @elibendersky
    Copy link
    Mannequin

    elibendersky mannequin commented Nov 27, 2010

    Objects/listobject.c has a static function named list_clear used internally. Is it possible to just expose this function as a clear() method?

    One problem is that it has this strange comment in the end:

    /* Never fails; the return value can be ignored.
       Note that there is no guarantee that the list is actually empty
       at this point, because XDECREF may have populated it again! */
    

    However, looking at the code I'm not sure the list can be cleared any more than the function does, and it actually deallocates the ob_item field of the list.

    @IIIIllllIIIIllllIIIIllllIIIIllllIIIIll
    Copy link
    Mannequin

    IIIIllllIIIIllllIIIIllllIIIIllllIIIIll mannequin commented Nov 27, 2010

    Hi, I'm also looking at listobject.c also... if we want list.clear() to behave exactly like del list[], we may be able to just call list_ass_slice on the list. Similarly for list.copy which should behave like a=l[:]

    @elibendersky
    Copy link
    Mannequin

    elibendersky mannequin commented Nov 27, 2010

    Hi, I'm also looking at listobject.c also... if we want list.clear() to
    behave exactly like del list[], we may be able to just call list_ass_slice
    on the list. Similarly for list.copy which should behave like a=l[:]

    Note that when executed to do 'del lst[:]' (i.e. with argument v set to 0
    and ilow/ihigh to the maximal range of the list), list_ass_slice will just
    call list_clear anyway, which is a cue that this indeed is the right way to
    do it, despite the strange comment I mentioned in my message above.

    @birkenfeld
    Copy link
    Member

    birkenfeld commented Nov 27, 2010

    Yes, list_clear should be called, but no, it cannot be used directly because a method needs a PyObject* return value. So a wrapper method is needed that looks like listappend() does for list.append(). list_copy() will just look like list_slice() with the index fiddling removed.

    @IIIIllllIIIIllllIIIIllllIIIIllllIIIIll
    Copy link
    Mannequin

    IIIIllllIIIIllllIIIIllllIIIIllllIIIIll mannequin commented Nov 27, 2010

    That's good if it's so... can you explain why list_clear doesn't guarantee that the list is empty? Why would XDECREF populate the list? I don't quite understand it.

    eli: are you writing a patch for this?

    @elibendersky
    Copy link
    Mannequin

    elibendersky mannequin commented Nov 27, 2010

    Attaching a patch for list.clear():

    1. Implements a new function in Objects/listobject.c named listclear() (to be consistent with the other "method functions")
    2. listclear() is registered in list_methods and just calls list_clear(), returning None
    3. A documentation string is modeled after dict.clear(), but shaped a bit differently to follow the conventions of other list method docs.

    If this look fine to the more experienced devs, things left to do are:

    1. Add tests
    2. Implement the .copy() method in a similar manner + tests for it

    Some random observations:

    1. The naming of functions/methods could be made more consistent. See, for example, list_reversed vs. listreverse.
    2. The documentation style of list and dict methods is different for no apparent reason:

    help({}.pop) gives:

    pop(...)
        D.pop(k[,d]) -> v, remove specified key and return the corresponding value.
        If key is not found, d is returned if given, otherwise KeyError is raised

    While help([].pop) gives:

    pop(...)
        L.pop([index]) -> item -- remove and return item at index (default last).
        Raises IndexError if list is empty or index is out of range.

    Note the '--' which separates the signature from description in the list version.

    @elibendersky
    Copy link
    Mannequin

    elibendersky mannequin commented Dec 4, 2010

    Was list.copy() also approved? After all, there are many ways to achieve the same even now:

    1. L[:]
    2. list(L)
    3. import copy and then copy.copy

    Especially re the last one: list.copy() can be deep or shallow, which one should it be?

    @elibendersky
    Copy link
    Mannequin

    elibendersky mannequin commented Dec 4, 2010

    Also, where is the *official* place to document list objects and their methods?

    @birkenfeld
    Copy link
    Member

    birkenfeld commented Dec 4, 2010

    Yes, list.copy was also approved IIRC. And it should be a shallow copy, like all other copy methods on builtins.

    @Retro
    Copy link
    Mannequin

    Retro mannequin commented Dec 4, 2010

    This is really welcome. It makes Python even more readable.

    If 'a' is a list object, a[:] is not so obvious at first to a newcomer, but
    a.copy() is.

    Also, a.clear() is so perfect and understandable. I wish you could decorate Python versions prior to 3.3 with this two new list methods.

    @elibendersky
    Copy link
    Mannequin

    elibendersky mannequin commented Dec 4, 2010

    Attaching a patch with the following:

    1. list.copy() and list.clear() implemented in Objects/listobject.c, with appropriate doc strings (modeled after dict's docstrings)
    2. Same methods implemented in collections.UserList
    3. Tests added that exercise the methods in both list and UserList

    Re the documentation, it's currently unclear where it should go. I asked on docs@python.org.

    @merwok
    Copy link
    Member

    merwok commented Dec 4, 2010

    Hi Eli,

    I think the right place is 4.6.4,
    http://docs.python.org/dev/library/stdtypes#mutable-sequence-types

    It starts with “List and bytearray objects support additional operations
    that allow in-place modification of the object”.

    For methods not supported by bytearray, you can use the fake footnote
    (8) and edit its texte (“sort() is not supported by bytearray objects”).

    Regards

    @rhettinger rhettinger added the easy label Dec 4, 2010
    @rhettinger rhettinger self-assigned this Dec 4, 2010
    @elibendersky
    Copy link
    Mannequin

    elibendersky mannequin commented Dec 4, 2010

    Following Éric's suggestion, I'm attaching an updated patch with with the documentation in Doc/library/stdtypes.rst updated with the new methods.

    There seems to be a slight Sphinx markup problem with this addition. I rewrote note (8) as:

    :meth:`clear`, :meth:`copy` and :meth:`sort` are not supported by
    :class:`bytearray` objects.

    Unfortunately, :meth:`copy` is getting linked to the copy module.

    @rhettinger
    Copy link
    Contributor

    rhettinger commented Dec 4, 2010

    Nothing will happen on this until 3.2 is done and the py3k branch starts with 3.3 submissions.

    @merwok
    Copy link
    Member

    merwok commented Dec 4, 2010

    Eli, I learned this trick recently: :meth:`!copy`.

    @elibendersky
    Copy link
    Mannequin

    elibendersky mannequin commented Dec 5, 2010

    Éric - thanks, it works [attaching updated patch]. However, don't you think the core problem is a Sphinx bug we should report?

    Raymond - this happens after final 3.2 release (on Feb 05 2011 if it's on schedule), right?

    @Retro
    Copy link
    Mannequin

    Retro mannequin commented Dec 5, 2010

    I'm troubled with one little letter:

    "L.copy() -> list -- a shallow copy of L"); should be

    "L.copy() -> list -- shallow copy of L"); without the letter 'a',
    because other sentences also don't say "L.__sizeof__() -- *A* size of
    L in memory, in bytes");

    Please fix this.

    @Retro
    Copy link
    Mannequin

    Retro mannequin commented Dec 5, 2010

    Can you please help me find the definition of the copy() method of dict in
    the Python sources? I want to see how that method is defined and compare the
    definition to the one in Eli's patch.

    @terryjreedy
    Copy link
    Member Author

    terryjreedy commented Dec 5, 2010

    Objects/dictobject.c

    @elibendersky
    Copy link
    Mannequin

    elibendersky mannequin commented Dec 6, 2010

    Boštjan,

    "a shallow copy": I took this directly from the documentation of dicts, which says:

    "D.copy() -> a shallow copy of D")

    As I mentioned in an earlier message, the doc-strings of list and dict methods are inconsistent in more than one way, so I'm going to leave this decision to the committer. I'll be happy to help with fixes too.

    Re your other question, in the Python source root, dictionaries are mostly implemented in Objects/dictobject.c - there's an array called mapp_methods that lists the functions used to implement relevant methods. For copy() it lists:

    {"copy",            (PyCFunction)dict_copy,         METH_NOARGS,
    

    So you need dict_copy. Note that it's just a wrapper (of another wrapper, by the way) bit it's a good place to start. Arm yourself with an editor or IDE with some code-searching capabilities.

    @Retro
    Copy link
    Mannequin

    Retro mannequin commented Dec 6, 2010

    mapp_methods ? Don't you mean map_methods ?

    @birkenfeld
    Copy link
    Member

    birkenfeld commented Dec 6, 2010

    No, he means mapp_methods. Why don't you simply look at the file?

    @ysjray
    Copy link
    Mannequin

    ysjray mannequin commented Feb 24, 2011

    Please modify the patch so that it can be applied to current py3k trunk cleanly. (Notice that Lib/collections.py has been changed to a package in bpo-11085).

    @merwok
    Copy link
    Member

    merwok commented Feb 24, 2011

    Ray: Eli can just refresh his patch and commit. Note that the patch program will prompt you for a file name if it can’t find the file for a diff hunk, so it should be trivial to apply a patch across a rename.

    Eli: Would you mind changing two nits before committing the nice patch?

    +"L.clear() -> None -- remove all items from L");
    It looks like other methods that return None just omit the “-> type” part.

    +

    +
    That’s one new line too many. Some trailing spaces are also included.

    @rhettinger
    Copy link
    Contributor

    rhettinger commented Feb 24, 2011

    Eli doesn't need to post a new patch. I'm sure he will fix any nits in his commit.

    @elibendersky
    Copy link
    Mannequin

    elibendersky mannequin commented Feb 24, 2011

    On Thu, Feb 24, 2011 at 05:26, Éric Araujo <report@bugs.python.org> wrote:

    Éric Araujo <merwok@netwok.org> added the comment:

    Ray: Eli can just refresh his patch and commit.  Note that the patch program will prompt you for a file name if it can’t find the file for a diff hunk, so it should be trivial to apply a patch across a rename.

    Eli: Would you mind changing two nits before committing the nice patch?

    +"L.clear() -> None -- remove all items from L");
    It looks like other methods that return None just omit the “-> type” part.

    That’s one new line too many.  Some trailing spaces are also included.

    I will fix and commit over the weekend.

    @rhettinger
    Copy link
    Contributor

    rhettinger commented Feb 24, 2011

    [Éric Araujo]

    +"L.clear() -> None -- remove all items from L");
    It looks like other methods that return None
    just omit the “-> type” part.

    These kind of nitty comments really aren't helpful.
    It consumes more time to talk about them than they're worth.
    In this case, Eli was modeling after the docstring in dictobject.c:

    PyDoc_STRVAR(clear__doc__,
    "D.clear() -> None. Remove all items from D.");

    Just because list.remove.__doc__ failed to consistently follow that convention doesn't make Eli's patch incorrect.

    @elibendersky
    Copy link
    Mannequin

    elibendersky mannequin commented Feb 25, 2011

    A slightly revised patch committed in revision 88554:

    1. Fixed Éric's whitespace comment
    2. Fixed a test in test_descrtut.py which was listing list's methods
    3. Moved the change to collections.py onto Lib/collections/init.py
    4. Added NEWS entry

    Éric - as I mentioned earlier in this issue, I chose to leave the syntax of the docstring for the new methods similar to the same methods in dict (dict docstring look better and more internally consistent anyhow). I propose to move further discussion of this matter into a separate issue which will deal with the overall (in)consistency in the docstrings of list and dict.

    @birkenfeld
    Copy link
    Member

    birkenfeld commented Feb 25, 2011

    Reading "clear and copy are not supported by bytearray": shouldn't they be? ("sort" probably really makes no sense on bytearrays.)

    @elibendersky
    Copy link
    Mannequin

    elibendersky mannequin commented Feb 25, 2011

    On Fri, Feb 25, 2011 at 10:11, Georg Brandl <report@bugs.python.org> wrote:

    Georg Brandl <georg@python.org> added the comment:

    Reading "clear and copy are not supported by bytearray": shouldn't they be?

    Perhaps they should, and it's not a big deal to implement. But I'm not
    100% clear on the policy of how such changes are approved. Should this
    be discussed in the list?

    @rhettinger
    Copy link
    Contributor

    rhettinger commented Feb 25, 2011

    Unless someone raises a controversial and non-trivial issue about adding clear() and copy() to bytearray, there is no need for a python-dev discussion on the subject. Just post a patch to the tracker.

    @birkenfeld
    Copy link
    Member

    birkenfeld commented Feb 25, 2011

    Yes, it should be discussed on python-dev.

    @birkenfeld
    Copy link
    Member

    birkenfeld commented Feb 25, 2011

    In any case, this issue can be closed.

    @Retro
    Copy link
    Mannequin

    Retro mannequin commented Feb 25, 2011

    I have installed Python 3.2 final on my Windows machine and I get an
    exception when doing list.copy or list.clear in the interpreter. Why is that
    so?

    @birkenfeld
    Copy link
    Member

    birkenfeld commented Feb 25, 2011

    Because they got added *after* 3.2 was released?

    @Retro
    Copy link
    Mannequin

    Retro mannequin commented Feb 25, 2011

    Right, right. My bad. Can't wait for Python 3.3! ;)

    @rhettinger
    Copy link
    Contributor

    rhettinger commented Feb 25, 2011

    Georg, what is the issue? Is there some reason that bytearrays should not be copied or cleared? Is there some reason to prefer the current:

    dup = b[:] # copy
    del b[:] # clear

    @birkenfeld
    Copy link
    Member

    birkenfeld commented Feb 25, 2011

    No -- but the question is if copy() and clear() mightn't be added to the (mutable) sequence ABC if we make all builtin such sequences implement them.

    @rhettinger
    Copy link
    Contributor

    rhettinger commented Feb 25, 2011

    The ABCs are subset of the methods for the concrete APIs. We've avoided the likes of copy() because it requires knowledge of the constructor's signature -- for example, MutableMapping does not cover copy().

    It is okay for Eli to add MutableSequence.clear() because it can be implemented in terms of pop(), much like we do for MutableMapping.clear().

    Eli, feel free to create a patch to add clear() and copy() to bytearray and to add clear() to MutableSequence. Assign the patch to me for review.

    @elibendersky
    Copy link
    Mannequin

    elibendersky mannequin commented Feb 26, 2011

    Attaching a patch adding copy() and clear() to bytearrays, with tests and doc.

    I didn't add the methods to MutableSequence because I have a doubt about it - in particular which exception get raised by .pop if it's empty. Curiously, lists and bytearrays raise different exceptions in this case - IndexError and OverflowError, respectively.

    @elibendersky elibendersky mannequin reopened this Feb 26, 2011
    @elibendersky elibendersky mannequin assigned rhettinger and unassigned elibendersky Feb 26, 2011
    @rhettinger
    Copy link
    Contributor

    rhettinger commented Mar 1, 2011

    The patch is fine. Do consider using assertIsNot() in the tests. Then go ahead and apply it.

    The OverflowError in bytearray.pop() is a bug, please open a separate report for it and make a patch changing it to IndexError. Assign that bug report to me.

    Go ahead and propose a patch for MutableSequence.clear() implemented with MutableSequence.pop() and catching an IndexError when empty.

    Thanks for your efforts.

    @birkenfeld
    Copy link
    Member

    birkenfeld commented Mar 1, 2011

    Hmm, shouldn't self.__class__(self) be a good default implementation of copy()?

    I'd expect any sequence to support this way of creation from another sequence, even if it's inefficient.

    @rhettinger
    Copy link
    Contributor

    rhettinger commented Mar 1, 2011

    Hmm, shouldn't self.__class__(self) be a
    good default implementation of copy()?

    I'd expect any sequence to support this way
    of creation from another sequence, even if it's inefficient.

    The copy() method isn't being proposed for MutableSequence because it presumes that we know something about the constructor's signature. For example, the constructor of array() needs the element storage type as an argument. We refuse the temptation to guess.

    In the Set api, we had no choice because many set-methods necessarily create a new set. To handle the constructor signature problem, the creation step was factored-out into the from_iterable() method so that a user could override it if necessary.

    Also copy() is handled oddly in the builtin types. To handle the constructor signature issue for subclasses, they ignore the subclass and return a instance of the base class. For example, the inherited copy() method on a subclass of list or dict will create an instance of list or dict, not of the subclass itself. Accordingly, subclasses that want instances of themselves have to override the inherited copy() method. They would have to do this anyway if they subclass contained any other data in the class dictionary that would need to be passed along to a copy.

    In short, we're better-off not supplying copy() as part of the MutableSequence ABC.

    @elibendersky
    Copy link
    Mannequin

    elibendersky mannequin commented Mar 3, 2011

    Committed the bytearray methods in revision 88733.

    Closing this issue. Will handle wrong exception and MutableSequence ABC method addition in separate issues.

    @elibendersky elibendersky mannequin closed this as completed Mar 3, 2011
    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Mar 17, 2012

    New changeset 958a98bf924e by Eli Bendersky in branch 'default':
    updated whatsnew/3.3.rst with the new methods added to list and bytearray (bpo-10516)
    http://hg.python.org/cpython/rev/958a98bf924e

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    easy interpreter-core Interpreter core (Objects, Python, Grammar, and Parser dirs) type-feature A feature request or enhancement
    Projects
    None yet
    Development

    No branches or pull requests

    6 participants