PersistentList slicing returns a non-persistent list on python3

[pakal]

Since __setslice__() and the likes have been killed in python3 (in favor of __getitem__(slice(...)), it seems that PersistentList must be updated, else slices return regular lists and the DB gets dangerously filled with non-persistent mutables:

Python 3.7.2 (tags/v3.7.2:9a3ffc0492, Dec 23 2018, 23:09:28) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from persistent.list import PersistentList
>>> a = PersistentList([1, 2,3])
>>> isinstance(a, PersistentList)
True
>>> isinstance(a[1:], PersistentList)
False

[jamadden]

I admit to being surprised by this. Sure enough, on Python 2, slicing a subclass of UserList returns an instance of that type (which I wouldn't have expected):

    def __getslice__(self, i, j):
        i = max(i, 0); j = max(j, 0)
        return self.__class__(self.data[i:j])

Whereas on Python 3 it does not:

def __getitem__(self, i): return self.data[i]

Apparently the Python 2 behaviour even flip-flopped a few times in early releases: in 2.1, it returned the subclass, in 2.2b2 it returned just list (as Python 3 does).

Given that it has worked this way for 7 years on Python 3, changing it to be like Python 2 would be an observable change in behaviour that people may have come to count on (e.g., isinstance(persistent_list[1:], list) would change). OTOH, of course, it could ease porting from Python 2 for those that relied on the old behaviour. I don't know what the right answer is.

[mgedmin]

Sounds like a bug to me (one that hasn't been discovered for 7 years because almost nobody is using persistent on Python 3, or at least almost nobody was storing slices back into the DB).

[freddrake]

On Thu, May 2, 2019 at 7:44 AM Marius Gedminas ***@***.***> wrote: Sounds like a bug to me

To me as well. (one that hasn't been discovered for 7 years because almost nobody is using

persistent on Python 3, or at least almost nobody was storing slices back into the DB).

We've long advised against using PersistentList, PersistentDict classes without being very aware that they're stored in a single database object, and I can't recall a single instance of using those intentionally myself. So it may well be that those classes aren't used much (which may be exactly what you meant). -Fred

…

-- Fred L. Drake, Jr. <fred at fdrake.net> "A storm broke loose in my mind." --Albert Einstein

[d-maurer]

Marius Gedminas wrote at 2019-5-2 04:44 -0700:

Sounds like a bug to me (one that hasn't been discovered for 7 years because almost nobody is using persistent on Python 3, or at least almost nobody was storing slices back into the DB).

In Python 3, slicing returns a base list in general: ``` Python 3.6.7 (default, Oct 22 2018, 11:32:17)

>> class L(list): pass

...

>> l=L() >> l[:] = [1,2,3] >> type(l)

<class '__main__.L'>

>> type(l[:])

<class 'list'> ``` Thus, it is (at least) consistent when a `PersistentList` slice is a base list.

[pakal]

We've long advised against using PersistentList, PersistentDict classes without being very aware that they're stored in a single database object, and I can't recall a single instance of using those intentionally myself.

Yes, but aren't those the most standard ways of storing little sequences and mappings in ZODB (which don't require buckets and the likes) ?

If it's consistent with the behaviour of standard python slices, maybe it'd juste need a big warning in docs, so that people take extra care when slicing persistent objects (and when doing other operations which lose persistence, are there?)

[jamadden]

Python 3.7.4 recognized this as a bug and flip-flopped again. However, it wasn't backported to 3.6 or earlier (which we still support).