Pure Python pickle module should not depend on _pickle.PickleBuffer

[tiran]

BPO	37210
Nosy	@pitrou, @vstinner, @tiran, @moreati, @miss-islington
PRs	bpo-37210: Fix pure Python pickle: proto 5 requires _pickle.PickleBuffer #14016 [3.8] bpo-37210: Fix pure Python pickle when _pickle is unavailable (GH-14016) #14055
Files	bp0-37201_make_test_20190610.txt: make test output for pickle._PickleBuffer attempt

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-06-13.12:28:44.146>
created_at = <Date 2019-06-09.20:23:32.229>
labels = ['3.8', 'type-bug', 'library', '3.9']
title = 'Pure Python pickle module should not depend on _pickle.PickleBuffer'
updated_at = <Date 2019-06-13.12:28:44.145>
user = 'https://github.com/tiran'

bugs.python.org fields:

activity = <Date 2019-06-13.12:28:44.145>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-06-13.12:28:44.146>
closer = 'vstinner'
components = ['Library (Lib)']
creation = <Date 2019-06-09.20:23:32.229>
creator = 'christian.heimes'
dependencies = []
files = ['48410']
hgrepos = []
issue_num = 37210
keywords = ['patch', '3.8regression']
message_count = 15.0
messages = ['345088', '345089', '345090', '345093', '345094', '345095', '345197', '345198', '345261', '345262', '345360', '345362', '345503', '345511', '345512']
nosy_count = 5.0
nosy_names = ['pitrou', 'vstinner', 'christian.heimes', 'Alex.Willmer', 'miss-islington']
pr_nums = ['14016', '14055']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue37210'
versions = ['Python 3.8', 'Python 3.9']

[tiran]

In https://twitter.com/moreati/status/1137815804530049028 Alex Willmer wrote:

In CPython 3.8dev the pure http://pickle.py module now depends on the C _pickle module, but only for one class

cpython/Lib/pickle.py

Line 39 in c879ff2

from _pickle import PickleBuffer

Antoine, would it make sense turn _pickle.PickleBuffer into an optional dependency? The class is only used for pickle 5 as described in PEP-574, https://www.python.org/dev/peps/pep-0574/

[moreati]

I noticed this because I was experimenting with pickle.py from the 3.8 branch to support protocol 5 in https://github.com/moreati/pikl (and later to https://pypi.org/project/zodbpickle/).

However I want to make it clear, if CPython maintainers wish to keep the code as it is I will deal with it. Maximizing our convenience, and minimizing your maintenance workload is paramount.

[moreati]

Arggh, typo. I mean maximizing *your* convenience is paramount.

[pitrou]

Note that PickleBuffer is really a built-in type (it's defined in Objects/picklebufobject.c). However it's exposed from _pickle.c because it doesn't make sense to expose it in the builtins module.

[pitrou]

In short I'm not against making PickleBuffer optional but I'm not sure it makes a lot of sense. Would you make memoryview optional because it's written in C? Is anyone annoyed by the fact that something comes from _pickle.c.

(note that the pure Python Pickler class *does* work with PickleBuffer; so you don't lose any extensability or customizability by using PickleBuffer)

In any case, I won't do the work myself, so feel free to draft a PR so that we see the magnitude of the changes required.

[moreati]

Attempting a PR

[moreati]

I don't think I can do this. My WIP code is in https://github.com/moreati/cpython/pull/new/bpo-37210, and associated make test output is attached.

Principal blockers

• _pickle.PickleBuffer.raw() can return a contiguous buffer from either a c_contiguous, or f_contiguous buffer. memoryview() (and hence pickle._PickleBuffer.raw()) requires a c_contiguous buffer.
• I'm unsure how to allow _PickleBuffer -> bytearray conversion

The most common cause of test failures is TypeError: cannot pickle 'memoryview' object. I guess Python is seeing a lack of reduce(), and failing back to walking the object attributes. Implementing reduce() and or reduce_ex() might fix this, but seems moot given the other blockers.

[pitrou]

Right, it is probably not possible to write a pure Python PickleBuffer. There's a reason it's written in C...

When you said "make PickleBuffer an optional dependency", I thought you intended to have a usable pure Python Pickler, but without support for the PickleBuffer class.

[moreati]

it is probably not possible to write a pure Python PickleBuffer

Fair enough

a usable pure Python Pickler, but without support for the PickleBuffer class.

That makes sense. However, for third party packages (e.g. zodbpickle, pikl) wanting a pure Python pickle module to use as a basis, I think it would make more sense to use the CPython 3.7 branch as a basis. Adding a pickle._Pickler variant to the 3.8 stdlib that can't do protocol 5, just for third-parties imposes a maintenance burden on the core devs for zero benefit to anyone.

I propose closing this ticket as WONTFIX.

[pitrou]

Ok, closing then.