Use pickle protocol 4 by default?

[serhiy-storchaka]

BPO	23403
Nosy	@pitrou, @vstinner, @avassalotti, @ambv, @serhiy-storchaka, @MojoVampire, @applio
PRs	bpo-23403: Bump pickle.DEFAULT_PROTOCOL to 4 #6355 bpo-23403: Bump pickle.DEFAULT_PROTOCOL to 4 #6355 bpo-23403: [lib2to3] Make grammar pickling faster #6491
Dependencies	bpo-33209: Repeated Pickler.dump() doesn't reset the state in the C implementation of pickle

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ambv'
closed_at = <Date 2018-07-24.09:17:14.288>
created_at = <Date 2015-02-06.21:28:18.329>
labels = ['3.8', 'type-feature', 'library']
title = 'Use pickle protocol 4 by default?'
updated_at = <Date 2018-07-24.09:17:14.288>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2018-07-24.09:17:14.288>
actor = 'serhiy.storchaka'
assignee = 'lukasz.langa'
closed = True
closed_date = <Date 2018-07-24.09:17:14.288>
closer = 'serhiy.storchaka'
components = ['Library (Lib)']
creation = <Date 2015-02-06.21:28:18.329>
creator = 'serhiy.storchaka'
dependencies = ['33209']
files = []
hgrepos = []
issue_num = 23403
keywords = ['patch', 'patch']
message_count = 15.0
messages = ['235498', '235499', '235501', '235502', '235504', '235507', '251858', '275440', '314841', '314842', '314927', '315075', '315080', '315117', '322282']
nosy_count = 9.0
nosy_names = ['pitrou', 'vstinner', 'alexandre.vassalotti', 'Arfrever', 'lukasz.langa', 'sbt', 'serhiy.storchaka', 'josh.r', 'davin']
pr_nums = ['6355', '6355', '6491']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue23403'
versions = ['Python 3.8']

[serhiy-storchaka]

Why not use pickle protocol 4 by default? It allows to pickle some objects which are not pickleable with lower protocols and is more efficient.

[pitrou]

Because pickles will not be compatible with Python < 3.4 anymore. People who want maximum efficiency without compatibility can pass the right protocol manually.

[serhiy-storchaka]

Multiprocessing uses default protocol and there is no simple (without hacking the stdlib) way to pass the right protocol manually.

[pitrou]

Multiprocessing uses default protocol

AFAIK, this is so you can interact with processes using another version of Python.

[MojoVampire]

multiprocessing spawns the other processes itself from the same executable used to launch the main process. It's not subprocess. How would a different version of Python get involved?

[pitrou]

It is possible to have independent processes communicate together, although that's not the most widely-used feature. See:
https://docs.python.org/3/library/multiprocessing.html#multiprocessing-listeners-clients
Also:
https://docs.python.org/3/library/multiprocessing.html#managers

[vstinner]

Can't we "negociate" the protocol automatically?

[applio]

Closing in deference to the enhancement described in bpo-28053.

[ambv]

Now that Python 3.3 is dead and 3.4 is soon to follow, it's safe to bump DEFAULT_PROTOCOL to 4.

[serhiy-storchaka]

Should we bump the pickle protocol for shelve?

[ambv]

New changeset c51d8c9 by Łukasz Langa in branch 'master':
bpo-23403: Bump pickle.DEFAULT_PROTOCOL to 4 (bpo-6355)
c51d8c9

[pitrou]

Is there something left to be done here?

[serhiy-storchaka]

Shelve still uses protocol 3 by default. Should it be bumped too?

[pitrou]

Shelve still uses protocol 3 by default. Should it be bumped too?

That sounds reasonable. Perhaps open a separate issue for it?

[serhiy-storchaka]

Opened bpo-34204 for shelve.

[gpshead]

It is possible to have independent processes communicate together, although that's not the most widely-used feature. See:
https://docs.python.org/3/library/multiprocessing.html#multiprocessing-listeners-clients
Also:
https://docs.python.org/3/library/multiprocessing.html#managers

Wow. I really wish we did not support this. multiprocessing was not intended to be a general purpose RPC mechanism between uncoordinated processes.