Bump the default pickle protocol in shelve

[serhiy-storchaka]

BPO	34204
Nosy	@rhettinger, @vstinner, @avassalotti, @ambv, @serhiy-storchaka, @ZackerySpytz, @marco-c
PRs	bpo-34204: Use pickle.DEFAULT_PROTOCOL in shelve #19639 bpo-34204: Make the shelve library use Pickle's default protocol version instead of defaulting to version 3 #22751

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-10-29.09:46:33.380>
created_at = <Date 2018-07-24.09:16:42.390>
labels = ['type-feature', 'library', '3.10']
title = 'Bump the default pickle protocol in shelve'
updated_at = <Date 2020-10-29.09:46:33.379>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2020-10-29.09:46:33.379>
actor = 'cheryl.sabella'
assignee = 'none'
closed = True
closed_date = <Date 2020-10-29.09:46:33.380>
closer = 'cheryl.sabella'
components = ['Library (Lib)']
creation = <Date 2018-07-24.09:16:42.390>
creator = 'serhiy.storchaka'
dependencies = []
files = []
hgrepos = []
issue_num = 34204
keywords = ['patch']
message_count = 5.0
messages = ['322281', '370137', '379812', '379813', '379861']
nosy_count = 7.0
nosy_names = ['rhettinger', 'vstinner', 'alexandre.vassalotti', 'lukasz.langa', 'serhiy.storchaka', 'ZackerySpytz', 'marco-c']
pr_nums = ['19639', '22751']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue34204'
versions = ['Python 3.10']

[serhiy-storchaka]

The default pickle protocol is 4 now. But shelve still uses the pickle protocol 3. Shouldn't it be bumped? Shouldn't shelve use pickle.DEFAULT_PROTOCOL by default?

Disadvantages:

1. This will make shelve files incompatible with Python 3.3 by default.

2. Protocol 4 adds 9 bytes of overhead in comparison with protocol 3. This can be too large for the shelve containing a lot of small objects. Maybe strip redundant frame header for small pickles?

[vstinner]

I wrote a short script to see the impact of file size depending on the protocol:
---

import shelve
import os.path

print("== Short value ==")
for proto in (0, 1, 2, 3, 4, 5):
    filename = 'shelve-picklev%s' % proto
    with shelve.open(filename, protocol=proto) as db:
        assert db._protocol == proto
        for x in range(1000):
            db[str(x)] = str(x)
    print(f'Protocol {proto}: {os.path.getsize(filename)} bytes')
    os.unlink(filename)
print()
print("== Large value ==")
large_value = [str(x) for x in range(1000)]
for proto in (0, 1, 2, 3, 4, 5):
    filename = 'shelve-picklev%s' % proto
    with shelve.open(filename, protocol=proto) as db:
        assert db._protocol == proto
        for x in range(10):
            db[str(x)] = large_value
    print(f'Protocol {proto}: {os.path.getsize(filename)} bytes')
    os.unlink(filename)

---

Output with Python 3.9.0b1 (on Fedora 32):
---
== Short value ==
Protocol 0: 90112 bytes
Protocol 1: 94208 bytes
Protocol 2: 94208 bytes
Protocol 3: 94208 bytes
Protocol 4: 94208 bytes
Protocol 5: 94208 bytes

== Large value ==
Protocol 0: 139264 bytes
Protocol 1: 139264 bytes
Protocol 2: 139264 bytes
Protocol 3: 139264 bytes
Protocol 4: 98304 bytes
Protocol 5: 98304 bytes
---

For short string values, protocol 0 produces smaller files than protocol 1 and higher.

For large value, protocol 4 and higher produce smaller files than protocol 3 and lower.

[marco-c]

I've opened #22751 to fix this, I know there was already a PR, but it seems to have been abandoned.

[ZackerySpytz]

It has not been abandoned.

[miss-islington]

New changeset df59273 by Zackery Spytz in branch 'master':
bpo-34204: Use pickle.DEFAULT_PROTOCOL in shelve (GH-19639)
df59273