Repeated Pickler.dump() doesn't reset the state in the C implementation of pickle

[serhiy-storchaka]

BPO	33209
Nosy	@pitrou, @avassalotti, @ambv, @serhiy-storchaka, @miss-islington
PRs	bpo-23403: Bump pickle.DEFAULT_PROTOCOL to 4 #6355 bpo-33209: End framing at the end of C implementation of pickle.Pickler.dump(). #6363 [3.7] bpo-33209: End framing at the end of C implementation of pickle.Pickler.dump(). (GH-6363) #6366 [3.6] bpo-33209: End framing at the end of C implementation of pickle.Pickler.dump(). (GH-6363) #6367

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-04-04.05:08:08.692>
created_at = <Date 2018-04-03.01:08:23.835>
labels = ['extension-modules', '3.8', 'type-bug', '3.7']
title = "Repeated Pickler.dump() doesn't reset the state in the C implementation of pickle"
updated_at = <Date 2018-04-04.05:08:08.692>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2018-04-04.05:08:08.692>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = True
closed_date = <Date 2018-04-04.05:08:08.692>
closer = 'serhiy.storchaka'
components = ['Extension Modules']
creation = <Date 2018-04-03.01:08:23.835>
creator = 'serhiy.storchaka'
dependencies = []
files = []
hgrepos = []
issue_num = 33209
keywords = ['patch']
message_count = 4.0
messages = ['314847', '314898', '314899', '314903']
nosy_count = 5.0
nosy_names = ['pitrou', 'alexandre.vassalotti', 'lukasz.langa', 'serhiy.storchaka', 'miss-islington']
pr_nums = ['6355', '6363', '6366', '6367']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue33209'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

[serhiy-storchaka]

test_clear_pickler_memo tested only the default protocol and fails when bump the default protocol to 4. For some reasons C implementation writes FRAME before PROTO when call Pickler.dump() repeatedly.

Python implementation:
b'\x80\x04\x95\x13\x00\x00\x00\x00\x00\x00\x00]\x94(\x8c\x07abcdefg\x94h\x01K,e.'
0: \x80 PROTO 4
2: \x95 FRAME 19
11: ] EMPTY_LIST
12: \x94 MEMOIZE (as 0)
13: ( MARK
14: \x8c SHORT_BINUNICODE 'abcdefg'
23: \x94 MEMOIZE (as 1)
24: h BINGET 1
26: K BININT1 44
28: e APPENDS (MARK at 13)
29: . STOP
highest protocol among opcodes = 4

C implementation:
b'\x95\x15\x00\x00\x00\x00\x00\x00\x00\x80\x04]\x94(\x8c\x07abcdefg\x94h\x01K,e.'
0: \x95 FRAME 21
9: \x80 PROTO 4
11: ] EMPTY_LIST
12: \x94 MEMOIZE (as 0)
13: ( MARK
14: \x8c SHORT_BINUNICODE 'abcdefg'
23: \x94 MEMOIZE (as 1)
24: h BINGET 1
26: K BININT1 44
28: e APPENDS (MARK at 13)
29: . STOP
highest protocol among opcodes = 4

[ambv]

New changeset c869529 by Łukasz Langa (Serhiy Storchaka) in branch 'master':
bpo-33209: End framing at the end of C implementation of pickle.Pickler.dump(). (GH-6363)
c869529

[ambv]

New changeset 74735e2 by Łukasz Langa (Miss Islington (bot)) in branch '3.7':
bpo-33209: End framing at the end of C implementation of pickle.Pickler.dump(). (GH-6366)
74735e2

[miss-islington]

New changeset a5c8830 by Miss Islington (bot) in branch '3.6':
bpo-33209: End framing at the end of C implementation of pickle.Pickler.dump(). (GH-6363)
a5c8830