marshal.dumps represents the same list object differently

[Dreeseaw]

BPO	46900
Nosy	@rhettinger, @ericvsmith, @Dreeseaw

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2022-03-02.15:55:38.713>
created_at = <Date 2022-03-02.14:40:17.209>
labels = ['expert-C-API', 'type-bug', 'invalid', '3.11']
title = 'marshal.dumps represents the same list object differently'
updated_at = <Date 2022-03-02.15:55:38.713>
user = 'https://github.com/Dreeseaw'

bugs.python.org fields:

activity = <Date 2022-03-02.15:55:38.713>
actor = 'Dreeseaw'
assignee = 'none'
closed = True
closed_date = <Date 2022-03-02.15:55:38.713>
closer = 'Dreeseaw'
components = ['C API']
creation = <Date 2022-03-02.14:40:17.209>
creator = 'Dreeseaw'
dependencies = []
files = []
hgrepos = []
issue_num = 46900
keywords = []
message_count = 5.0
messages = ['414362', '414363', '414366', '414367', '414369']
nosy_count = 3.0
nosy_names = ['rhettinger', 'eric.smith', 'Dreeseaw']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue46900'
versions = ['Python 3.11']

[Dreeseaw]

Hello,

I've been working with the marshal package and came across this issue (I think) -

Python 3.9.10 Interpreter
(Same output in 3.11.0a5+ (heads/master:b6b711a1aa) on darwin)
>>> import marshal
>>> var_example = [(1,2,3),(4,5,6)]
>>> var_marshaled = marshal.dumps(var_example)
>>> raw_marshaled = marshal.dumps([(1,2,3),(4,5,6)])
>>> def pp(to_print):
>>>     [print(byt) for byt in to_print]
>>> pp(var_marshaled)
219
2
0
0
0
41
3
233
1
0
0
0
233
2
0
0
0
233
3
0
0
0
169
3
233
4
0
0
0
233
5
0
0
0
233
6
0
0
0
91
2
0
0
0
41
3
233
1
0
0
0
233
2
0
0
0
233
3
0
0
0
169
3
233
4
0
0
0
233
5
0
0
0
233
6
0
0
0
>>> pp(raw_marshaled)
219
2
0
0
0
169
3
233
1
0
0
0
233
2
0
0
0
233
3
0
0
0
169
3
233
4
0
0
0
233
5
0
0
0
233
6
0
0
0
91
2
0
0
0
169
3
233
1
0
0
0
233
2
0
0
0
233
3
0
0
0
169
3
233
4
0
0
0
233
5
0
0
0
233
6
0
0
0

The difference above lies in the byte representation of the tuple type (41 in the variable version and 169 in the raw version). Is this intended behavior?

[Dreeseaw]

I've made a very bad copy & paste error with the terminal output below, I apologize. The corrected output is

>>> pp(var_marshaled)
91
2
0
0
0
41
3
233
1
0
0
0
233
2
0
0
0
233
3
0
0
0
41
3
233
4
0
0
0
233
5
0
0
0
233
6
0
0
0

>>> pp(raw_marshaled)
91
2
0
0
0
169
3
233
1
0
0
0
233
2
0
0
0
233
3
0
0
0
169
3
233
4
0
0
0
233
5
0
0
0
233
6
0
0
0

[rhettinger]

The difference is the FLAG_REF which is set to 128 (0x80).

>>> import marshal
>>> var_example = [(1,2,3),(4,5,6)]
>>> vm = marshal.dumps(var_example)
>>> rm = marshal.dumps([(1,2,3),(4,5,6)])
>>> [v ^ r for v, r in zip(vm, rm)]
[128, 0, 0, 0, 0, 128, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 128, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

Whether a flag for possible reuse is generated depends on the reference count of a object.

When passing in the list as variable, the reference count is higher than passing it as a literal.

That flag tells marshal whether to generate an index entry. Whether that occurs re

[ericvsmith]

From https://github.com/python/cpython/blob/main/Python/marshal.c:

41 is:

#define TYPE_SMALL_TUPLE        ')'

The difference between 41 and 169 is 128:

#define FLAG_REF                '\x80' /* with a type, add obj to index */

So the difference is the FLAG_REF bit being set. I'm not sure if that helps you or not.

In any event, this doesn't look like a bug. You might want to ask on python-list or Stack Overflow for more help.

[Dreeseaw]

You two are both correct, this is not a bug and is the intended functionality.

The difference between 41 and 169 is 128:

This realization helps a ton. Thanks.