struct.unpack returns null pascal strings - [first] bug report

[jonheiner]

BPO	23816
Nosy	@mdickinson, @meadori, @zhangyangyu
Files	unpack_pascal.py: repro case

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-05-04.04:04:11.844>
created_at = <Date 2015-03-30.21:27:03.519>
labels = ['invalid', 'library']
title = 'struct.unpack returns null pascal strings - [first] bug report'
updated_at = <Date 2017-05-04.07:40:43.320>
user = 'https://bugs.python.org/jonheiner'

bugs.python.org fields:

activity = <Date 2017-05-04.07:40:43.320>
actor = 'mark.dickinson'
assignee = 'none'
closed = True
closed_date = <Date 2017-05-04.04:04:11.844>
closer = 'xiang.zhang'
components = ['Library (Lib)']
creation = <Date 2015-03-30.21:27:03.519>
creator = 'jonheiner'
dependencies = []
files = ['38745']
hgrepos = []
issue_num = 23816
keywords = []
message_count = 3.0
messages = ['239644', '292960', '292968']
nosy_count = 4.0
nosy_names = ['mark.dickinson', 'meador.inge', 'jonheiner', 'xiang.zhang']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue23816'
versions = ['Python 2.7']

[jonheiner]

I believe there is an issue with the _struct.c handling of Pascal style strings.

In the _struct.c:s_unpack_internal() function (reading 2.7.6 and 2.7.9 source from tgz ball), the size parameter 'n' is clamped to code->size-1.

As far as I can tell, 'n' is set to the correct deserialized value, but the code->size value is not set to 255. I could be incorrect, as I'm not running in a debugger.

I've attached a short repro case. Note the use of unpack_from() as otherwise unpac() will thrown an error. Additionally, I may be using it wrong, but this feels correct.

[zhangyangyu]

My previous two messages are not clear enough so I delete them. Sorry for the noise. :-(

When unpacking a pascal string, you cannot simply specify a p format character otherwise struct calculate a wrong size of the format. That's why unpack fails. When count conflicts with the length byte, the smaller one is chosen. I think this is consistent with packing.

[mdickinson]

Specifically, I believe what's happening here is that "8s4spp" is interpreted as "8s4s1p1p", so it decodes a single byte (which can only encode an empty string) for each of the "1p" cases.

I wonder whether the struct module should raise an exception if the length byte read from the encoded data exceeds the count given in the format.