binascii.crc32 is not 64-bit clean when USE_ZLIB_CRC32

[zmk]

BPO	38256
Nosy	@gpshead, @Savier
PRs	bpo-38256: Fix binascii.crc32() when inputs are 4+GiB #32000 [3.10] bpo-38256: Fix binascii.crc32 large input. #32013 [3.9] bpo-38256: Fix binascii.crc32 large input. #32015

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/gpshead'
closed_at = <Date 2022-03-21.06:37:04.336>
created_at = <Date 2019-09-23.07:48:20.996>
labels = ['type-bug', '3.8', '3.9', '3.10', '3.11', '3.7', 'library']
title = 'binascii.crc32 is not 64-bit clean when USE_ZLIB_CRC32'
updated_at = <Date 2022-03-21.06:37:04.335>
user = 'https://bugs.python.org/zmk'

bugs.python.org fields:

activity = <Date 2022-03-21.06:37:04.335>
actor = 'gregory.p.smith'
assignee = 'gregory.p.smith'
closed = True
closed_date = <Date 2022-03-21.06:37:04.336>
closer = 'gregory.p.smith'
components = ['Library (Lib)']
creation = <Date 2019-09-23.07:48:20.996>
creator = 'zmk'
dependencies = []
files = []
hgrepos = []
issue_num = 38256
keywords = ['patch']
message_count = 10.0
messages = ['352992', '352993', '352994', '352995', '415577', '415578', '415628', '415636', '415667', '415668']
nosy_count = 4.0
nosy_names = ['gregory.p.smith', 'nadeem.vawda', 'zmk', 'iamsav']
pr_nums = ['32000', '32013', '32015']
priority = 'normal'
resolution = 'fixed'
stage = 'commit review'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue38256'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9', 'Python 3.10', 'Python 3.11']

[zmk]

Following code:
----------------------------

import binascii, zlib
bigdata=memoryview(bytearray((1<<32) + 100))

print(binascii.crc32(bigdata))
crc = binascii.crc32(bigdata[:1000])
crc = binascii.crc32(bigdata[1000:], crc)
print(crc)

print(zlib.crc32(bigdata))
crc = zlib.crc32(bigdata[:1000])
crc = zlib.crc32(bigdata[1000:], crc)
print(crc)

---

Outputs with python3.7

$ python3.7 x.py 
2575877834
2838121701
2838121701
2838121701

[Savier]

I've got
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD64)]
2838121701
2838121701
2838121701
2838121701

[zmk]

Looking at the code, the bug is dependant on USE_ZLIB_CRC32 define and it's unfixed in master. Cause is zlib crc32 that takes length as integer, zlibmodule has workaround, binascii has not.

[zmk]

Found zlibmodule fix: https://bugs.python.org/issue10276

[gpshead]

$ python3.8
Python 3.8.10 (default, Nov 26 2021, 20:14:08) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import binascii, zlib
>>> bigdata=memoryview(bytearray((1<<32) + 100))
>>> 
>>> print(binascii.crc32(bigdata))
2838121701
>>> crc = binascii.crc32(bigdata[:1000])
>>> crc = binascii.crc32(bigdata[1000:], crc)
>>> print(crc)
2838121701
>>> 
>>> print(zlib.crc32(bigdata))
2838121701
>>> crc = zlib.crc32(bigdata[:1000])
>>> crc = zlib.crc32(bigdata[1000:], crc)
>>> print(crc)
2838121701

[gpshead]

it depends on the build. USE_ZLIB_CRC32 causes it due to zlib's 32-bitness as noted my marko.

$ ./python 
Python 3.11.0a6+ (heads/main-dirty:b3f2d4c8ba, Mar 19 2022, 15:32:04) [GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import binascii, zlib
>>> bigdata=memoryview(bytearray((1<<32) + 100))
>>> 
>>> print(binascii.crc32(bigdata))
2575877834
>>> crc = binascii.crc32(bigdata[:1000])
>>> crc = binascii.crc32(bigdata[1000:], crc)
>>> print(crc)
2838121701
>>> 
>>> print(zlib.crc32(bigdata))
2838121701
>>> crc = zlib.crc32(bigdata[:1000])
>>> crc = zlib.crc32(bigdata[1000:], crc)
>>> print(crc)
2838121701
>>>

[gpshead]

New changeset 9d1c4d6 by Gregory P. Smith in branch 'main':
bpo-38256: Fix binascii.crc32() when inputs are 4+GiB (GH-32000)
9d1c4d6

[gpshead]

New changeset 4c989e1 by Gregory P. Smith in branch '3.10':
[3.10] bpo-38256: Fix binascii.crc32 large input. (GH-32000) (GH-32013)
4c989e1

[gpshead]

New changeset 58a7e13 by Gregory P. Smith in branch '3.9':
bpo-38256: Fix binascii.crc32 large input. (GH-32000) (GH-32013) (GH-32015)
58a7e13

[gpshead]

3.8 and older are in security fix only mode. 3.9+ have been fixed. realistically this was a rare issue with multiple workarounds.

The 3.9 and 3.10 fixes were strictly bugfix only. The 3.11 fix included some performance improvements.