Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Unpickling can corrupt an internal cache of bytes object (py3) #370

rlamy opened this Issue Jul 30, 2012 · 3 comments


None yet
3 participants

rlamy commented Jul 30, 2012

The Python 3 interpreter has an internal cache of single-byte bytes objects, from which objects are retrieved in a few operations, notably computing length-1 slices of bytes objects. Pickling and unpickling certain arrays can give direct write-access into that cache. A simple way to reproduce the problem is:

>>> a = numpy.array([1], 'b')
>>> b = pickle.loads(pickle.dumps(a))
>>> b[0] = 77
>>> b'\x01  '[0:1]

This applies not only to master, but also to released versions, e.g. to Ubuntu 12.04's python3-numpy, which has numpy 1.6.1 and Python 3.2.3.

A similar problem has apparently been encountered before, as the test suite has test_pickle_string_overwrite() in numpy/core/tests/test_regression.py, which triggers the cache corruption but doesn't detect it.


pv commented Jul 30, 2012

Relevant source code location: numpy/core/src/multiarray/methods.c:1588
The comment /* Bytes are never interned */ there is invalid.
The fix is one-liner, just take the branch unconditionally.


rlamy commented Jul 31, 2012

Thanks, I did exactly that.


njsmith commented Aug 31, 2012

This got fixed a while ago :-).

@njsmith njsmith closed this Aug 31, 2012

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment