pickle of record array elements results in corruption and my cause segfault #3003

Closed
ovillellas opened this Issue Feb 19, 2013 · 1 comment

Comments

Projects
None yet
2 participants
Contributor

ovillellas commented Feb 19, 2013

It seems that pickle of record scalars does not work properly and results in corruption. That may even cause segfaults if a field happens to be an object. Note that the first field seems to be ok.

(Note, all tested on mac os X 64 bit)

Here is a sample script that raises the problem:

import numpy as np
import pickle

print np.__version__

a = np.empty((10,10), dtype=np.dtype('i4,i4'))

for j in range(10):
    a[j] = [(i, 10-i) for i in range(10)]

print 'pickle/unpickle array of records'
b = pickle.loads(pickle.dumps(a))

print 'result is ', b

print 'pickle/unpickle subarray of array of records'
b = pickle.loads(pickle.dumps(a[0]))

print 'result is ', b

print 'pickle/unpickle element of array of records'
b = pickle.loads(pickle.dumps(a[0,0]))

print 'result is ', b

Running in a console this results in:

1.7.0b2
pickle/unpickle array of records
result is  [[(0, 10) (1, 9) (2, 8) (3, 7) (4, 6) (5, 5) (6, 4) (7, 3) (8, 2) (9, 1)]
 [(0, 10) (1, 9) (2, 8) (3, 7) (4, 6) (5, 5) (6, 4) (7, 3) (8, 2) (9, 1)]
 [(0, 10) (1, 9) (2, 8) (3, 7) (4, 6) (5, 5) (6, 4) (7, 3) (8, 2) (9, 1)]
 [(0, 10) (1, 9) (2, 8) (3, 7) (4, 6) (5, 5) (6, 4) (7, 3) (8, 2) (9, 1)]
 [(0, 10) (1, 9) (2, 8) (3, 7) (4, 6) (5, 5) (6, 4) (7, 3) (8, 2) (9, 1)]
 [(0, 10) (1, 9) (2, 8) (3, 7) (4, 6) (5, 5) (6, 4) (7, 3) (8, 2) (9, 1)]
 [(0, 10) (1, 9) (2, 8) (3, 7) (4, 6) (5, 5) (6, 4) (7, 3) (8, 2) (9, 1)]
 [(0, 10) (1, 9) (2, 8) (3, 7) (4, 6) (5, 5) (6, 4) (7, 3) (8, 2) (9, 1)]
 [(0, 10) (1, 9) (2, 8) (3, 7) (4, 6) (5, 5) (6, 4) (7, 3) (8, 2) (9, 1)]
 [(0, 10) (1, 9) (2, 8) (3, 7) (4, 6) (5, 5) (6, 4) (7, 3) (8, 2) (9, 1)]]
pickle/unpickle subarray of array of records
result is  [(0, 10) (1, 9) (2, 8) (3, 7) (4, 6) (5, 5) (6, 4) (7, 3) (8, 2) (9, 1)]
pickle/unpickle element of array of records
result is  (0, 536870912)

Note the garbage in the second field of the unpickled object.

A similar script with objects:

import numpy as np
import pickle

print np.__version__

a = np.empty((10,10), dtype=np.dtype('i4,O'))

for j in range(10):
    a[j] = [(i, 'i'*i) for i in range(10)]

print 'pickle/unpickle array of records'
b = pickle.loads(pickle.dumps(a))

print 'result is ', b

print 'pickle/unpickle subarray of array of records'
b = pickle.loads(pickle.dumps(a[0]))

print 'result is ', b

print 'pickle/unpickle element of array of records'
b = pickle.loads(pickle.dumps(a[0,0]))

print 'result is ', b

results in a segfault

1.7.0b2
pickle/unpickle array of records
result is  [[(0, '') (1, 'i') (2, 'ii') (3, 'iii') (4, 'iiii') (5, 'iiiii')
  (6, 'iiiiii') (7, 'iiiiiii') (8, 'iiiiiiii') (9, 'iiiiiiiii')]
 [(0, '') (1, 'i') (2, 'ii') (3, 'iii') (4, 'iiii') (5, 'iiiii')
  (6, 'iiiiii') (7, 'iiiiiii') (8, 'iiiiiiii') (9, 'iiiiiiiii')]
 [(0, '') (1, 'i') (2, 'ii') (3, 'iii') (4, 'iiii') (5, 'iiiii')
  (6, 'iiiiii') (7, 'iiiiiii') (8, 'iiiiiiii') (9, 'iiiiiiiii')]
 [(0, '') (1, 'i') (2, 'ii') (3, 'iii') (4, 'iiii') (5, 'iiiii')
  (6, 'iiiiii') (7, 'iiiiiii') (8, 'iiiiiiii') (9, 'iiiiiiiii')]
 [(0, '') (1, 'i') (2, 'ii') (3, 'iii') (4, 'iiii') (5, 'iiiii')
  (6, 'iiiiii') (7, 'iiiiiii') (8, 'iiiiiiii') (9, 'iiiiiiiii')]
 [(0, '') (1, 'i') (2, 'ii') (3, 'iii') (4, 'iiii') (5, 'iiiii')
  (6, 'iiiiii') (7, 'iiiiiii') (8, 'iiiiiiii') (9, 'iiiiiiiii')]
 [(0, '') (1, 'i') (2, 'ii') (3, 'iii') (4, 'iiii') (5, 'iiiii')
  (6, 'iiiiii') (7, 'iiiiiii') (8, 'iiiiiiii') (9, 'iiiiiiiii')]
 [(0, '') (1, 'i') (2, 'ii') (3, 'iii') (4, 'iiii') (5, 'iiiii')
  (6, 'iiiiii') (7, 'iiiiiii') (8, 'iiiiiiii') (9, 'iiiiiiiii')]
 [(0, '') (1, 'i') (2, 'ii') (3, 'iii') (4, 'iiii') (5, 'iiiii')
  (6, 'iiiiii') (7, 'iiiiiii') (8, 'iiiiiiii') (9, 'iiiiiiiii')]
 [(0, '') (1, 'i') (2, 'ii') (3, 'iii') (4, 'iiii') (5, 'iiiii')
  (6, 'iiiiii') (7, 'iiiiiii') (8, 'iiiiiiii') (9, 'iiiiiiiii')]]
pickle/unpickle subarray of array of records
result is  [(0, '') (1, 'i') (2, 'ii') (3, 'iii') (4, 'iiii') (5, 'iiiii')
 (6, 'iiiiii') (7, 'iiiiiii') (8, 'iiiiiiii') (9, 'iiiiiiiii')]
pickle/unpickle element of array of records
Segmentation fault: 11

My bet: only first field gets unpickled and the rest is left uninitialized

Owner

charris commented Feb 15, 2014

Looks to be fixed in 1.9-devel, maybe earlier.

@charris charris closed this Feb 15, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment