Error when using numpy.load

[MichaelStetner]

I want to save some quaternion data to a file and load it again later. The saving part works fine:

import numpy as np
import quaternion

q = np.quaternion(1, 2, 3, 4)
qarray = np.array([q, q])

filename = 'test.npy'
np.save(filename, qarray)

However, when I try to load the file, I get an error:

np.load(filename)

Here is the full error message:

Traceback (most recent call last):
  File "/Users/stetnermit/miniconda3/envs/neurophysvideo/lib/python3.5/site-packages/numpy/lib/format.py", line 510, in _read_array_header
    dtype = numpy.dtype(d['descr'])
TypeError: data type "<q32" not understood

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test.py", line 9, in <module>
    np.load(filename)
  File "/Users/stetnermit/miniconda3/envs/neurophysvideo/lib/python3.5/site-packages/numpy/lib/npyio.py", line 406, in load
    pickle_kwargs=pickle_kwargs)
  File "/Users/stetnermit/miniconda3/envs/neurophysvideo/lib/python3.5/site-packages/numpy/lib/format.py", line 622, in read_array
    shape, fortran_order, dtype = _read_array_header(fp, version)
  File "/Users/stetnermit/miniconda3/envs/neurophysvideo/lib/python3.5/site-packages/numpy/lib/format.py", line 513, in _read_array_header
    raise ValueError(msg % (d['descr'],))
ValueError: descr is not a valid dtype descriptor: '<q32'

How can I load my saved data?

[moble]

I think this might actually be a bug in numpy itself. Or the documentation is unclear, and I've screwed something up. Either way, I'll ask the numpy developers about this, but for now I'll give you a workaround (or two).

Workaround

You can get around this problem by simply converting the quaternion array to a float array when saving, and back when loading. You would change the relevant lines above to

np.save(filename, quaternion.as_float_array(qarray))

and

quaternion.as_quat_array(np.load(filename))

This trick will always work, even if I do manage to fix the more obvious way of using save/load.

Rescuing data

Don't use this method going forward, but if you've already saved a file from some big procedure that you can't or don't care to reproduce, you can salvage the file by editing it (test.npy in your original example) using a good text editor like emacs or (I assume) vim. The first line of the file will contain something like

{'descr': '<q32', 'fortran_order': False, 'shape': (2,), }

Change that to something like

{'descr': '<f8', 'fortran_order': False, 'shape': (2, 4), }

That is, change <q32 to <f8 and add a 4 at the end of the shape. When you load that again, you'll have an array of floats, which you can convert back to quaternions using quaternion.as_quat_array.

[moble]

This problem is "solved" in the sense that save and load will now do something. Unfortunately, numpy has no way of saving the fact that these are our particular type of user-defined type — quaternions — so you have to tell it the type when you load the data back in. The array you get back after loading will have dtype void, but you can change that by taking another view of the same data. For example, the following code works correctly:

import numpy as np
import quaternion

a = quaternion.as_quat_array(np.random.rand(5,3,4))
np.save("test.npy", a)
b = np.load("test.npy").view(dtype=np.quaternion)
np.array_equal(a, b)  # returns `True`

Note the .view(dtype=np.quaternion) where it's loaded.