Pickling/unpickling a dtype resets isbuiltin flag

[inducer]

This just happened to me with numpy 1.8.0 (from Debian):

>>> import numpy as np
>>> x = np.dtype(np.float32)
>>> from pickle import dumps, loads
>>> y = loads(dumps(x))
>>> x.isbuiltin
1
>>> y.isbuiltin
0

[inducer]

Note that this also appears to change the hash of the dtype, post-unpickling.

[inducer]

Still in 1.10.4.

[njsmith]

I guess this hasn't been a priority because, while probably a bug, it seems rather trivial so it's unlikely to make it to the top of anyone's todo list... but if you are checking on it two years later than maybe that assessment is wrong? Does it matter somehow?

(I can see how the hash issue could cause problems, but I can't reproduce that on 1.11.0rc1:

In [15]: x = np.dtype(np.float32)

In [16]: y = loads(dumps(x))

In [17]: hash(x) == hash(y)
Out[17]: True

In [18]: x == y
Out[18]: True

)

[kaymes]

Still in 1.11.0

I just came across this issue after spending some time debugging a weird caching issue.

I feed the data into a graphics library which checks whether the data fed to it is a builtin type. Since this fails, the application crashes.

To make things worse, there seems to be no simple way to convert the data type to a builtin type again. The only reliable solution I could find is to convert the array into a list and then create a new array which is slow.

[mattip]

Still exists

[mattip]

The documentation seems to be too terse. np.dtype('S0').isbuiltin is 1 but np.dtype('S10').isbuiltin is 0. What is that attribute meant to convey?

[djhoese]

One of our vispy users (see vispy/vispy#1741) has run in to this issue when trying to provide pickled data to vispy. Vispy does some checks on the provided data arrays including checking the isbuiltin flag.

I tried looking for the various __getstate__/__setstate__ methods that would apply to the dtype object being saved, but couldn't pick decipher the numpy source. If one of the numpy devs could point me to the relevant code I can try to find time to track this down.

[mattip]

I think you may be looking for arraydescr_reduce and arraydescr_setstate in numpy/core/src/multiarray/descriptor.c. Make sure there are new tests in numpy/core/tests/test_dtype.py

[djhoese]

Thanks @mattip. I have a guess but won't have time to test things out right now. I'm a little confused by the use of NULL versus Py_None in the descriptor.c file. In the isbuiltin getter we see:

numpy/numpy/core/src/multiarray/descriptor.c

Lines 1968 to 1970 in 9656e4c

	if (self->fields == Py_None) {
	val = 1;
	}

And this == Py_None check is seen other places in this file, but there are also checks that do PyDataType_HASFIELDS(self) which is a check if self->names == NULL. Then in other places of this file there are checks if self->fields == NULL. Some of the other usages of == Py_None against the fields make sense (I think) because numpy is taking what the user gave and converting it to NULL:

numpy/numpy/core/src/multiarray/descriptor.c

Lines 1745 to 1747 in 9656e4c

	if (newdescr->fields == Py_None) {
	newdescr->fields = NULL;
	}

It's a start.

[djhoese]

According to:

numpy/numpy/core/include/numpy/ndarraytypes.h

Lines 624 to 628 in dbcdeb9

	* The fields dictionary for this type
	* For statically defined descr this
	* is always Py_None
	*/
	PyObject *fields;

fields should/could be Py_None...I'm a little worried NULL versus Py_None are two valid but different cases.