Surprising error when setting dtype

[mcara]

My apologies if this was already reported or if the bug is in our code but I am getting unexpected (no deprecation warning was issued in previous numpy releases) error when running drizzlepac. This error or warning was not present in numpy 1.21.0.

Reproducing code example:

First, create a new Python 3.8 environment and install:

$ pip install numpy==1.21.1 drizzlepac==3.2.1 photutils==1.0.2

Activate environment and run:

import numpy as np
import os, glob, shutil
from astroquery.mast import Observations
from drizzlepac import astrodrizzle

# download data
obsTable = Observations.query_criteria(obstype='all', obs_collection='HST', obs_id=['j9irw4040'])
products = Observations.get_product_list(obsTable)
Observations.download_products(products,download_dir='', mrp_only=False, productSubGroupDescription='FLC')
input_flcs = glob.glob(os.path.join('mastDownload', 'HST', '*', 'j*flc.fits'))
for flc in input_flcs:
    shutil.copy(flc, os.path.basename(flc))

# run astrodrizzle
astrodrizzle.AstroDrizzle(input_flcs, output='f606w', preserve=False)

Error message:

Traceback (most recent call last):
  File "/Users/mcara/virtualenv/hnp/lib/python3.8/site-packages/drizzlepac/util.py", line 225, in wrapper
    result = func(*args, **kwargs)
  File "/Users/mcara/virtualenv/hnp/lib/python3.8/site-packages/drizzlepac/astrodrizzle.py", line 204, in run
    sky.subtractSky(imgObjList, configobj, procSteps=procSteps)
  File "/Users/mcara/virtualenv/hnp/lib/python3.8/site-packages/drizzlepac/sky.py", line 197, in subtractSky
    _skymatch(imageObjList, paramDict, inmemory, clean, log)
  File "/Users/mcara/virtualenv/hnp/lib/python3.8/site-packages/drizzlepac/sky.py", line 305, in _skymatch
    (mask, mext) = _buildStaticDQUserMask(imageList[i], extlist[k],
  File "/Users/mcara/virtualenv/hnp/lib/python3.8/site-packages/drizzlepac/sky.py", line 386, in _buildStaticDQUserMask
    mask = img.buildMask(img[ext]._chip,bits=sky_bits)
  File "/Users/mcara/virtualenv/hnp/lib/python3.8/site-packages/drizzlepac/imageObject.py", line 706, in buildMask
    dqmask = buildmask.buildMask(dqarr,bits)
  File "/Users/mcara/virtualenv/hnp/lib/python3.8/site-packages/drizzlepac/buildmask.py", line 84, in buildMask
    return bitfield_to_boolean_mask(dqarr, bitvalue, good_mask_value=1,
  File "/Users/mcara/virtualenv/hnp/lib/python3.8/site-packages/stsci/tools/bitmask.py", line 429, in bitfield_to_boolean_mask
    ignore_mask = np.bitwise_not(ignore_mask, dtype=bitfield.dtype,
TypeError: The `dtype` and `signature` arguments to ufuncs only select the general DType and not details such as the byte order or time unit (with rare exceptions see release notes).  To avoid this warning please use the scalar types `np.float64`, or string notation.
In rare cases where the time unit was preserved, either cast the inputs or provide an output array. In the future NumPy may transition to allow providing `dtype=` to denote the outputs `dtype` as well

The interesting part is that, doing the following (similar to what data are being passed to bitwise_not above except for the array size) directly in the interpreter does not raise an exception or warning:

np.bitwise_not(0, dtype=np.array([1], dtype=np.int16).dtype, casting='unsafe')

In addition, the test of the TypeError message is inaccurate since it implies this is a warning:

... To avoid this warning please use the scalar types ...

NumPy/Python version information:

1.21.1 3.8.11 (default, Aug  6 2021, 08:56:27)
[Clang 10.0.0 ]

[seberg]

Grrr :(. This is indeed supposed to be a warning and not an error.

That said, the warning should be valid and you should probably check bitfield.dtype actually is. It must be either something weird, or it is a datetime, or – fairly likely – a byte-swapped dtype. The last would be OK and more or less fixable right now using bitfield.dtype.type (admittedly, that is theoretically not perfectly future-proof for potentially new user-defined dtypes)

[mcara]

@seberg Thank you! Upon further inspection of the dtype, real data is indeed a big-endian: '>i2'.

np.bitwise_not(0, dtype=np.array([1], dtype=np.dtype('>i2')).dtype, casting='unsafe')

indeed raises the same exception.

[seberg]

Thanks for the report and checking.

Since in the code where it fails, mask is boolean anyway, I suspect you could just cast to bool rather than bitfield.dtype to begin with (I have not quite thought it through)? To keep it simple and avoid this subtlety (independent from the fact that we have to turn this into a warning in NumPy).

EDIT: The point being the bitwise_and actually looks like it is equivalent to a logical and here, so casting bitfield and ignore_mask to boolean would make no difference (that call could also use dtype=bool to force that cast, although, I am pretty much planning to build that forced cast for the logic operators into the next NumPy version anyway :)).

[mcara]

I am sorry, I am not sure I follow your suggestion. Could you, please, illustrate the idea? (Or make a PR)

The code is designed to create a boolean mask (or integer type with 1s and 0s) based on specific bits being turned on in the input data quality array values.

It has been quite some time since I wrote this code and I truly hope there was a good reason for doing things the way I did them, but fundamentally, using default parameters, the code is designed to compute (using pure Python):

not bool(dq & ~flag)

where dq is an integer like 2+4+8 and flag is 1<<i, i.e., 4. The tricky part, if I recall it correctly, was dealing with unsigned integers and largest flags. I just do not see how bitwise_and can be replaced with logical_and in the code.

[seberg]

Sorry, I misread/misunderstood the code... The bitfield.dtype.type workaround is probably the only good option and since you check for np.integer already, seems just as well.

The only other solution that might be reasonable is casting ignore_mask ahead of time (and do the thing in-place).