You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I was working on regression tests for the bitmask module from the stsci.tools when I have encountered a strange OverflowError that was happening in some cases when ignore_flags contained "too high" bits (for particular bitfield arguments). The bitfield module in ccdproc should be having same issues. See spacetelescope/stsci.tools#56 and spacetelescope/stsci.tools#57 for more detail.
The second command fails because internally interpret_bit_flags() called by bitfield_to_boolean_mask() converts np.uint64(1<<63) flag to Python int (or long). In the first command there is no such conversion. This PR forces interpret_bit_flags() to always return Python integer type. This will lead to the crash shown in the second statement above.
>>># This works:>>>ccdproc.bitfield_to_boolean_mask(np.array([-1,0,1], dtype=np.uint64), [np.uint64(1<<63)])
array([False, True, False], dtype=bool)
>>>>>># but the following fails:>>>ccdproc.bitfield_to_boolean_mask(np.array([-1,0,1], dtype=np.uint64), np.uint64(1<<63), True)
Traceback (mostrecentcalllast):
File"<stdin>", line1, in<module>File"/.../ccdproc/ccdproc/core.py", line1862, inbitfield_to_boolean_maskgood_mask_value=False, dtype=bool)
File"/.../ccdproc/ccdproc/extern/bitfield.py", line421, inbitfield_to_boolean_maskignore_mask=np.bitwise_not(bitfield.dtype.type(ignore_mask))
OverflowError: PythoninttoolargetoconverttoClong>>>>>># At the same time:>>>ccdproc.bitfield_to_boolean_mask(np.array([-1,0,1], dtype=np.uint8), np.uint16(1<<15), True)
array([ True, True, True], dtype=bool)
Most of these issues are due to the fact when bit flags contain bits beyond what numpy.int64 or numpy.uint64 types can hold, numpy cannot "down-convert" those flags to largest representable by numpy type. Nevertheless, numpy can do this down-conversion from uint16 to uint8 as shown above.
This PR adresses these failures by dropping all the bits that cannot be represented by the type of the bitfield argument passed to bitfield_to_boolean_mask().
Finally, this PR adds support for the long type in Python 2.7.
The text was updated successfully, but these errors were encountered:
I was working on regression tests for the
bitmask
module from thestsci.tools
when I have encountered a strangeOverflowError
that was happening in some cases whenignore_flags
contained "too high" bits (for particularbitfield
arguments). Thebitfield
module inccdproc
should be having same issues. See spacetelescope/stsci.tools#56 and spacetelescope/stsci.tools#57 for more detail.Here are several examples:
The second command fails because internally
interpret_bit_flags()
called bybitfield_to_boolean_mask()
convertsnp.uint64(1<<63)
flag to Pythonint
(orlong
). In the first command there is no such conversion. This PR forcesinterpret_bit_flags()
to always return Python integer type. This will lead to the crash shown in the second statement above.... And finally,
Most of these issues are due to the fact when bit flags contain bits beyond what
numpy.int64
ornumpy.uint64
types can hold,numpy
cannot "down-convert" those flags to largest representable bynumpy
type. Nevertheless,numpy
can do this down-conversion fromuint16
touint8
as shown above.This PR adresses these failures by dropping all the bits that cannot be represented by the type of the
bitfield
argument passed tobitfield_to_boolean_mask()
.Finally, this PR adds support for the
long
type in Python 2.7.The text was updated successfully, but these errors were encountered: