Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for large integer bit flags #610

Closed
mcara opened this issue Jan 31, 2018 · 1 comment
Closed

Add support for large integer bit flags #610

mcara opened this issue Jan 31, 2018 · 1 comment

Comments

@mcara
Copy link
Contributor

mcara commented Jan 31, 2018

I was working on regression tests for the bitmask module from the stsci.tools when I have encountered a strange OverflowError that was happening in some cases when ignore_flags contained "too high" bits (for particular bitfield arguments). The bitfield module in ccdproc should be having same issues. See spacetelescope/stsci.tools#56 and spacetelescope/stsci.tools#57 for more detail.

Here are several examples:

>>> import ccdproc
>>> import numpy as np
>>> ccdproc.bitfield_to_boolean_mask([-1,0,1], np.uint64(1<<63))
array([ True, False,  True], dtype=bool)
>>> # ... but:
>>> ccdproc.bitfield_to_boolean_mask([-1,0,1], [np.uint64(1<<63)]) # flag internally converted to int
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/.../ccdproc/ccdproc/core.py", line 1862, in bitfield_to_boolean_mask
    good_mask_value=False, dtype=bool)
  File "/.../ccdproc/ccdproc/extern/bitfield.py", line 421, in bitfield_to_boolean_mask
    ignore_mask = np.bitwise_not(bitfield.dtype.type(ignore_mask))
OverflowError: Python int too large to convert to C long

The second command fails because internally interpret_bit_flags() called by bitfield_to_boolean_mask() converts np.uint64(1<<63) flag to Python int (or long). In the first command there is no such conversion. This PR forces interpret_bit_flags() to always return Python integer type. This will lead to the crash shown in the second statement above.

>>> # This works:
>>> ccdproc.bitfield_to_boolean_mask(np.array([-1,0,1], dtype=np.uint64), [np.uint64(1<<63)])
array([False,  True, False], dtype=bool)
>>>
>>> # but the following fails:
>>> ccdproc.bitfield_to_boolean_mask(np.array([-1,0,1], dtype=np.uint64), np.uint64(1<<63), True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/.../ccdproc/ccdproc/core.py", line 1862, in bitfield_to_boolean_mask
    good_mask_value=False, dtype=bool)
  File "/.../ccdproc/ccdproc/extern/bitfield.py", line 421, in bitfield_to_boolean_mask
    ignore_mask = np.bitwise_not(bitfield.dtype.type(ignore_mask))
OverflowError: Python int too large to convert to C long
>>>
>>> # At the same time:
>>> ccdproc.bitfield_to_boolean_mask(np.array([-1,0,1], dtype=np.uint8), np.uint16(1<<15), True)
array([ True,  True,  True], dtype=bool)

... And finally,

>>> np.uint8(~int(np.uint16(1<<15)))
255
>>> np.uint64(1<<63)
9223372036854775808
>>> np.uint64(~int(np.uint64(1<<63)))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: Python int too large to convert to C long

Most of these issues are due to the fact when bit flags contain bits beyond what numpy.int64 or numpy.uint64 types can hold, numpy cannot "down-convert" those flags to largest representable by numpy type. Nevertheless, numpy can do this down-conversion from uint16 to uint8 as shown above.

This PR adresses these failures by dropping all the bits that cannot be represented by the type of the bitfield argument passed to bitfield_to_boolean_mask().

Finally, this PR adds support for the long type in Python 2.7.

@crawfordsm
Copy link
Member

Thanks for taking care of this @mcara!

Closing with the merging of #611

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants