Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for large integer bit flags #56

Merged
merged 2 commits into from Jan 29, 2018

Conversation

mcara
Copy link
Member

@mcara mcara commented Jan 29, 2018

I was working on regression tests for the bitmask module when I have encountered a strange OverflowError that was happening in some cases when ignore_flags contained "too high" bits (for particular bitfield arguments).

Here are several examples:

>>> from stsci.tools import bitmask
>>> import numpy as np
>>> bitmask.bitfield_to_boolean_mask([-1,0,1], np.uint64(1<<63))
array([False,  True, False], dtype=bool)
>>> 
>>> # ... but:
>>> bitmask.bitfield_to_boolean_mask([-1,0,1], [np.uint64(1<<63)]) # flag internally converted to int
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/.../stsci.tools/lib/stsci/tools/bitmask.py", line 412, in bitfield_to_boolean_mask
    ignore_mask = np.bitwise_not(bitfield.dtype.type(ignore_mask))
OverflowError: Python int too large to convert to C long

The second command fails because internally interpret_bit_flags() called by bitfield_to_boolean_mask() converts np.uint64(1<<63) flag to Python int (or long). In the first command there is no such conversion. This PR forces interpret_bit_flags() to always return Python integer type. This will lead to the crash shown in the second statement above.

>>> # This works:
>>> bitmask.bitfield_to_boolean_mask(np.array([-1,0,1], dtype=np.uint64), [np.uint64(1<<63)])
array([False,  True, False], dtype=bool)
>>>
>>> # but the following fails:
>>> bitmask.bitfield_to_boolean_mask(np.array([-1,0,1], dtype=np.uint64), np.uint64(1<<63), True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/.../stsci.tools/lib/stsci/tools/bitmask.py", line 412, in bitfield_to_boolean_mask
    ignore_mask = np.bitwise_not(bitfield.dtype.type(ignore_mask))
OverflowError: Python int too large to convert to C long
>>>
>>> # At the same time:
>>> bitmask.bitfield_to_boolean_mask(np.array([-1,0,1], dtype=np.uint8), np.uint16(1<<15), True)
array([ True,  True,  True], dtype=bool)

... And finally,

>>> np.uint8(~int(np.uint16(1<<15)))
255
>>> np.uint64(1<<63)
9223372036854775808
>>> np.uint64(~int(np.uint64(1<<63)))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: Python int too large to convert to C long

Most of these issues are due to the fact when bit flags contain bits beyond what numpy.int64 or numpy.uint64 types can hold, numpy cannot "down-convert" those flags to largest representable by numpy type. Nevertheless, numpy can do this down-conversion from uint16 to uint8 as shown above.

This PR adresses these failures by dropping all the bits that cannot be represented by the type of the bitfield argument passed to bitfield_to_boolean_mask().

Finally, this PR adds support for the long type in Python 2.7.

@mcara mcara self-assigned this Jan 29, 2018
@@ -107,7 +117,7 @@ def interpret_bit_flags(bit_flags, flip_bits=None):

Returns
-------
bitmask : int or None
bitmask : int, or None
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpick: No need for comma here.

@mcara mcara merged commit cacbff1 into spacetelescope:master Jan 29, 2018
@mcara mcara deleted the uint-flags branch January 29, 2018 22:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants