Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

np.left_shift and np.right_shift fail on np.uint64 scalar types (Trac #1931) #2524

Open
numpy-gitbot opened this issue Oct 19, 2012 · 7 comments
Open

Comments

@numpy-gitbot
Copy link

@numpy-gitbot numpy-gitbot commented Oct 19, 2012

Original ticket http://projects.scipy.org/numpy/ticket/1931 on 2011-08-10 by trac user tlatorre, assigned to unknown.

>>> np.uint64(5) << 1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: ufunc 'left_shift' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule 'safe'
>>> np.uint64(5) >> 3
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: ufunc 'right_shift' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule 'safe'
@numpy-gitbot
Copy link
Author

@numpy-gitbot numpy-gitbot commented Oct 19, 2012

@charris wrote on 2011-08-13

This is because the shift number is converted as a signed type and there is no signed integer type big enough to hold a uint64. The shift operators need to be special cased, they can also fail on Intel because the hardware masks out the unneeded bits, i.e., for int32 only the lower 5 bits are looked at.

@numpy-gitbot
Copy link
Author

@numpy-gitbot numpy-gitbot commented Oct 19, 2012

trac user parejkoj wrote on 2012-07-27

I was able to code around this with the following lambda:

lshift = lambda x,s: np.uint64(x)*np.uint64(2**s)

This works, but is not ideal. Getting this fixed in numpy would be very useful.

@jason-s
Copy link

@jason-s jason-s commented Nov 30, 2017

it's five years later... isn't this an important issue? unsigned integers should have no ambiguity about shifts.

@mhvk
Copy link
Contributor

@mhvk mhvk commented Dec 1, 2017

The easier work-around is to use an unsigned integer for the shift:

np.uint64(5) << np.uint64(1)

As @charris mentioned above, the problem is in the way dtypes are made uniform. It is not that trivial to fix generically (see also #5668), though perhaps the shift operators can be special-cased. (PR welcome, though, as said, this is not trivial, but requires fair understanding of how the internal loops in ufucs are decided on...).

@jason-s
Copy link

@jason-s jason-s commented Dec 1, 2017

The easier work-around is to use an unsigned integer for the shift:

np.uint64(5) << np.uint64(1)

...which does work if you know you are operating on numpy integers, but if you are using generic code (esp. something that has already been written in another module and is out of your control) like

def f(x):
    return x | (x >> 1)

then you can't make it work on both regular Python integers and np.uint64 types.

I'm not trying to state that this is trivial (wayyyy above my skillset) but I do think it should be a high priority.

@mhvk
Copy link
Contributor

@mhvk mhvk commented Dec 1, 2017

@jason-s - I agree that the problem is very annoying... Sadly, there not that many of us well-enough versed in how the ufuncs work internally...

Perhaps one could at least special-case scalars for the shift in the method, though, instead of relying on the general ufunc machinery. @charris: was this what you had in mind (if you can remember that after 6 years ;-).

@ahaldane
Copy link
Member

@ahaldane ahaldane commented Dec 1, 2017

See also #8002.

Personally, if we could do everything over again, I would favor redoing all the casting/coercion so that numpy casting essentially behaves like C casting, and follows the spirit of C-casting for cases that don't exist in C. The numpy casting rules are kind of weird and sometimes confusing (eg the conversion of uint64 + int64 to float64). But we've discussed this a lot on the list, it seems hard to change.

When I come across situations like this (see my comment there) I just wrap every single value in np.uint64.

Babicaa pushed a commit to paulscherrerinstitute/bsread_cpp that referenced this issue Aug 31, 2018
…mestamp is packed into unsigned 64bit int. To retreive the timestamp in python client the following snippet can be used:

                ##Note that the timestamp needs to be received as two unsigned 32 bit integers
                ## instead of a single 64bit int. This due to the fact the rshift on 64bit int doesn't
                ## work on numpy [numpy/numpy#2524]
                raw_data=socket.recv()
                value = numpy.fromstring(raw_data, dtype='u4')
                secPastEpich = value[1]
                nsec = value[0]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants