Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

integer overflow in linalg.sh_legendre on windows #58

Closed
jorenham opened this issue Oct 30, 2023 · 0 comments · Fixed by #69
Closed

integer overflow in linalg.sh_legendre on windows #58

jorenham opened this issue Oct 30, 2023 · 0 comments · Fixed by #69
Assignees
Labels
bug Something isn't working easy Simple to implement

Comments

@jorenham
Copy link
Owner

See: https://github.com/jorenham/Lmo/actions/runs/6687308385/job/18167801378

The np.int_ is a C long, which is 32bits in win64, because, well.. err... logic...?

... anyway, this is could be solved by replacing all np.int_'s by np.int64.

Another option could be to switch _sh_jacobi_f in the cases where _sh_jacobi_i will overflow given k and dtype in
sh_legendre, and similarly in sh_jacobi(k, a, b, dtype).

Note that in practise, this is only a problem in _lm._l_weights_pwm. The _lm._l_weights_ostat method is slower, slightly less precise, but a lot less likely to overflow.

For any users facing this issue; the workaround is to use e.g. trim=1e-15 instead of trim=0. This forces it to use the ostat weights, since the pwm weights can only handle integer trimming.

@jorenham jorenham added the bug Something isn't working label Oct 30, 2023
@jorenham jorenham self-assigned this Oct 30, 2023
@jorenham jorenham added the easy Simple to implement label Oct 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working easy Simple to implement
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant