You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When passed numeric arguments with more than ten significant digits, exponent_multiplier overflows. The mantissa is then multiplied by 1073441896 instead of the correct multiplier.
Changing exponent_multiplier from long to float fixes the issue, with only a trivial loss of precision.
The text was updated successfully, but these errors were encountered:
Thanks for this. I'm a bit wary of switching to float, but then again I guess it is uncommon to need more than ten digits of precision. I'm a bit snowed under at the moment, but will have a look when I can to see if there is another solution that doesn't require floating-point, otherwise switch exponent_multiplier to float as you suggest.
I didn't choose 10 digits! I was using TestController to generate automated test values and it produced the 10 digit numbers. I found a function in TC to trim the number of digits, so all is well.
When passed numeric arguments with more than ten significant digits, exponent_multiplier overflows. The mantissa is then multiplied by 1073441896 instead of the correct multiplier.
Changing exponent_multiplier from long to float fixes the issue, with only a trivial loss of precision.
The text was updated successfully, but these errors were encountered: