Large/small exponents frequently give inappropriate results #127

Closed
trevyn opened this Issue Mar 16, 2012 · 10 comments

Comments

Projects
None yet
2 participants

trevyn commented Mar 16, 2012

3.1a3's DisplayPretty:

in: [5.0e-200]
out: [500000000000000000000000000000000000000000000000000000000]

in: [5.0e-300]
out: [0.00000000000000000000000000000000000000000005]

in: [5.0e200]
out: [0.00000000000000000000000000000000000000000000000000000005]

in: [5.0e300]
out: [500000000000000000000000000000000000000000000]

Owner

stig commented Mar 16, 2012

Hmmm… those are indeed very large exponents. I use NSDecimalNumber to handle anything that won't fit in a double. But these too may have an upper limit. I should probably throw an over/underflow error in these cases. Let me check it a bit more.

Stig

Sent from my iPhone

On 16 Mar 2012, at 05:51, Edenreply@reply.github.com wrote:

3.1a3's DisplayPretty:

in: [5.0e-200]
out: [500000000000000000000000000000000000000000000000000000000]

in: [5.0e-300]
out: [0.00000000000000000000000000000000000000000005]

in: [5.0e200]
out: [0.00000000000000000000000000000000000000000000000000000005]

in: [5.0e300]
out: [500000000000000000000000000000000000000000000]


Reply to this email directly or view it on GitHub:
#127

trevyn commented Mar 16, 2012

Ah, I see. Is there a reason you're using NSDecimalNumber? doubles can have exponent from -1022 to 1023: http://en.wikipedia.org/wiki/Double-precision_floating-point_format#Exponent_encoding

trevyn commented Mar 16, 2012

Oops, base 2 exponent from -1022 to 1023, so:

4.9406564584124654e−324 (Min subnormal positive double)
1.7976931348623157e308 (Max Double)

Which is actually how I ended up here in the first place, I was parsing 5e-324, JavaScript's MIN_VALUE.

Owner

stig commented Mar 16, 2012

Um, I don't remember. (I thought I was using NSDecimalNumber because it had higher precision than double.) In other words, more research needed… :-(

Sent from my iPhone

On 16 Mar 2012, at 18:29, Edenreply@reply.github.com wrote:

Oops, base 2 exponent from -1022 to 1023, so:

4.9406564584124654e−324 (Min subnormal positive double)
1.7976931348623157e308 (Max Double)

Which is actually how I ended up here in the first place, I was parsing 5e-324, JavaScript's MIN_VALUE.


Reply to this email directly or view it on GitHub:
#127 (comment)

Owner

stig commented Mar 19, 2012

Yep, that is a bug. I've been treating NSDecimalNumber as a BigNum, which is not the case.

Owner

stig commented Mar 20, 2012

This is a bit of a wasp's nest. As you say an NSNumber representing a double supports base 10 exponents up to 308, with just under 17 decimal digits of precision. NSDecimalNumber doesn't support as high maximum values (base 10 exponent only up to 127) but in exchange supports much higher precision (38 digits).

There are several options:

  1. Always use NSNumber (i.e. a double) for floating point numbers. This would support larger/smaller values than now, at the cost of lower precision.
  2. Continue to use NSDecimalNumber, meaning we support high precision but not so large/small numbers.
  3. Try to do intelligently pick the best choice of the above based on the length of the mantissa and/or exponent.

Regardless of the method we choose, we should return an error unless we can faithfully represent the number that we parsed. (I.e. if there would be a rounding error.)

Owner

stig commented Mar 20, 2012

First (and ugly) cut at 2 + error on overflow above: master...issue-127

trevyn commented Mar 20, 2012

For purely selfish reasons, I support either 1 or 3. ;) Also, if you take the "JSON is a subset of JavaScript" view, double is JavaScript's native number format.

Owner

stig commented Mar 20, 2012

Yeah, 2 is a bit of a cop-out. I agree that 1 or 3 would be better solutions, but I would think that initially reporting an error rather than silently truncating the number is a good first step while I think about how to implement the improvement. The number parsing is getting rather hairy and I'd like to take this opportunity to clean it up a bit.

stig closed this in b95c5b6 Mar 21, 2012

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment