Int64.float_of_bits weirdness on x86 ports

[vicuna]

Original bug ID: 5038
Reporter: @alainfrisch
Assigned to: @xavierleroy
Status: closed (set by @xavierleroy on 2010-05-19T12:35:37Z)
Resolution: not a bug
Priority: normal
Severity: minor
Category: ~DO NOT USE (was: OCaml general)
Related to: #4948

Bug description

Let's consider:

Printf.printf "%Lx\n%!" (Int64.bits_of_float (Int64.float_of_bits 0x7F_F0_00_00_00_00_00_01L));;

On Windows and Linux 64-bit ports, I get as expected:

7ff0000000000001

but on 32-ports, I get:

7ff8000000000001

As a consequence, Pervasives.nan is represented differently on 32-bit and 64-bit ports, and this gives different results when marshalling it.

[vicuna]

Comment author: @xavierleroy

(For background info about NaNs, see http://en.wikipedia.org/wiki/NaN.)

7FF0... is a signaling NaN while 7FF8... is the corresponding quiet NaN.

The IEEE semantics for signaling NaNs is that when they occur as argument to a float operation, an error is recorded somewhere or the operation traps, depending on the processor configuration. However, operations that produce NaNs (either because one argument was a NaN or because the operation is undefined like 0.0 / 0.0) always produce quiet NaNs, which do not cause errors nor traps.

What is happening in your example, I guess, is the following:

• On x86-32 bits, an fld; fstp sequence is generated, causing a double conversion 64 bits -> 80 bits -> 64 bits, and those conversions are treated like "operations" in the IEEE semantics sense, so the sNaN becomes a qNaN.

• On x86-64 bits, the C compiler generates movq or movsd instructions, which are not treated like IEEE "operations" and preserve sNaNs exactly.

This said, I see that Pervasives.nan is a signaling NaN, and I'm not sure why.

Finally, the qNaNs produced by undefined operations (0.0 / 0.0) are not specified down to the last bit, AFAIK, so bit-level reproducibility of NaNs across platforms is probably too much to expect.

[vicuna]

Comment author: @alainfrisch

Would it make sense to adapt Int64.bits_of_float and the marshaling function to normalize NaN values to a well-defined sequence of bits?

I agree this is a very minor point and bit-level equivalence for floating points across platforms cannot be achieved without the ia32/SSE2 port anyway ;-)

Feel free to close this report if you don't think it is worth the effort.

[vicuna]

Comment author: @xavierleroy

Normalization is a bit of an extreme solution, but if you really need it... (What's up with all those NaNs? Are you trying to price Greek bonds?)

A quick hack would be to define Pervasives.nan as 0xFFF8000... This seems to be the NaN that x86 processors produce on undefined operations (0.0 / 0.0, inf - inf, etc), both in x87 and SSE2 modes, and it is propagated unchanged through float operations. PowerPCs produce 0x7FF8000...., though, but if you're the only one to care about this issue and you care only about x86 processors...

[vicuna]

Comment author: @alainfrisch

I've locally patched our own serialisation routine (which uses Int64.bits_of_float) to get consistent results between 32- and 64-bit systems, so the problem is solved on our side and it's probably not worth taking the risk to break something else if nobody cares about the issue.

(Here, we use NaN is a very specific situation to denote absence of data in a float array. Conceptually, this should be a 'float option array', but it would be a lot less compact in memory. This is not related at all with my concern about NaN value popping up during real computations and incurring a huge slowdown with x87.)

[vicuna]

Comment author: @alainfrisch

Xavier: I suggest to close this issue unless you feel it is worth applying the quick hack you mention. (As said in the previous note, we use a work-around for this issue, so we do not really care any more.)

[vicuna]

Comment author: @xavierleroy

All right, I close this PR. I'm still unsure whether Pervasives.nan should be a qNaN or a sNaN, but for the moment I'm not changing it.