Skip to content

Commit

Permalink
More precise bounds for of_float conversion to small ints (#137)
Browse files Browse the repository at this point in the history
This patch improves the bounds used in of_float to decide whether the double can fit into an OCaml 63-bit tagged int.

The upper bound is 0x3ffffffffffffe00, which is exactly representable as a double (2^53-1)*2^9 = (2-2^-52) * 2^61 as it is 1.1...1 * 2^61 with 52 ones after the 1, and is less than 2^62, so it is representable as an OCaml int.   The double just after 0x3ffffffffffffe00 is 0x4000000000000000, or 2^62, which does not fit an OCaml int.

For the lower bound, -0x4000000000000000 is exactly representable as a double, but also the smallest number that fits an OCaml int.
  • Loading branch information
antoinemine committed Jul 18, 2023
1 parent 6cc5793 commit 1958fd6
Showing 1 changed file with 3 additions and 5 deletions.
8 changes: 3 additions & 5 deletions caml_z.c
Original file line number Diff line number Diff line change
Expand Up @@ -160,12 +160,10 @@ extern "C" {
#endif
#define Z_FITS_INT(v) ((v) >= Z_MIN_INT && (v) <= Z_MAX_INT)

/* Z_MAX_INT may not be representable exactly as a double => we use a
lower approximation to be safe
*/
/* greatest/smallest double that can fit in an int */
#ifdef ARCH_SIXTYFOUR
#define Z_MAX_INT_FL 0x3ffffffffffff000
#define Z_MIN_INT_FL (-Z_MAX_INT_FL)
#define Z_MAX_INT_FL 0x3ffffffffffffe00
#define Z_MIN_INT_FL (-0x4000000000000000)
#else
#define Z_MAX_INT_FL Z_MAX_INT
#define Z_MIN_INT_FL Z_MIN_INT
Expand Down

0 comments on commit 1958fd6

Please sign in to comment.