Skip to content

Commit

Permalink
Fixes AcademySoftwareFoundation#252, incorrect math computing half di…
Browse files Browse the repository at this point in the history
…gits

Based on float / double math for base 10 digits, with 1 bit of rounding
error, the equation should be floor( mantissa_digits - 1 ) * log10(2) ),
which in the case of half becomes floor( 10 * log10(2) ) or 3

Signed-off-by: Kimball Thurston <kdt3rd@gmail.com>
  • Loading branch information
kdt3rd committed Jul 21, 2019
1 parent 9aa10cf commit 0000624
Showing 1 changed file with 4 additions and 1 deletion.
5 changes: 4 additions & 1 deletion IlmBase/Half/half.h
Expand Up @@ -274,9 +274,12 @@ HALF_EXPORT void printBits (char c[35], float f);
#define HALF_MANT_DIG 11 // Number of digits in mantissa
// (significand + hidden leading 1)

#define HALF_DIG 2 // Number of base 10 digits that
//
// floor( (HALF_MANT_DIG - 1) * log10(2) ) => 3.01... -> 3
#define HALF_DIG 3 // Number of base 10 digits that
// can be represented without change

// ceil(HALF_MANT_DIG * log10(2) + 1) => 4.31... -> 5
#define HALF_DECIMAL_DIG 5 // Number of base-10 digits that are
// necessary to uniquely represent all
// distinct values
Expand Down

0 comments on commit 0000624

Please sign in to comment.