Skip to content

use reduced precision float base class#150

Merged
asfgit merged 1 commit intoapache:masterfrom
laimis:controlfp_s
Jun 19, 2015
Merged

use reduced precision float base class#150
asfgit merged 1 commit intoapache:masterfrom
laimis:controlfp_s

Conversation

@laimis
Copy link
Copy Markdown
Contributor

@laimis laimis commented Jun 18, 2015

We noticed that running tests in 32 bit CPUs was introducing floating point rounding issues and tests were failing as the same calculation could produce different result if executed multiple times in a single test. After some discussions on the mailing list it was determined that on 32 bit machines different instructions are used from 64 bit machines to process floating point numbers. Intermediate results when transferred with higher precision to the next operations eventually lead to rounding errors once a final calculation is returned as float.

From all the reading and trying several approaches so far, the only way I was able to consistently to get floating point conversion issues to go away is to pinvoke _controlfp_s to set floating point precision to 24 bits.

I got the idea from https://randomascii.wordpress.com/2013/07/16/floating-point-determinism/, see section "Floating-point settings (runtime)", last paragraph. controlfp seems to come up in other suggestions around this issue as well.

Here is the information on _controlfp_s: https://msdn.microsoft.com/en-us/library/c9676k6h.aspx. And it seems like there is an equivalent function in *nix environments although I haven't tried them or researched them in more depth yet.

For tests that rely on floating point calculations to be consistent, a variant of LuceneTestCase is used that sets the precision mask to 24 bits before each test run. I put this in the test as I don't think this code belongs to the core itself.

@asfgit asfgit merged commit c11e368 into apache:master Jun 19, 2015
@laimis laimis deleted the controlfp_s branch June 21, 2015 19:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants