Addressing clang-UBSan Undefined Behavior Memcheck #12
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose
This PR fixes undefined behavior uncovered by the CRAN clang-UBSan memcheck raised earlier this week. cran_comments.md goes into detail on how this bug can be reproduced.
Explanation
The bug was found and fixed in src/prime_factors.cpp. The
prime_factors_
helper function takes anint
input that is cast todouble
, passed tostd::sqrt
, then cast back toint
. The return value ofstd::sqrt
when encountering domain error (input < 0) is undefined but expected to beNaN
, which cannot be represented when cast back toint
. Thus we have undefined behavior.The Fix
I have added an early exit for values that would cause a domain error.