Fixed range of random produced in BN_is_prime_fasttest_ex() #6547
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
According to the FIPS 186-4 C.3.1 (4.1) The range is expected to be 1 < b < w-1.
It was using 1 <= b < w (which is wrong by 1 on both ends).
This is because rand_range(n) returns 0 <= b < w-1 (and then it added 1).
So instead use 2 + rand_range(w-3).
There is also a check for the number '3' added that early exits (in the case where trial_divisions is not used).
Also removed a redundant check for is_zero after subtracting 1 as it already checks <= 1
Checklist