New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix error propagatation in BN_check_prime() #19314
Conversation
crypto/bn/bn_prime.c
Outdated
@@ -308,9 +308,10 @@ static int bn_is_prime_int(const BIGNUM *w, int checks, BN_CTX *ctx, | |||
goto err; | |||
#endif | |||
|
|||
ret = ossl_bn_miller_rabin_is_prime(w, checks, ctx, cb, 0, &status); | |||
if (!ret) | |||
if (ossl_bn_miller_rabin_is_prime(w, checks, ctx, cb, 0, &status) != 1) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As the ossl_bn_miller_rabin_is_prime return is boolean could you please keep the original condition?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done!
BN_check_prime() is supposed to return 0 for a composite number and -1 on error. Properly translate the return value of the internal function ossl_bn_miller_rabin_is_prime(), where 0 means an error. The confusion prevented BN_GENCB callbacks from aborting the primality test or key generation routines utilizing this.
2ccd1d0
to
f1f2143
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Nice catch!
My only nit is on the commit message, to fix the typo on "propagation".
This seems minor enough that it maybe can be fixed by whomever merges, without requiring reapproval.
This pull request is ready to merge |
BN_check_prime() is supposed to return 0 for a composite number and -1 on error. Properly translate the return value of the internal function ossl_bn_miller_rabin_is_prime(), where 0 means an error. The confusion prevented BN_GENCB callbacks from aborting the primality test or key generation routines utilizing this. Reviewed-by: Tomas Mraz <tomas@openssl.org> Reviewed-by: Nicola Tuveri <nic.tuv@gmail.com> Reviewed-by: Paul Dale <pauli@openssl.org> (Merged from #19314)
BN_check_prime() is supposed to return 0 for a composite number and -1 on error. Properly translate the return value of the internal function ossl_bn_miller_rabin_is_prime(), where 0 means an error. The confusion prevented BN_GENCB callbacks from aborting the primality test or key generation routines utilizing this. Reviewed-by: Tomas Mraz <tomas@openssl.org> Reviewed-by: Nicola Tuveri <nic.tuv@gmail.com> Reviewed-by: Paul Dale <pauli@openssl.org> (Merged from #19314) (cherry picked from commit 0b38676)
Merged to both branches, thanks for the contribution. |
BN_check_prime() is supposed to return 0 for a composite number and -1 on error. Properly translate the return value of the internal function ossl_bn_miller_rabin_is_prime(), where 0 means an error. The confusion prevented BN_GENCB callbacks from aborting the primality test or key generation routines utilizing this. Reviewed-by: Tomas Mraz <tomas@openssl.org> Reviewed-by: Nicola Tuveri <nic.tuv@gmail.com> Reviewed-by: Paul Dale <pauli@openssl.org> (Merged from openssl#19314)
BN_check_prime() is supposed to return 0 for a composite number and -1 on error. Properly translate the return value of the internal function ossl_bn_miller_rabin_is_prime(), where 0 means an error.
The confusion prevented BN_GENCB callbacks from aborting the primality test or key generation routines utilizing this.
This patch applies to 3.0 and master.
Checklist