Added checks for input in mp_sqrtmod_prime that caused infinite loops

[czurnieden]

Bugfix regarding issue #486.

I cannot imagine how that issue might be exploited but because I cannot imagine it doesn't mean there is no way at all.

• check that parameter prime is odd and > 2
• the check if parameter prime is congruent to 3 mod 4 is done at bit-level now (which makes that check constant time, too)
• variables M, s, and i changed to native signed integer int (values cannot exceed INT_MAX )
• ( mp_sqrtmod_prime can get caught in an infinite loop if "prime" isn't prime #486 ) check for (Z|prime) == 0 in the first loop and return MP_VAL in that case. Although if prime (p) is an odd prime Jacobi(Z|p) == 0 for Z \cong 0 (mod p) there is at least one non-quadratic residue before Z>=p if p is an odd prime, hence no false positives.
• ( mp_sqrtmod_prime can get caught in an infinite loop if "prime" isn't prime #486 ) check condition M < i in the loop searching for smallest 0 < i < M for t^(2^i)

The extra checks are cheap and the function itself should even be a little bit faster together with less need for stack and heap with the other changes.

I did not pick any reviewer explicitly because of the current situation and its accompanying mess but I would appreciate it highly if somebody would take a look. Thank's!

[sjaeckel]

LGTM

@friedrichsenm can you please verify

[friedrichsenm]

I have one question about a kronecker symbol check in mp_sqrtmod_prime.c. When checking (n | prime) it might make sense to check instead that legendre != 1 instead of legendre == -1 to handle the case when it is 0.

[czurnieden]

When checking (n | prime) it might make sense to check instead that legendre != 1 instead of legendre == -1 to handle the case when it is 0.

Yepp, missed that. Good find, thanks!