Not all operand cases supported for BN_mod_exp

[rbost]

Not all cases for the BN_mod_exp operands are supported. Namely, for now, it only works when all the operands have the BN_FLG_CONSTTIME flag unset (call to BN_mod_exp_recp), or when all the operands have BN_FLG_CONSTTIME set and the modulus is odd (call to BN_mod_exp_mont_word). All the other cases fail (in particular when the BN_FLG_CONSTTIME flags are mixed).

Although this is not a problem when the function is used inside OpenSSL, it might be for third-party devs who rely on OpenSSL BN library.

For odd modulus, this seems easy to fix as one could fallback to the Montgomery constant-time implementation. For even modulus, it seems a bit more tricky. Note that calling BN_mod_exp with an even modulus and a BN_FLG_CONSTTIME flag operand could be explicitly forbidden in the documentation.

[bernd-edlinger]

@mattcaswell fixed by #5080 ?

[mattcaswell]

Not entirely no - that just fixed the most immediate and pressing problem. It might be as simple as a documentation fix - but it requires more thought.