Fix a crash or unbounded allocation in RSA_padding_add_PKCS1_PSS_mgf1

[bernd-edlinger]

and RSA_verify_PKCS1_PSS_mgf1 with 512-bit RSA vs. sha-512.

@guidovranken: here is how I would like to fix the pss signature functions.
I hope this makes it clear what I meant with the comments on #2699.

I have test cases but don't know how to integrate that in the test framework.

openssl genrsa -out rsa.pem 512
openssl dgst -sign rsa.pem -sha512  -sigopt rsa_padding_mode:pss -sigopt rsa_pss_saltlen:-3 -sigopt rsa_mgf1_md:sha512 < /dev/null  > sig
=>crash
openssl dgst -sign rsa.pem -sha512  -sigopt rsa_padding_mode:pss -sigopt rsa_pss_saltlen:2147483647 -sigopt rsa_mgf1_md:sha1 < /dev/null  > sig
=>allocates 2GB and crashes if that succeeds

for the problem in RSA_padding_verify I need first a data block
that can be decrypted by the RSA key and clear text ends with 0xBC.

openssl dgst -sign rsa.pem -sha1  -sigopt rsa_padding_mode:pss -sigopt rsa_pss_saltlen:-3 -sigopt rsa_mgf1_md:sha512 < /dev/null  > sig
=> prepares test data, does not crash
openssl dgst -prverify rsa.pem -sha512  -sigopt rsa_padding_mode:pss -sigopt rsa_pss_saltlen:-3 -sigopt rsa_mgf1_md:sha512 < /dev/null  -signature sig
=> tries to allocate -1, which fails silently but is flagged by asan

[bernd-edlinger]

rebased.

[mattcaswell]

LGTM. What branches is this for?

[bernd-edlinger]

This is for master, I need to make an extra PRs for 1.1.0 and 1.0.2.
In 1.1.0 we do not have sLen = -3, so the test case can not be used as it is.
Porting the test case to 1.0.2 will probably not be worth the effort.

PS: Should I squash the two commits?

[mattcaswell]

No need to squash the commits IMO.

[dot-asm]

Applied to master. Thanks!