set MGF1 digest correctly

[jamuir]

update rsa_set_ctx_params() so that the digest function used in the MGF1 construction is set correctly.

Fixes #19290

Testing:
scaro-axway gave code to reproduce the defect in the github issue. The code is supposed to set the rsa-oaep hash function to SHA2-256 and mgf1 hash function to SHA1. Here is the code:

  #include <openssl/evp.h>
  #include <openssl/core_names.h>
  #include <openssl/rsa.h>
  #include <stdio.h>

  void main(int argc, char **argv)
  {
    EVP_PKEY *key = EVP_RSA_gen(1024);
    EVP_PKEY_CTX *keyCtx = EVP_PKEY_CTX_new_from_pkey(0, key, 0);

    { // Set params
        int padding = RSA_PKCS1_OAEP_PADDING;
        OSSL_PARAM params[4];
        params[0] = OSSL_PARAM_construct_int(OSSL_SIGNATURE_PARAM_PAD_MODE, & padding);
        params[1] = OSSL_PARAM_construct_utf8_string(OSSL_ASYM_CIPHER_PARAM_OAEP_DIGEST,
                                                     SN_sha256, 0);
        params[2] = OSSL_PARAM_construct_utf8_string(OSSL_ASYM_CIPHER_PARAM_MGF1_DIGEST,
                                                     SN_sha1, 0);
        params[3] = OSSL_PARAM_construct_end();

        EVP_PKEY_encrypt_init_ex(keyCtx, params);
    }
    { // Read params
        OSSL_PARAM params[3];
        char       oaepmd[30] = { '\0' };
        char       mgf1md[30] = { '\0' };

        params[0] = OSSL_PARAM_construct_utf8_string(OSSL_ASYM_CIPHER_PARAM_OAEP_DIGEST,
                                                     oaepmd, sizeof(oaepmd));
        params[1] = OSSL_PARAM_construct_utf8_string(OSSL_ASYM_CIPHER_PARAM_MGF1_DIGEST,
                                                     mgf1md, sizeof(mgf1md));
        params[2] = OSSL_PARAM_construct_end();

        EVP_PKEY_CTX_get_params(keyCtx, params);
        printf("oaep = %s / mgf1 = %s\n", oaepmd, mgf1md);
    }
  }

before this commit:

  openssl/tmp$ make -B && ./issue-19290
  gcc -Wall -g -O0 -I../include -c issue-19290.c -o issue-19290.o
  gcc -Wall -g -O0 -I../include issue-19290.o ../libcrypto.a -o issue-19290
  oaep = SHA2-256 / mgf1 = SHA2-256

after this commit:

  openssl/tmp$ make -B && ./issue-19290
  gcc -Wall -g -O0 -I../include -c issue-19290.c -o issue-19290.o
  gcc -Wall -g -O0 -I../include issue-19290.o ../libcrypto.a -o issue-19290
  oaep = SHA2-256 / mgf1 = SHA1

Checklist

• documentation is added or updated
• tests are added or updated

[t8m]

Instead of having the code to reproduce the issue in the git commit message, would you like to add it as a testcase somewhere - for example to evp_extra_test.c?

[t8m]

Approving, although having a testcase would be nice. Also the commit message should be amended as it is not necessary to have the reproducer there. And there should be Fixes #19290 on a separate line in the commit message instead of having the reference in the commit message title.

[jamuir]

Thanks, Tomas. I will add a test-case and update the commit message.

[jamuir]

@t8m : I've made the updates.

[t8m]

Just a few minor adjustments needed for the testcase.

[beldmit]

LGTM

[slontis]

There are a few other places that use the 'str = ', like this, but they seem to do the right thing..

[openssl-machine]

This pull request is ready to merge

[t8m]

Merged to master and 3.0 branches. Thank you for your contribution.