MGF1 digest not set correctly when configuring RSA EVP_PKEY_CTX with OSSL_PARAMS

[scaro-axway]

Hi,

I came across an issue when using OpenSSL 3.0.5 API to encrypt and decrypt RSA data.
When using a single OSSL_PARAM array to configure both OAEP_DIGEST and MGF1_DIGEST, the mgf1 digest algorithm is not set as expected.

Here is a code sample showing the issue: mgf1 is set to "Sha256" instead of "Sha1".

#include <openssl/evp.h>
#include <openssl/core_names.h>
#include <openssl/rsa.h>
#include <stdio.h>

void main(int argc, char **argv)
{
  EVP_PKEY *key = EVP_RSA_gen(1024);
  EVP_PKEY_CTX *keyCtx = EVP_PKEY_CTX_new_from_pkey(0, key, 0);

  { // Set params
      int padding = RSA_PKCS1_OAEP_PADDING;
      OSSL_PARAM params[4];
      params[0] = OSSL_PARAM_construct_int(OSSL_SIGNATURE_PARAM_PAD_MODE, & padding);
      params[1] = OSSL_PARAM_construct_utf8_string(OSSL_ASYM_CIPHER_PARAM_OAEP_DIGEST,
                                                   SN_sha256, 0);
      params[2] = OSSL_PARAM_construct_utf8_string(OSSL_ASYM_CIPHER_PARAM_MGF1_DIGEST,
                                                   SN_sha1, 0);
      params[3] = OSSL_PARAM_construct_end();

      EVP_PKEY_encrypt_init_ex(keyCtx, params);
  }
  { // Read params
      OSSL_PARAM params[3];
      char       oaepmd[30] = { '\0' };
      char       mgf1md[30] = { '\0' };

      params[0] = OSSL_PARAM_construct_utf8_string(OSSL_ASYM_CIPHER_PARAM_OAEP_DIGEST,
                                                   oaepmd, sizeof(oaepmd));
      params[1] = OSSL_PARAM_construct_utf8_string(OSSL_ASYM_CIPHER_PARAM_MGF1_DIGEST,
                                                   mgf1md, sizeof(mgf1md));
      params[2] = OSSL_PARAM_construct_end();

      EVP_PKEY_CTX_get_params(keyCtx, params);
      printf("oaep = %s / mgf1 = %s\n", oaepmd, mgf1md);
  }
}

Trying to debug the issue lead to code located at rsa_enc.c:449.
In this use case, the 'str' pointer is no longer pointing to 'mdname': it was changed when processing the OAEP digest parameter at rsa_enc.c:440.

[jamuir]

I have a fix for this and will push it up shortly.