EVP_PKEY_encrypt seems to suffer from a memory leak

[qathy]

Hello,

I'm using openssl 3.0 (github checkout, compiled using clang on ubunutu 20.04, with -api 3.0 no-deprecated options):

LD_LIBRARY_PATH=/usr/local/openssl3.0-nodep-dbg-clang/lib64 /usr/local/openssl3.0-no
dep-dbg-clang/bin/openssl version -a
OpenSSL 3.0.11-dev  (Library: OpenSSL 3.0.11-dev )
built on: Thu Aug 31 16:14:52 2023 UTC
platform: linux-x86_64
options:  bn(64,64)
compiler: /usr/bin/clang-13 -fPIC -pthread -m64 -Wa,--noexecstack -Wall -O3 -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_BUILDING_OPENSSL -DNDEBUG
OPENSSLDIR: "/usr/local/openssl3.0-nodep-dbg-clang/ssl"
ENGINESDIR: "/usr/local/openssl3.0-nodep-dbg-clang/lib64/engines-3"
MODULESDIR: "/usr/local/openssl3.0-nodep-dbg-clang/lib64/ossl-modules"
Seeding source: os-specific
CPUINFO: OPENSSL_ia32cap=0xfefab2235f8bffff:0x9c27a9

Using the following very simple sample of code (stripped from real error management), clang sanitize-address reports memory leak which I don't seem to have the hand on:

int RSA_encrypt_with_OAEP_padding(EVP_PKEY* pkey, unsigned char* to, size_t* ptlen, const unsigned char* from, size_t flen, const char* hmacName)
{
    EVP_PKEY_CTX evp_rsa_ctx;
	EVP_MD digest;

	evp_rsa_ctx = EVP_PKEY_CTX_new(pkey, NULL);

	EVP_PKEY_encrypt_init(evp_rsa_ctx);

	EVP_PKEY_CTX_set_rsa_padding(evp_rsa_ctx, RSA_PKCS1_OAEP_PADDING);

	digest = EVP_MD_fetch(nullptr, hmacName, nullptr));

	/* as specified, use same hash function for both OEAP and MGF operations */
	EVP_PKEY_CTX_set_rsa_mgf1_md(evp_rsa_ctx, digest);
	EVP_PKEY_CTX_set_rsa_oaep_md(evp_rsa_ctx, digest);

	EVP_PKEY_encrypt(evp_rsa_ctx, to, ptlen, from, flen);

	return 1;
}

Extract of binary (compiled with clang and option -fsanitize=address) output:

Indirect leak of 12416 byte(s) in 16 object(s) allocated from:
#0 0x4acadd in malloc (/DT/DEV/dev/lucp/External/libsec/clang/ut_sec+0x4acadd)
#1 0x7f364e5d91ab in CRYPTO_malloc /DT/DEV/dev/lucp/External/openssl/crypto/mem.c:190:12
#2 0x7f364e5d91e2 in CRYPTO_zalloc /DT/DEV/dev/lucp/External/openssl/crypto/mem.c:197:11
#3 0x7f364e48919a in bn_expand_internal /DT/DEV/dev/lucp/External/openssl/crypto/bn/bn_lib.c:281:13
#4 0x7f364e48902a in bn_expand2 /DT/DEV/dev/lucp/External/openssl/crypto/bn/bn_lib.c:305:23
#5 0x7f364e489447 in bn_wexpand /DT/DEV/dev/lucp/External/openssl/crypto/bn/bn_lib.c:1021:37
#6 0x7f364e48a13f in BN_set_bit /DT/DEV/dev/lucp/External/openssl/crypto/bn/bn_lib.c:695:13
#7 0x7f364e48c74d in BN_MONT_CTX_set /DT/DEV/dev/lucp/External/openssl/crypto/bn/bn_mont.c:395:10
#8 0x7f364e48c994 in BN_MONT_CTX_set_locked /DT/DEV/dev/lucp/External/openssl/crypto/bn/bn_mont.c:451:10
#9 0x7f364e63c41e in rsa_ossl_public_encrypt /DT/DEV/dev/lucp/External/openssl/crypto/rsa/rsa_ossl.c:142:14
#10 0x7f364e635d0b in RSA_public_encrypt /DT/DEV/dev/lucp/External/openssl/crypto/rsa/rsa_crpt.c:36:12
#11 0x7f364e6d1ea4 in rsa_encrypt /DT/DEV/dev/lucp/External/openssl/providers/implementations/asymciphers/rsa_enc.c:183:15
#12 0x7f364e574011 in EVP_PKEY_encrypt /DT/DEV/dev/lucp/External/openssl/crypto/evp/asymcipher.c:246:11

[mattcaswell]

evp_rsa_ctx = EVP_PKEY_CTX_new(pkey, NULL);

The evp_rsa_ctx value needs to be freed when you have finished using it via EVP_PKEY_CTX_free(evp_rsa_ctx).

digest = EVP_MD_fetch(nullptr, hmacName, nullptr));

The digest value also needs to be freed (EVP_MD_free(digest)) after use.

[qathy]

That's true, of course.
But despite the forgotten calls in the copy I've given, they are present in the real code.
That was the first thing I've checked..
Nevertheless, sanitize reports the leaks even with both EVP_PKEY_CTX_free() and EVP_MD_free() properly used.

[qathy]

In fact, I opened this as an issue, just because this sort of discrepancy between expected behavior (memory properly freed by the appropriate functions) and the observed one (complaint from sanitizer, but also from valgrind).

[t8m]

Could you please paste the full leak sanitizer output? Or is this everything?

How do you release the EVP_PKEY object?

[qathy]

This may be the answer.
Just have to check this with colleagues

[qathy]

Thank you @t8m !
Yes, EVP_PKEY_free is the right answer.
Even though we're trying hard to call it, in some use cases.