2 hybrid KEMs failing #23

baentsch · 2021-06-03T13:51:56Z

Interop tests against https://test.openquantumsafe.org fail for p384_kyber90s768 and p256_oqs_kem_default

baentsch · 2021-06-03T13:56:29Z

Tagging @bhess : Is there anything in your mind that could explain this specific failure combination? I'm puzzled.

bhess · 2021-06-03T15:13:24Z

Thanks @baentsch for the report. It seems that we don't register "oqs_kem_default" which explains why the second test fails.

I will take a closer look at p384_kyber90s768. Just to confirm: kyber90s768 (just KEM) and p384_kyber768 work fine? And was p384_kyber90s768 interop so far tested e.g. between BoringSSL and OSSL111?

baentsch · 2021-06-03T16:35:39Z

kyber90s768 (just KEM) and p384_kyber768 work fine?

Yes: You can also see for yourself: ./testrun.sh openquantumsafe/oqs-ossl3 (in oqs-demos/openssl3/fulltest).

Testing ecdsap256:
    Tested KEM * successfully.
    Tested KEM X25519 successfully.
Error testing p256_oqs_kem_default: 
 Call to SSL_CONF_cmd(-groups, p256_oqs_kem_default) failed
 

    Tested KEM frodo640aes successfully.
    Tested KEM frodo640shake successfully.
    Tested KEM frodo976aes successfully.
    Tested KEM frodo976shake successfully.
    Tested KEM frodo1344aes successfully.
    Tested KEM frodo1344shake successfully.
    Tested KEM bike1l1cpa successfully.
    Tested KEM bike1l3cpa successfully.
    Tested KEM kyber512 successfully.
    Tested KEM kyber768 successfully.
    Tested KEM kyber1024 successfully.
    Tested KEM ntru_hps2048509 successfully.
    Tested KEM ntru_hps2048677 successfully.
    Tested KEM ntru_hps4096821 successfully.
    Tested KEM ntru_hrss701 successfully.
    Tested KEM lightsaber successfully.
    Tested KEM saber successfully.
    Tested KEM firesaber successfully.
    Tested KEM sidhp434 successfully.
    Tested KEM sidhp503 successfully.
    Tested KEM sidhp610 successfully.
    Tested KEM sidhp751 successfully.
    Tested KEM sikep434 successfully.
    Tested KEM sikep503 successfully.
    Tested KEM sikep610 successfully.
    Tested KEM sikep751 successfully.
    Tested KEM bike1l1fo successfully.
    Tested KEM bike1l3fo successfully.
    Tested KEM kyber90s512 successfully.
    Tested KEM kyber90s768 successfully.
    Tested KEM kyber90s1024 successfully.
    Tested KEM hqc128 successfully.
    Tested KEM hqc192 successfully.
    Tested KEM hqc256 successfully.
    Tested KEM ntrulpr653 successfully.
    Tested KEM ntrulpr761 successfully.
    Tested KEM ntrulpr857 successfully.
    Tested KEM sntrup653 successfully.
    Tested KEM sntrup761 successfully.
    Tested KEM sntrup857 successfully.
    Tested KEM p256_frodo640aes successfully.
    Tested KEM p256_frodo640shake successfully.
    Tested KEM p384_frodo976aes successfully.
    Tested KEM p384_frodo976shake successfully.
    Tested KEM p521_frodo1344aes successfully.
    Tested KEM p521_frodo1344shake successfully.
    Tested KEM p256_bike1l1cpa successfully.
    Tested KEM p384_bike1l3cpa successfully.
    Tested KEM p256_kyber512 successfully.
    Tested KEM p384_kyber768 successfully.
    Tested KEM p521_kyber1024 successfully.
    Tested KEM p256_ntru_hps2048509 successfully.
    Tested KEM p384_ntru_hps2048677 successfully.
    Tested KEM p521_ntru_hps4096821 successfully.
    Tested KEM p384_ntru_hrss701 successfully.
    Tested KEM p256_lightsaber successfully.
    Tested KEM p384_saber successfully.
    Tested KEM p521_firesaber successfully.
    Tested KEM p256_sidhp434 successfully.
    Tested KEM p256_sidhp503 successfully.
    Tested KEM p384_sidhp610 successfully.
    Tested KEM p521_sidhp751 successfully.
    Tested KEM p256_sikep434 successfully.
    Tested KEM p256_sikep503 successfully.
    Tested KEM p384_sikep610 successfully.
    Tested KEM p521_sikep751 successfully.
    Tested KEM p256_bike1l1fo successfully.
    Tested KEM p384_bike1l3fo successfully.
    Tested KEM p256_kyber90s512 successfully.
Error testing p384_kyber90s768: 
 CONNECTED(00000003)
481B31CD447F0000:error:0A000417:SSL routines:ssl3_read_bytes:sslv3 alert illegal parameter:ssl/record/rec_layer_s3.c:1588:SSL alert number 47
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 1521 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
 

    Tested KEM p521_kyber90s1024 successfully.
    Tested KEM p256_hqc128 successfully.
    Tested KEM p384_hqc192 successfully.
    Tested KEM p521_hqc256 successfully.
    Tested KEM p256_ntrulpr653 successfully.
    Tested KEM p384_ntrulpr761 successfully.
    Tested KEM p384_ntrulpr857 successfully.
    Tested KEM p256_sntrup653 successfully.
    Tested KEM p384_sntrup761 successfully.
    Tested KEM p384_sntrup857 successfully.
  Successfully concluded testing ecdsap256

The first error message indeed is different, so your assessment probably already has nailed that one. Nice.

And was p384_kyber90s768 interop so far tested e.g. between BoringSSL and OSSL111

Yes, here's the test between OSSL111 and nginx:

[...all other SIGs...]
Testing rsa3072_sphincsshake256128frobust:
[...all other KEMs with that SIG....]
    Tested KEM p384_kyber90s768 successfully.

Interop between OSSL111 and Boring should be tested with each PR & merge. Last successful run here

bhess · 2021-06-03T22:29:12Z

Thanks @baentsch, I could reproduce the issue.

It turns out that the origin is an incorrect claimed_security_level for kyber90s768 (1, should be 3), which is inherited from the upstream-yml:
https://github.com/open-quantum-safe/liboqs/blob/f8e339dcae8432b41e60621d25a4a3596bd14a14/src/kem/kyber/kem_kyber_768_90s.c#L18

Oqsprovider combines hybrids according to the claimed security level taken from the liboqs algorithm struct. So in this case, the hybrid used was actually p256/kyber90s768. The self-tests pass that way, but luckily we have the interop tests.

I can add two PRs to solve the issue: 1. in oqsprovider to register "oqs_kem_default", 2. in liboqs to correct the claimed security level (temporarily as a patch for copy-from-upstream to not block this here).
..and then a third one in pqcrystals to correct the yml-file used by copy-from-upstream.

baentsch · 2021-06-04T05:06:21Z

Thanks for finding the issue so quickly and looking forward to the PRs.

It turns out that the origin is an incorrect claimed_security_level for kyber90s768

This is a bummer. But then isn't there something more broken: If the claimed NIST level is wrong, why did the hybrid read "p384_kyber90s768" and not "p256_kyber90s768"? That mismatch would have been obvious immediately. In any case, this makes https://github.com/open-quantum-safe/liboqs/discussions/1011 moot.

baentsch · 2021-06-04T05:48:11Z

..and then a third one in pqcrystals to correct the yml-file used by copy-from-upstream

One more thought: Instead of doing a temporary patch in liboqs' copy_from_upstream (that will have to be reverted again) shouldn't it be quicker to correct it first in pqcrystals? For all I know that code base does not depend on this YML, so a fix should be seamless/quick to approve there, no? If that code base had used the YML (e.g., to generate the code constants), the error would never have occurred (or become obvious very fast with KATs failing), right?

bhess · 2021-06-04T06:51:38Z

If the claimed NIST level is wrong, why did the hybrid read "p384_kyber90s768" and not "p256_kyber90s768"? That mismatch would have been obvious immediately.

It isn't optimal. The names (p384_..) are generated at compile time using the bit_security fields from https://github.com/open-quantum-safe/openssl/blob/OQS-OpenSSL_1_1_1-stable/oqs-template/generate.yml. The hybrid pairs are derived from the claimed_security_level which is available at runtime. We seem to not automatically sync these fields (which we should probably do/check from time to time) so it can lead to inconsistencies.
Since OSSL111/Boring use only the bit_security yml fields it doesn't affect them. I think we should do the same in oqsprovider to be more robust.

One more thought: Instead of doing a temporary patch in liboqs' copy_from_upstream (that will have to be reverted again) shouldn't it be quicker to correct it first in pqcrystals? For all I know that code base does not depend on this YML, so a fix should be seamless/quick to approve there, no? If that code base had used the YML (e.g., to generate the code constants), the error would never have occurred (or become obvious very fast with KATs failing), right?

The ymls from pqcrystals were added for oqs and yes I think their code base doesn't depend on it. I agree that it's better to not do a patch and revert it later again. If I do the fix mentioned above to only use bit_security in oqsprovider, the fix doesn't depend on fixing the claimed_security_level, which makes it not a blocker for here (although it should be quick to approve in upstream).

baentsch · 2021-06-04T07:14:22Z

Thanks & agreed to all of the above. FYI, I'm working on open-quantum-safe/openssl#315 (working around the issues you also point out makes this indeed a bit more convoluted than I originally thought). It'd be goodness if oqsprovider wouldn't have the same weakness (as per your suggestion in the fix). Nevertheless, this is a blocker for liboqs.

This commit fixes various memory leaks that have been found using ASan. 1. Memory leak in `oqs_test_signatures`. === In `oqs_test_signatures`, [`OPENSSL_free`] was used with `EVP_PKEY_CTX`. However, the right destructor function for `EVP_PKEY_CTX` is [`EVP_PKEY_CTX_free`]. This commmit fixes this minor bug. [`OPENSSL_free`](https://www.openssl.org/docs/man3.0/man3/OPENSSL_free.html) [`EVP_PKEY_CTX_free`](https://www.openssl.org/docs/man3.0/man3/EVP_PKEY_CTX_free.html) 2. a selftest key that wasn't freed. === In `oqsprov/oqsprov_keys.c`, `d2i_PrivateKey` is used to verify that we the key we've just generated is sound. However, the returned object wasn't freed. ASan trace that helped me finding this one: ``` Indirect leak of 96 byte(s) in 4 object(s) allocated from: #0 0x5651eef1428e in malloc (/home/pawn/work/oqs-provider/build/test/oqs_test_signatures+0xb928e) (BuildId: 6cc7ebd4bc73a0c78b2ab8d7b9dcb8b165d88e74) open-quantum-safe#1 0x7fb7a23a946b in CRYPTO_malloc /home/pawn/work/openssl/crypto/mem.c:190:12 open-quantum-safe#2 0x7fb7a23a94a2 in CRYPTO_zalloc /home/pawn/work/openssl/crypto/mem.c:197:11 open-quantum-safe#3 0x7fb7a209cdcd in BN_new /home/pawn/work/openssl/crypto/bn/bn_lib.c:247:16 open-quantum-safe#4 0x7fb7a2224a00 in ossl_ec_GFp_mont_group_set_curve /home/pawn/work/openssl/crypto/ec/ecp_mont.c:169:11 open-quantum-safe#5 0x7fb7a220f5fa in EC_GROUP_set_curve /home/pawn/work/openssl/crypto/ec/ec_lib.c:562:12 open-quantum-safe#6 0x7fb7a2202d75 in EC_GROUP_new_curve_GFp /home/pawn/work/openssl/crypto/ec/ec_cvt.c:61:10 open-quantum-safe#7 0x7fb7a220184d in ec_group_new_from_data /home/pawn/work/openssl/crypto/ec/ec_curve.c:3179:22 open-quantum-safe#8 0x7fb7a2200ff9 in EC_GROUP_new_by_curve_name_ex /home/pawn/work/openssl/crypto/ec/ec_curve.c:3287:19 open-quantum-safe#9 0x7fb7a2201ed9 in EC_GROUP_new_by_curve_name /home/pawn/work/openssl/crypto/ec/ec_curve.c:3303:12 open-quantum-safe#10 0x7fb7a21f84e2 in EC_GROUP_new_from_ecpkparameters /home/pawn/work/openssl/crypto/ec/ec_asn1.c:863:20 open-quantum-safe#11 0x7fb7a21f8f28 in d2i_ECPrivateKey /home/pawn/work/openssl/crypto/ec/ec_asn1.c:956:22 open-quantum-safe#12 0x7fb7a270b4b5 in der2key_decode /home/pawn/work/openssl/providers/implementations/encode_decode/decode_der2key.c:220:19 open-quantum-safe#13 0x7fb7a226eae8 in decoder_process /home/pawn/work/openssl/crypto/encode_decode/decoder_lib.c:971:14 open-quantum-safe#14 0x7fb7a226d106 in OSSL_DECODER_from_bio /home/pawn/work/openssl/crypto/encode_decode/decoder_lib.c:82:10 open-quantum-safe#15 0x7fb7a226f007 in OSSL_DECODER_from_data /home/pawn/work/openssl/crypto/encode_decode/decoder_lib.c:157:9 open-quantum-safe#16 0x7fb7a200e123 in d2i_PrivateKey_decoder /home/pawn/work/openssl/crypto/asn1/d2i_pr.c:74:11 open-quantum-safe#17 0x7fb7a200dbba in d2i_PrivateKey_ex /home/pawn/work/openssl/crypto/asn1/d2i_pr.c:162:11 open-quantum-safe#18 0x7fb7a200e3d2 in d2i_PrivateKey /home/pawn/work/openssl/crypto/asn1/d2i_pr.c:172:12 open-quantum-safe#19 0x7fb79dc6a22e in oqsx_key_gen_evp_key /home/pawn/work/oqs-provider/oqsprov/oqsprov_keys.c:1036:25 open-quantum-safe#20 0x7fb79dc68f3f in oqsx_key_gen /home/pawn/work/oqs-provider/oqsprov/oqsprov_keys.c:1077:16 open-quantum-safe#21 0x7fb79dc7164a in oqsx_genkey /home/pawn/work/oqs-provider/oqsprov/oqs_kmgmt.c:496:9 open-quantum-safe#22 0x7fb79dc6d9f8 in oqsx_gen /home/pawn/work/oqs-provider/oqsprov/oqs_kmgmt.c:509:12 open-quantum-safe#23 0x7fb7a23373c7 in evp_keymgmt_gen /home/pawn/work/openssl/crypto/evp/keymgmt_meth.c:366:12 open-quantum-safe#24 0x7fb7a2334b58 in evp_keymgmt_util_gen /home/pawn/work/openssl/crypto/evp/keymgmt_lib.c:517:20 open-quantum-safe#25 0x7fb7a235a775 in EVP_PKEY_generate /home/pawn/work/openssl/crypto/evp/pmeth_gn.c:189:13 open-quantum-safe#26 0x5651eef4f2bd in test_oqs_signatures /home/pawn/work/oqs-provider/test/oqs_test_signatures.c:39:48 open-quantum-safe#27 0x5651eef4ee3d in main /home/pawn/work/oqs-provider/test/oqs_test_signatures.c:114:17 open-quantum-safe#28 0x7fb7a1b671c9 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 ``` 3. a `EVP_PKEY_CTX` that wasn't freed. === In `oqsx_key_free`, `key->oqsx_provider_ctx.oqsx_evp_ctx->ctx` is freed if the keytype is `ECP_HYB_KEM` or `ECX_HYB_KEM`. However, `key->oqsx_provider_ctx.oqsx_evp_ctx->ctx` is also used for signatures. In that case, a call to `EVP_PKEY_CTX_free` was missing. ASan trace that helped me finding this one: ``` Indirect leak of 7 byte(s) in 2 object(s) allocated from: #0 0x55b6fc8d427e in malloc (/home/pawn/work/oqs-provider/build/test/oqs_test_signatures+0xb927e) (BuildId: 3a8bff15d3315ba7ac1746fdc7b67eb464c607a1) open-quantum-safe#1 0x7ff34dfa946b in CRYPTO_malloc /home/pawn/work/openssl/crypto/mem.c:190:12 open-quantum-safe#2 0x7ff34dfac50a in CRYPTO_strndup /home/pawn/work/openssl/crypto/o_str.c:43:11 open-quantum-safe#3 0x7ff34df9d152 in ossl_algorithm_get1_first_name /home/pawn/work/openssl/crypto/core_algorithm.c:195:11 open-quantum-safe#4 0x7ff34df353a1 in keymgmt_from_algorithm /home/pawn/work/openssl/crypto/evp/keymgmt_meth.c:50:31 open-quantum-safe#5 0x7ff34df15e44 in construct_evp_method /home/pawn/work/openssl/crypto/evp/evp_fetch.c:219:14 open-quantum-safe#6 0x7ff34df9e3ce in ossl_method_construct_this /home/pawn/work/openssl/crypto/core_fetch.c:109:19 open-quantum-safe#7 0x7ff34df9d740 in algorithm_do_map /home/pawn/work/openssl/crypto/core_algorithm.c:77:13 open-quantum-safe#8 0x7ff34df9ce88 in algorithm_do_this /home/pawn/work/openssl/crypto/core_algorithm.c:122:15 open-quantum-safe#9 0x7ff34dfcce3e in ossl_provider_doall_activated /home/pawn/work/openssl/crypto/provider_core.c:1423:14 open-quantum-safe#10 0x7ff34df9ca64 in ossl_algorithm_do_all /home/pawn/work/openssl/crypto/core_algorithm.c:162:9 open-quantum-safe#11 0x7ff34df9dd64 in ossl_method_construct /home/pawn/work/openssl/crypto/core_fetch.c:153:5 open-quantum-safe#12 0x7ff34df13b6f in inner_evp_generic_fetch /home/pawn/work/openssl/crypto/evp/evp_fetch.c:312:23 open-quantum-safe#13 0x7ff34df13226 in evp_generic_fetch /home/pawn/work/openssl/crypto/evp/evp_fetch.c:364:14 open-quantum-safe#14 0x7ff34df36a42 in EVP_KEYMGMT_fetch /home/pawn/work/openssl/crypto/evp/keymgmt_meth.c:220:12 open-quantum-safe#15 0x7ff34df5d6e4 in int_ctx_new /home/pawn/work/openssl/crypto/evp/pmeth_lib.c:280:23 open-quantum-safe#16 0x7ff34df5f0a9 in EVP_PKEY_CTX_new_id /home/pawn/work/openssl/crypto/evp/pmeth_lib.c:469:12 open-quantum-safe#17 0x7ff349865bcb in oqsx_hybsig_init /home/pawn/work/oqs-provider/oqsprov/oqsprov_keys.c:578:20 open-quantum-safe#18 0x7ff349865157 in oqsx_key_new /home/pawn/work/oqs-provider/oqsprov/oqsprov_keys.c:791:16 open-quantum-safe#19 0x7ff3498715a3 in oqsx_genkey /home/pawn/work/oqs-provider/oqsprov/oqs_kmgmt.c:487:16 open-quantum-safe#20 0x7ff34986da08 in oqsx_gen /home/pawn/work/oqs-provider/oqsprov/oqs_kmgmt.c:509:12 open-quantum-safe#21 0x7ff34df373c7 in evp_keymgmt_gen /home/pawn/work/openssl/crypto/evp/keymgmt_meth.c:366:12 open-quantum-safe#22 0x7ff34df34b58 in evp_keymgmt_util_gen /home/pawn/work/openssl/crypto/evp/keymgmt_lib.c:517:20 open-quantum-safe#23 0x7ff34df5a775 in EVP_PKEY_generate /home/pawn/work/openssl/crypto/evp/pmeth_gn.c:189:13 open-quantum-safe#24 0x55b6fc90f2ad in test_oqs_signatures /home/pawn/work/oqs-provider/test/oqs_test_signatures.c:39:48 open-quantum-safe#25 0x55b6fc90ee2d in main /home/pawn/work/oqs-provider/test/oqs_test_signatures.c:114:17 open-quantum-safe#26 0x7ff34d8461c9 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 ``` 4. Missing `EVP_PKEY_CTX_free` in case of error. === In `oqsx_hybsig_init`, a `EVP_PKEY_CTX` is initialized using `EVP_PKEY_CTX_new_id`. However, later in the code, some extra work is done, and in case of error, the `EVP_PKEY_CTX` wasn't freed. (I couldn't recover the ASan message for this one…) 5. `OPENSSL_free` instead of `EVP_PKEY_free`. === In `oqsx_key_gen`, when a classical private key is generated, the field `classical_pkey` of a `OQSX_KEY` is set with a value of type `EVP_PKEY`. However, in `oqsx_key_free`, `OPENSSL_free` was used on `classical_pkey` instead of `EVP_PKEY_free`. ASan trace that helped me finding this one: ``` #0 0x555a6b1b728e in malloc (/home/pawn/work/oqs-provider/build/test/oqs_test_signatures+0xb928e) (BuildId: 6cc7ebd4bc73a0c78b2ab8d7b9dcb8b165d88e74) open-quantum-safe#1 0x7f59a93a946b in CRYPTO_malloc /home/pawn/work/openssl/crypto/mem.c:190:12 open-quantum-safe#2 0x7f59a93a94a2 in CRYPTO_zalloc /home/pawn/work/openssl/crypto/mem.c:197:11 open-quantum-safe#3 0x7f59a909cdcd in BN_new /home/pawn/work/openssl/crypto/bn/bn_lib.c:247:16 open-quantum-safe#4 0x7f59a920b955 in ossl_ec_group_new_ex /home/pawn/work/openssl/crypto/ec/ec_lib.c:61:25 open-quantum-safe#5 0x7f59a920dd55 in EC_GROUP_dup /home/pawn/work/openssl/crypto/ec/ec_lib.c:273:14 open-quantum-safe#6 0x7f59a9207774 in EC_KEY_set_group /home/pawn/work/openssl/crypto/ec/ec_key.c:733:18 open-quantum-safe#7 0x7f59a975b3b5 in ec_gen_assign_group /home/pawn/work/openssl/providers/implementations/keymgmt/ec_kmgmt.c:1236:12 open-quantum-safe#8 0x7f59a9758bdb in ec_gen /home/pawn/work/openssl/providers/implementations/keymgmt/ec_kmgmt.c:1274:11 open-quantum-safe#9 0x7f59a93373c7 in evp_keymgmt_gen /home/pawn/work/openssl/crypto/evp/keymgmt_meth.c:366:12 open-quantum-safe#10 0x7f59a9334b58 in evp_keymgmt_util_gen /home/pawn/work/openssl/crypto/evp/keymgmt_lib.c:517:20 open-quantum-safe#11 0x7f59a935a775 in EVP_PKEY_generate /home/pawn/work/openssl/crypto/evp/pmeth_gn.c:189:13 open-quantum-safe#12 0x7f59a935b404 in EVP_PKEY_keygen /home/pawn/work/openssl/crypto/evp/pmeth_gn.c:274:12 open-quantum-safe#13 0x7f59a4c69a51 in oqsx_key_gen_evp_key /home/pawn/work/oqs-provider/oqsprov/oqsprov_keys.c:1012:12 open-quantum-safe#14 0x7f59a4c68faf in oqsx_key_gen /home/pawn/work/oqs-provider/oqsprov/oqsprov_keys.c:1084:16 open-quantum-safe#15 0x7f59a4c716ca in oqsx_genkey /home/pawn/work/oqs-provider/oqsprov/oqs_kmgmt.c:496:9 open-quantum-safe#16 0x7f59a4c6da78 in oqsx_gen /home/pawn/work/oqs-provider/oqsprov/oqs_kmgmt.c:509:12 open-quantum-safe#17 0x7f59a93373c7 in evp_keymgmt_gen /home/pawn/work/openssl/crypto/evp/keymgmt_meth.c:366:12 open-quantum-safe#18 0x7f59a9334b58 in evp_keymgmt_util_gen /home/pawn/work/openssl/crypto/evp/keymgmt_lib.c:517:20 open-quantum-safe#19 0x7f59a935a775 in EVP_PKEY_generate /home/pawn/work/openssl/crypto/evp/pmeth_gn.c:189:13 open-quantum-safe#20 0x555a6b1f2890 in test_oqs_signatures /home/pawn/work/oqs-provider/test/oqs_test_signatures.c:68:44 open-quantum-safe#21 0x555a6b1f1e3d in main /home/pawn/work/oqs-provider/test/oqs_test_signatures.c:114:17 open-quantum-safe#22 0x7f59a8c461c9 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 ``` 6. Bonus point: fix a potential double-free. === In `oqsx_key_gen`, when the keytype isn't `KEY_TYPE_HYB_SIG`, the generated `pkey` is freed, and the flow goes down to the `err` label: https://github.com/open-quantum-safe/oqs-provider/blob/cc3895f208730abfe1efc15f7c3ff4358ac6e4c0/oqsprov/oqsprov_keys.c#L1089-L1104 There is a potential double-free bug here: if `oqsx_key_gen_oqs` fails, then `EVP_PKEY_free` will be called twice on the same `pkey` pointer. Note that it is very unlikely that `oqsx_key_gen_oqs` fails, since it only calls `OQS_KEM_keypair`/`OQS_SIG_keypair`.

This commit fixes various memory leaks that have been found using ASan. Note: I will soon add a CI job for testing oqs-provider against memory leaks with ASan. 1. Memory leak in `oqs_test_signatures`. === In `oqs_test_signatures`, [`OPENSSL_free`] was used with `EVP_PKEY_CTX`. However, the right destructor function for `EVP_PKEY_CTX` is [`EVP_PKEY_CTX_free`]. This commmit fixes this minor bug. [`OPENSSL_free`](https://www.openssl.org/docs/man3.0/man3/OPENSSL_free.html) [`EVP_PKEY_CTX_free`](https://www.openssl.org/docs/man3.0/man3/EVP_PKEY_CTX_free.html) 2. a selftest key that wasn't freed. === In `oqsprov/oqsprov_keys.c`, `d2i_PrivateKey` is used to verify that we the key we've just generated is sound. However, the returned object wasn't freed. ASan trace that helped me finding this one: ``` Indirect leak of 96 byte(s) in 4 object(s) allocated from: #0 0x5651eef1428e in malloc (/home/pawn/work/oqs-provider/build/test/oqs_test_signatures+0xb928e) (BuildId: 6cc7ebd4bc73a0c78b2ab8d7b9dcb8b165d88e74) open-quantum-safe#1 0x7fb7a23a946b in CRYPTO_malloc /home/pawn/work/openssl/crypto/mem.c:190:12 open-quantum-safe#2 0x7fb7a23a94a2 in CRYPTO_zalloc /home/pawn/work/openssl/crypto/mem.c:197:11 open-quantum-safe#3 0x7fb7a209cdcd in BN_new /home/pawn/work/openssl/crypto/bn/bn_lib.c:247:16 open-quantum-safe#4 0x7fb7a2224a00 in ossl_ec_GFp_mont_group_set_curve /home/pawn/work/openssl/crypto/ec/ecp_mont.c:169:11 open-quantum-safe#5 0x7fb7a220f5fa in EC_GROUP_set_curve /home/pawn/work/openssl/crypto/ec/ec_lib.c:562:12 open-quantum-safe#6 0x7fb7a2202d75 in EC_GROUP_new_curve_GFp /home/pawn/work/openssl/crypto/ec/ec_cvt.c:61:10 open-quantum-safe#7 0x7fb7a220184d in ec_group_new_from_data /home/pawn/work/openssl/crypto/ec/ec_curve.c:3179:22 open-quantum-safe#8 0x7fb7a2200ff9 in EC_GROUP_new_by_curve_name_ex /home/pawn/work/openssl/crypto/ec/ec_curve.c:3287:19 open-quantum-safe#9 0x7fb7a2201ed9 in EC_GROUP_new_by_curve_name /home/pawn/work/openssl/crypto/ec/ec_curve.c:3303:12 open-quantum-safe#10 0x7fb7a21f84e2 in EC_GROUP_new_from_ecpkparameters /home/pawn/work/openssl/crypto/ec/ec_asn1.c:863:20 open-quantum-safe#11 0x7fb7a21f8f28 in d2i_ECPrivateKey /home/pawn/work/openssl/crypto/ec/ec_asn1.c:956:22 open-quantum-safe#12 0x7fb7a270b4b5 in der2key_decode /home/pawn/work/openssl/providers/implementations/encode_decode/decode_der2key.c:220:19 open-quantum-safe#13 0x7fb7a226eae8 in decoder_process /home/pawn/work/openssl/crypto/encode_decode/decoder_lib.c:971:14 open-quantum-safe#14 0x7fb7a226d106 in OSSL_DECODER_from_bio /home/pawn/work/openssl/crypto/encode_decode/decoder_lib.c:82:10 open-quantum-safe#15 0x7fb7a226f007 in OSSL_DECODER_from_data /home/pawn/work/openssl/crypto/encode_decode/decoder_lib.c:157:9 open-quantum-safe#16 0x7fb7a200e123 in d2i_PrivateKey_decoder /home/pawn/work/openssl/crypto/asn1/d2i_pr.c:74:11 open-quantum-safe#17 0x7fb7a200dbba in d2i_PrivateKey_ex /home/pawn/work/openssl/crypto/asn1/d2i_pr.c:162:11 open-quantum-safe#18 0x7fb7a200e3d2 in d2i_PrivateKey /home/pawn/work/openssl/crypto/asn1/d2i_pr.c:172:12 open-quantum-safe#19 0x7fb79dc6a22e in oqsx_key_gen_evp_key /home/pawn/work/oqs-provider/oqsprov/oqsprov_keys.c:1036:25 open-quantum-safe#20 0x7fb79dc68f3f in oqsx_key_gen /home/pawn/work/oqs-provider/oqsprov/oqsprov_keys.c:1077:16 open-quantum-safe#21 0x7fb79dc7164a in oqsx_genkey /home/pawn/work/oqs-provider/oqsprov/oqs_kmgmt.c:496:9 open-quantum-safe#22 0x7fb79dc6d9f8 in oqsx_gen /home/pawn/work/oqs-provider/oqsprov/oqs_kmgmt.c:509:12 open-quantum-safe#23 0x7fb7a23373c7 in evp_keymgmt_gen /home/pawn/work/openssl/crypto/evp/keymgmt_meth.c:366:12 open-quantum-safe#24 0x7fb7a2334b58 in evp_keymgmt_util_gen /home/pawn/work/openssl/crypto/evp/keymgmt_lib.c:517:20 open-quantum-safe#25 0x7fb7a235a775 in EVP_PKEY_generate /home/pawn/work/openssl/crypto/evp/pmeth_gn.c:189:13 open-quantum-safe#26 0x5651eef4f2bd in test_oqs_signatures /home/pawn/work/oqs-provider/test/oqs_test_signatures.c:39:48 open-quantum-safe#27 0x5651eef4ee3d in main /home/pawn/work/oqs-provider/test/oqs_test_signatures.c:114:17 open-quantum-safe#28 0x7fb7a1b671c9 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 ``` 3. a `EVP_PKEY_CTX` that wasn't freed. === In `oqsx_key_free`, `key->oqsx_provider_ctx.oqsx_evp_ctx->ctx` is freed if the keytype is `ECP_HYB_KEM` or `ECX_HYB_KEM`. However, `key->oqsx_provider_ctx.oqsx_evp_ctx->ctx` is also used for signatures. In that case, a call to `EVP_PKEY_CTX_free` was missing. ASan trace that helped me finding this one: ``` Indirect leak of 7 byte(s) in 2 object(s) allocated from: #0 0x55b6fc8d427e in malloc (/home/pawn/work/oqs-provider/build/test/oqs_test_signatures+0xb927e) (BuildId: 3a8bff15d3315ba7ac1746fdc7b67eb464c607a1) open-quantum-safe#1 0x7ff34dfa946b in CRYPTO_malloc /home/pawn/work/openssl/crypto/mem.c:190:12 open-quantum-safe#2 0x7ff34dfac50a in CRYPTO_strndup /home/pawn/work/openssl/crypto/o_str.c:43:11 open-quantum-safe#3 0x7ff34df9d152 in ossl_algorithm_get1_first_name /home/pawn/work/openssl/crypto/core_algorithm.c:195:11 open-quantum-safe#4 0x7ff34df353a1 in keymgmt_from_algorithm /home/pawn/work/openssl/crypto/evp/keymgmt_meth.c:50:31 open-quantum-safe#5 0x7ff34df15e44 in construct_evp_method /home/pawn/work/openssl/crypto/evp/evp_fetch.c:219:14 open-quantum-safe#6 0x7ff34df9e3ce in ossl_method_construct_this /home/pawn/work/openssl/crypto/core_fetch.c:109:19 open-quantum-safe#7 0x7ff34df9d740 in algorithm_do_map /home/pawn/work/openssl/crypto/core_algorithm.c:77:13 open-quantum-safe#8 0x7ff34df9ce88 in algorithm_do_this /home/pawn/work/openssl/crypto/core_algorithm.c:122:15 open-quantum-safe#9 0x7ff34dfcce3e in ossl_provider_doall_activated /home/pawn/work/openssl/crypto/provider_core.c:1423:14 open-quantum-safe#10 0x7ff34df9ca64 in ossl_algorithm_do_all /home/pawn/work/openssl/crypto/core_algorithm.c:162:9 open-quantum-safe#11 0x7ff34df9dd64 in ossl_method_construct /home/pawn/work/openssl/crypto/core_fetch.c:153:5 open-quantum-safe#12 0x7ff34df13b6f in inner_evp_generic_fetch /home/pawn/work/openssl/crypto/evp/evp_fetch.c:312:23 open-quantum-safe#13 0x7ff34df13226 in evp_generic_fetch /home/pawn/work/openssl/crypto/evp/evp_fetch.c:364:14 open-quantum-safe#14 0x7ff34df36a42 in EVP_KEYMGMT_fetch /home/pawn/work/openssl/crypto/evp/keymgmt_meth.c:220:12 open-quantum-safe#15 0x7ff34df5d6e4 in int_ctx_new /home/pawn/work/openssl/crypto/evp/pmeth_lib.c:280:23 open-quantum-safe#16 0x7ff34df5f0a9 in EVP_PKEY_CTX_new_id /home/pawn/work/openssl/crypto/evp/pmeth_lib.c:469:12 open-quantum-safe#17 0x7ff349865bcb in oqsx_hybsig_init /home/pawn/work/oqs-provider/oqsprov/oqsprov_keys.c:578:20 open-quantum-safe#18 0x7ff349865157 in oqsx_key_new /home/pawn/work/oqs-provider/oqsprov/oqsprov_keys.c:791:16 open-quantum-safe#19 0x7ff3498715a3 in oqsx_genkey /home/pawn/work/oqs-provider/oqsprov/oqs_kmgmt.c:487:16 open-quantum-safe#20 0x7ff34986da08 in oqsx_gen /home/pawn/work/oqs-provider/oqsprov/oqs_kmgmt.c:509:12 open-quantum-safe#21 0x7ff34df373c7 in evp_keymgmt_gen /home/pawn/work/openssl/crypto/evp/keymgmt_meth.c:366:12 open-quantum-safe#22 0x7ff34df34b58 in evp_keymgmt_util_gen /home/pawn/work/openssl/crypto/evp/keymgmt_lib.c:517:20 open-quantum-safe#23 0x7ff34df5a775 in EVP_PKEY_generate /home/pawn/work/openssl/crypto/evp/pmeth_gn.c:189:13 open-quantum-safe#24 0x55b6fc90f2ad in test_oqs_signatures /home/pawn/work/oqs-provider/test/oqs_test_signatures.c:39:48 open-quantum-safe#25 0x55b6fc90ee2d in main /home/pawn/work/oqs-provider/test/oqs_test_signatures.c:114:17 open-quantum-safe#26 0x7ff34d8461c9 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 ``` 4. Missing `EVP_PKEY_CTX_free` in case of error. === In `oqsx_hybsig_init`, a `EVP_PKEY_CTX` is initialized using `EVP_PKEY_CTX_new_id`. However, later in the code, some extra work is done, and in case of error, the `EVP_PKEY_CTX` wasn't freed. (I couldn't recover the ASan message for this one…) 5. `OPENSSL_free` instead of `EVP_PKEY_free`. === In `oqsx_key_gen`, when a classical private key is generated, the field `classical_pkey` of a `OQSX_KEY` is set with a value of type `EVP_PKEY`. However, in `oqsx_key_free`, `OPENSSL_free` was used on `classical_pkey` instead of `EVP_PKEY_free`. ASan trace that helped me finding this one: ``` #0 0x555a6b1b728e in malloc (/home/pawn/work/oqs-provider/build/test/oqs_test_signatures+0xb928e) (BuildId: 6cc7ebd4bc73a0c78b2ab8d7b9dcb8b165d88e74) open-quantum-safe#1 0x7f59a93a946b in CRYPTO_malloc /home/pawn/work/openssl/crypto/mem.c:190:12 open-quantum-safe#2 0x7f59a93a94a2 in CRYPTO_zalloc /home/pawn/work/openssl/crypto/mem.c:197:11 open-quantum-safe#3 0x7f59a909cdcd in BN_new /home/pawn/work/openssl/crypto/bn/bn_lib.c:247:16 open-quantum-safe#4 0x7f59a920b955 in ossl_ec_group_new_ex /home/pawn/work/openssl/crypto/ec/ec_lib.c:61:25 open-quantum-safe#5 0x7f59a920dd55 in EC_GROUP_dup /home/pawn/work/openssl/crypto/ec/ec_lib.c:273:14 open-quantum-safe#6 0x7f59a9207774 in EC_KEY_set_group /home/pawn/work/openssl/crypto/ec/ec_key.c:733:18 open-quantum-safe#7 0x7f59a975b3b5 in ec_gen_assign_group /home/pawn/work/openssl/providers/implementations/keymgmt/ec_kmgmt.c:1236:12 open-quantum-safe#8 0x7f59a9758bdb in ec_gen /home/pawn/work/openssl/providers/implementations/keymgmt/ec_kmgmt.c:1274:11 open-quantum-safe#9 0x7f59a93373c7 in evp_keymgmt_gen /home/pawn/work/openssl/crypto/evp/keymgmt_meth.c:366:12 open-quantum-safe#10 0x7f59a9334b58 in evp_keymgmt_util_gen /home/pawn/work/openssl/crypto/evp/keymgmt_lib.c:517:20 open-quantum-safe#11 0x7f59a935a775 in EVP_PKEY_generate /home/pawn/work/openssl/crypto/evp/pmeth_gn.c:189:13 open-quantum-safe#12 0x7f59a935b404 in EVP_PKEY_keygen /home/pawn/work/openssl/crypto/evp/pmeth_gn.c:274:12 open-quantum-safe#13 0x7f59a4c69a51 in oqsx_key_gen_evp_key /home/pawn/work/oqs-provider/oqsprov/oqsprov_keys.c:1012:12 open-quantum-safe#14 0x7f59a4c68faf in oqsx_key_gen /home/pawn/work/oqs-provider/oqsprov/oqsprov_keys.c:1084:16 open-quantum-safe#15 0x7f59a4c716ca in oqsx_genkey /home/pawn/work/oqs-provider/oqsprov/oqs_kmgmt.c:496:9 open-quantum-safe#16 0x7f59a4c6da78 in oqsx_gen /home/pawn/work/oqs-provider/oqsprov/oqs_kmgmt.c:509:12 open-quantum-safe#17 0x7f59a93373c7 in evp_keymgmt_gen /home/pawn/work/openssl/crypto/evp/keymgmt_meth.c:366:12 open-quantum-safe#18 0x7f59a9334b58 in evp_keymgmt_util_gen /home/pawn/work/openssl/crypto/evp/keymgmt_lib.c:517:20 open-quantum-safe#19 0x7f59a935a775 in EVP_PKEY_generate /home/pawn/work/openssl/crypto/evp/pmeth_gn.c:189:13 open-quantum-safe#20 0x555a6b1f2890 in test_oqs_signatures /home/pawn/work/oqs-provider/test/oqs_test_signatures.c:68:44 open-quantum-safe#21 0x555a6b1f1e3d in main /home/pawn/work/oqs-provider/test/oqs_test_signatures.c:114:17 open-quantum-safe#22 0x7f59a8c461c9 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 ``` 6. Bonus point: fix a potential double-free. === In `oqsx_key_gen`, when the keytype isn't `KEY_TYPE_HYB_SIG`, the generated `pkey` is freed, and the flow goes down to the `err` label: https://github.com/open-quantum-safe/oqs-provider/blob/cc3895f208730abfe1efc15f7c3ff4358ac6e4c0/oqsprov/oqsprov_keys.c#L1089-L1104 There is a potential double-free bug here: if `oqsx_key_gen_oqs` fails, then `EVP_PKEY_free` will be called twice on the same `pkey` pointer. Note that it is very unlikely that `oqsx_key_gen_oqs` fails, since it only calls `OQS_KEM_keypair`/`OQS_SIG_keypair`.

This commit fixes various memory leaks that have been found using ASan. Note: I will soon add a CI job for testing oqs-provider against memory leaks with ASan. # 1. Memory leak in `oqs_test_signatures`. In `oqs_test_signatures`, [`OPENSSL_free`] was used with `EVP_PKEY_CTX`. However, the right destructor function for `EVP_PKEY_CTX` is [`EVP_PKEY_CTX_free`]. This commmit fixes this minor bug. [`OPENSSL_free`](https://www.openssl.org/docs/man3.0/man3/OPENSSL_free.html) [`EVP_PKEY_CTX_free`](https://www.openssl.org/docs/man3.0/man3/EVP_PKEY_CTX_free.html) ## 2. a selftest key that wasn't freed. In `oqsprov/oqsprov_keys.c`, `d2i_PrivateKey` is used to verify that we the key we've just generated is sound. However, the returned object wasn't freed. ASan trace that helped me finding this one: ``` Indirect leak of 96 byte(s) in 4 object(s) allocated from: #0 0x5651eef1428e in malloc (/home/pawn/work/oqs-provider/build/test/oqs_test_signatures+0xb928e) (BuildId: 6cc7ebd4bc73a0c78b2ab8d7b9dcb8b165d88e74) open-quantum-safe#1 0x7fb7a23a946b in CRYPTO_malloc /home/pawn/work/openssl/crypto/mem.c:190:12 open-quantum-safe#2 0x7fb7a23a94a2 in CRYPTO_zalloc /home/pawn/work/openssl/crypto/mem.c:197:11 open-quantum-safe#3 0x7fb7a209cdcd in BN_new /home/pawn/work/openssl/crypto/bn/bn_lib.c:247:16 open-quantum-safe#4 0x7fb7a2224a00 in ossl_ec_GFp_mont_group_set_curve /home/pawn/work/openssl/crypto/ec/ecp_mont.c:169:11 open-quantum-safe#5 0x7fb7a220f5fa in EC_GROUP_set_curve /home/pawn/work/openssl/crypto/ec/ec_lib.c:562:12 open-quantum-safe#6 0x7fb7a2202d75 in EC_GROUP_new_curve_GFp /home/pawn/work/openssl/crypto/ec/ec_cvt.c:61:10 open-quantum-safe#7 0x7fb7a220184d in ec_group_new_from_data /home/pawn/work/openssl/crypto/ec/ec_curve.c:3179:22 open-quantum-safe#8 0x7fb7a2200ff9 in EC_GROUP_new_by_curve_name_ex /home/pawn/work/openssl/crypto/ec/ec_curve.c:3287:19 open-quantum-safe#9 0x7fb7a2201ed9 in EC_GROUP_new_by_curve_name /home/pawn/work/openssl/crypto/ec/ec_curve.c:3303:12 open-quantum-safe#10 0x7fb7a21f84e2 in EC_GROUP_new_from_ecpkparameters /home/pawn/work/openssl/crypto/ec/ec_asn1.c:863:20 open-quantum-safe#11 0x7fb7a21f8f28 in d2i_ECPrivateKey /home/pawn/work/openssl/crypto/ec/ec_asn1.c:956:22 open-quantum-safe#12 0x7fb7a270b4b5 in der2key_decode /home/pawn/work/openssl/providers/implementations/encode_decode/decode_der2key.c:220:19 open-quantum-safe#13 0x7fb7a226eae8 in decoder_process /home/pawn/work/openssl/crypto/encode_decode/decoder_lib.c:971:14 open-quantum-safe#14 0x7fb7a226d106 in OSSL_DECODER_from_bio /home/pawn/work/openssl/crypto/encode_decode/decoder_lib.c:82:10 open-quantum-safe#15 0x7fb7a226f007 in OSSL_DECODER_from_data /home/pawn/work/openssl/crypto/encode_decode/decoder_lib.c:157:9 open-quantum-safe#16 0x7fb7a200e123 in d2i_PrivateKey_decoder /home/pawn/work/openssl/crypto/asn1/d2i_pr.c:74:11 open-quantum-safe#17 0x7fb7a200dbba in d2i_PrivateKey_ex /home/pawn/work/openssl/crypto/asn1/d2i_pr.c:162:11 open-quantum-safe#18 0x7fb7a200e3d2 in d2i_PrivateKey /home/pawn/work/openssl/crypto/asn1/d2i_pr.c:172:12 open-quantum-safe#19 0x7fb79dc6a22e in oqsx_key_gen_evp_key /home/pawn/work/oqs-provider/oqsprov/oqsprov_keys.c:1036:25 open-quantum-safe#20 0x7fb79dc68f3f in oqsx_key_gen /home/pawn/work/oqs-provider/oqsprov/oqsprov_keys.c:1077:16 open-quantum-safe#21 0x7fb79dc7164a in oqsx_genkey /home/pawn/work/oqs-provider/oqsprov/oqs_kmgmt.c:496:9 open-quantum-safe#22 0x7fb79dc6d9f8 in oqsx_gen /home/pawn/work/oqs-provider/oqsprov/oqs_kmgmt.c:509:12 open-quantum-safe#23 0x7fb7a23373c7 in evp_keymgmt_gen /home/pawn/work/openssl/crypto/evp/keymgmt_meth.c:366:12 open-quantum-safe#24 0x7fb7a2334b58 in evp_keymgmt_util_gen /home/pawn/work/openssl/crypto/evp/keymgmt_lib.c:517:20 open-quantum-safe#25 0x7fb7a235a775 in EVP_PKEY_generate /home/pawn/work/openssl/crypto/evp/pmeth_gn.c:189:13 open-quantum-safe#26 0x5651eef4f2bd in test_oqs_signatures /home/pawn/work/oqs-provider/test/oqs_test_signatures.c:39:48 open-quantum-safe#27 0x5651eef4ee3d in main /home/pawn/work/oqs-provider/test/oqs_test_signatures.c:114:17 open-quantum-safe#28 0x7fb7a1b671c9 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 ``` ## 3. a `EVP_PKEY_CTX` that wasn't freed. In `oqsx_key_free`, `key->oqsx_provider_ctx.oqsx_evp_ctx->ctx` is freed if the keytype is `ECP_HYB_KEM` or `ECX_HYB_KEM`. However, `key->oqsx_provider_ctx.oqsx_evp_ctx->ctx` is also used for signatures. In that case, a call to `EVP_PKEY_CTX_free` was missing. ASan trace that helped me finding this one: ``` Indirect leak of 7 byte(s) in 2 object(s) allocated from: #0 0x55b6fc8d427e in malloc (/home/pawn/work/oqs-provider/build/test/oqs_test_signatures+0xb927e) (BuildId: 3a8bff15d3315ba7ac1746fdc7b67eb464c607a1) open-quantum-safe#1 0x7ff34dfa946b in CRYPTO_malloc /home/pawn/work/openssl/crypto/mem.c:190:12 open-quantum-safe#2 0x7ff34dfac50a in CRYPTO_strndup /home/pawn/work/openssl/crypto/o_str.c:43:11 open-quantum-safe#3 0x7ff34df9d152 in ossl_algorithm_get1_first_name /home/pawn/work/openssl/crypto/core_algorithm.c:195:11 open-quantum-safe#4 0x7ff34df353a1 in keymgmt_from_algorithm /home/pawn/work/openssl/crypto/evp/keymgmt_meth.c:50:31 open-quantum-safe#5 0x7ff34df15e44 in construct_evp_method /home/pawn/work/openssl/crypto/evp/evp_fetch.c:219:14 open-quantum-safe#6 0x7ff34df9e3ce in ossl_method_construct_this /home/pawn/work/openssl/crypto/core_fetch.c:109:19 open-quantum-safe#7 0x7ff34df9d740 in algorithm_do_map /home/pawn/work/openssl/crypto/core_algorithm.c:77:13 open-quantum-safe#8 0x7ff34df9ce88 in algorithm_do_this /home/pawn/work/openssl/crypto/core_algorithm.c:122:15 open-quantum-safe#9 0x7ff34dfcce3e in ossl_provider_doall_activated /home/pawn/work/openssl/crypto/provider_core.c:1423:14 open-quantum-safe#10 0x7ff34df9ca64 in ossl_algorithm_do_all /home/pawn/work/openssl/crypto/core_algorithm.c:162:9 open-quantum-safe#11 0x7ff34df9dd64 in ossl_method_construct /home/pawn/work/openssl/crypto/core_fetch.c:153:5 open-quantum-safe#12 0x7ff34df13b6f in inner_evp_generic_fetch /home/pawn/work/openssl/crypto/evp/evp_fetch.c:312:23 open-quantum-safe#13 0x7ff34df13226 in evp_generic_fetch /home/pawn/work/openssl/crypto/evp/evp_fetch.c:364:14 open-quantum-safe#14 0x7ff34df36a42 in EVP_KEYMGMT_fetch /home/pawn/work/openssl/crypto/evp/keymgmt_meth.c:220:12 open-quantum-safe#15 0x7ff34df5d6e4 in int_ctx_new /home/pawn/work/openssl/crypto/evp/pmeth_lib.c:280:23 open-quantum-safe#16 0x7ff34df5f0a9 in EVP_PKEY_CTX_new_id /home/pawn/work/openssl/crypto/evp/pmeth_lib.c:469:12 open-quantum-safe#17 0x7ff349865bcb in oqsx_hybsig_init /home/pawn/work/oqs-provider/oqsprov/oqsprov_keys.c:578:20 open-quantum-safe#18 0x7ff349865157 in oqsx_key_new /home/pawn/work/oqs-provider/oqsprov/oqsprov_keys.c:791:16 open-quantum-safe#19 0x7ff3498715a3 in oqsx_genkey /home/pawn/work/oqs-provider/oqsprov/oqs_kmgmt.c:487:16 open-quantum-safe#20 0x7ff34986da08 in oqsx_gen /home/pawn/work/oqs-provider/oqsprov/oqs_kmgmt.c:509:12 open-quantum-safe#21 0x7ff34df373c7 in evp_keymgmt_gen /home/pawn/work/openssl/crypto/evp/keymgmt_meth.c:366:12 open-quantum-safe#22 0x7ff34df34b58 in evp_keymgmt_util_gen /home/pawn/work/openssl/crypto/evp/keymgmt_lib.c:517:20 open-quantum-safe#23 0x7ff34df5a775 in EVP_PKEY_generate /home/pawn/work/openssl/crypto/evp/pmeth_gn.c:189:13 open-quantum-safe#24 0x55b6fc90f2ad in test_oqs_signatures /home/pawn/work/oqs-provider/test/oqs_test_signatures.c:39:48 open-quantum-safe#25 0x55b6fc90ee2d in main /home/pawn/work/oqs-provider/test/oqs_test_signatures.c:114:17 open-quantum-safe#26 0x7ff34d8461c9 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 ``` ## 4. Missing `EVP_PKEY_CTX_free` in case of error. In `oqsx_hybsig_init`, a `EVP_PKEY_CTX` is initialized using `EVP_PKEY_CTX_new_id`. However, later in the code, some extra work is done, and in case of error, the `EVP_PKEY_CTX` wasn't freed. (I couldn't recover the ASan message for this one…) ## 5. `OPENSSL_free` instead of `EVP_PKEY_free`. In `oqsx_key_gen`, when a classical private key is generated, the field `classical_pkey` of a `OQSX_KEY` is set with a value of type `EVP_PKEY`. However, in `oqsx_key_free`, `OPENSSL_free` was used on `classical_pkey` instead of `EVP_PKEY_free`. ASan trace that helped me finding this one: ``` #0 0x555a6b1b728e in malloc (/home/pawn/work/oqs-provider/build/test/oqs_test_signatures+0xb928e) (BuildId: 6cc7ebd4bc73a0c78b2ab8d7b9dcb8b165d88e74) open-quantum-safe#1 0x7f59a93a946b in CRYPTO_malloc /home/pawn/work/openssl/crypto/mem.c:190:12 open-quantum-safe#2 0x7f59a93a94a2 in CRYPTO_zalloc /home/pawn/work/openssl/crypto/mem.c:197:11 open-quantum-safe#3 0x7f59a909cdcd in BN_new /home/pawn/work/openssl/crypto/bn/bn_lib.c:247:16 open-quantum-safe#4 0x7f59a920b955 in ossl_ec_group_new_ex /home/pawn/work/openssl/crypto/ec/ec_lib.c:61:25 open-quantum-safe#5 0x7f59a920dd55 in EC_GROUP_dup /home/pawn/work/openssl/crypto/ec/ec_lib.c:273:14 open-quantum-safe#6 0x7f59a9207774 in EC_KEY_set_group /home/pawn/work/openssl/crypto/ec/ec_key.c:733:18 open-quantum-safe#7 0x7f59a975b3b5 in ec_gen_assign_group /home/pawn/work/openssl/providers/implementations/keymgmt/ec_kmgmt.c:1236:12 open-quantum-safe#8 0x7f59a9758bdb in ec_gen /home/pawn/work/openssl/providers/implementations/keymgmt/ec_kmgmt.c:1274:11 open-quantum-safe#9 0x7f59a93373c7 in evp_keymgmt_gen /home/pawn/work/openssl/crypto/evp/keymgmt_meth.c:366:12 open-quantum-safe#10 0x7f59a9334b58 in evp_keymgmt_util_gen /home/pawn/work/openssl/crypto/evp/keymgmt_lib.c:517:20 open-quantum-safe#11 0x7f59a935a775 in EVP_PKEY_generate /home/pawn/work/openssl/crypto/evp/pmeth_gn.c:189:13 open-quantum-safe#12 0x7f59a935b404 in EVP_PKEY_keygen /home/pawn/work/openssl/crypto/evp/pmeth_gn.c:274:12 open-quantum-safe#13 0x7f59a4c69a51 in oqsx_key_gen_evp_key /home/pawn/work/oqs-provider/oqsprov/oqsprov_keys.c:1012:12 open-quantum-safe#14 0x7f59a4c68faf in oqsx_key_gen /home/pawn/work/oqs-provider/oqsprov/oqsprov_keys.c:1084:16 open-quantum-safe#15 0x7f59a4c716ca in oqsx_genkey /home/pawn/work/oqs-provider/oqsprov/oqs_kmgmt.c:496:9 open-quantum-safe#16 0x7f59a4c6da78 in oqsx_gen /home/pawn/work/oqs-provider/oqsprov/oqs_kmgmt.c:509:12 open-quantum-safe#17 0x7f59a93373c7 in evp_keymgmt_gen /home/pawn/work/openssl/crypto/evp/keymgmt_meth.c:366:12 open-quantum-safe#18 0x7f59a9334b58 in evp_keymgmt_util_gen /home/pawn/work/openssl/crypto/evp/keymgmt_lib.c:517:20 open-quantum-safe#19 0x7f59a935a775 in EVP_PKEY_generate /home/pawn/work/openssl/crypto/evp/pmeth_gn.c:189:13 open-quantum-safe#20 0x555a6b1f2890 in test_oqs_signatures /home/pawn/work/oqs-provider/test/oqs_test_signatures.c:68:44 open-quantum-safe#21 0x555a6b1f1e3d in main /home/pawn/work/oqs-provider/test/oqs_test_signatures.c:114:17 open-quantum-safe#22 0x7f59a8c461c9 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 ``` ## 6. Some `OPENSSL_free` were missing in `oqs_test_kems.c` ``` ==569479==ERROR: LeakSanitizer: detected memory leaks Direct leak of 346035 byte(s) in 42 object(s) allocated from: #0 0x559c76ef524e in malloc (/home/thomas_bailleux/work/oqs-provider/build/test/oqs_test_kems+0xb924e) (BuildId: 76a98f5859ba3c90334cef4286e89aa76436b1ae) open-quantum-safe#1 0x7f04407a946b in CRYPTO_malloc /home/thomas_bailleux/work/openssl/crypto/mem.c:190:12 open-quantum-safe#2 0x559c76f30389 in test_oqs_kems /home/thomas_bailleux/work/oqs-provider/test/oqs_test_kems.c:44:26 open-quantum-safe#3 0x559c76f2fdfd in main /home/thomas_bailleux/work/oqs-provider/test/oqs_test_kems.c:93:17 open-quantum-safe#4 0x7f04400461c9 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 Direct leak of 2690 byte(s) in 42 object(s) allocated from: #0 0x559c76ef524e in malloc (/home/thomas_bailleux/work/oqs-provider/build/test/oqs_test_kems+0xb924e) (BuildId: 76a98f5859ba3c90334cef4286e89aa76436b1ae) open-quantum-safe#1 0x7f04407a946b in CRYPTO_malloc /home/thomas_bailleux/work/openssl/crypto/mem.c:190:12 open-quantum-safe#2 0x559c76f304a5 in test_oqs_kems /home/thomas_bailleux/work/oqs-provider/test/oqs_test_kems.c:47:29 open-quantum-safe#3 0x559c76f2fdfd in main /home/thomas_bailleux/work/oqs-provider/test/oqs_test_kems.c:93:17 open-quantum-safe#4 0x7f04400461c9 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 Direct leak of 2690 byte(s) in 42 object(s) allocated from: #0 0x559c76ef524e in malloc (/home/thomas_bailleux/work/oqs-provider/build/test/oqs_test_kems+0xb924e) (BuildId: 76a98f5859ba3c90334cef4286e89aa76436b1ae) open-quantum-safe#1 0x7f04407a946b in CRYPTO_malloc /home/thomas_bailleux/work/openssl/crypto/mem.c:190:12 open-quantum-safe#2 0x559c76f303e4 in test_oqs_kems /home/thomas_bailleux/work/oqs-provider/test/oqs_test_kems.c:45:29 open-quantum-safe#3 0x559c76f2fdfd in main /home/thomas_bailleux/work/oqs-provider/test/oqs_test_kems.c:93:17 open-quantum-safe#4 0x7f04400461c9 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 ``` ## 7. Bonus point: fix a potential double-free. In `oqsx_key_gen`, when the keytype isn't `KEY_TYPE_HYB_SIG`, the generated `pkey` is freed, and the flow goes down to the `err` label: https://github.com/open-quantum-safe/oqs-provider/blob/cc3895f208730abfe1efc15f7c3ff4358ac6e4c0/oqsprov/oqsprov_keys.c#L1089-L1104 There is a potential double-free bug here: if `oqsx_key_gen_oqs` fails, then `EVP_PKEY_free` will be called twice on the same `pkey` pointer. Note that it is very unlikely that `oqsx_key_gen_oqs` fails, since it only calls `OQS_KEM_keypair`/`OQS_SIG_keypair`.

This commit fixes various memory leaks that have been found using ASan. Note: I will soon add a CI job for testing oqs-provider against memory leaks with ASan. # 1. Memory leak in `oqs_test_signatures`. In `oqs_test_signatures`, [`OPENSSL_free`] was used with `EVP_PKEY_CTX`. However, the right destructor function for `EVP_PKEY_CTX` is [`EVP_PKEY_CTX_free`]. This commmit fixes this minor bug. [`OPENSSL_free`](https://www.openssl.org/docs/man3.0/man3/OPENSSL_free.html) [`EVP_PKEY_CTX_free`](https://www.openssl.org/docs/man3.0/man3/EVP_PKEY_CTX_free.html) ## 2. a selftest key that wasn't freed. In `oqsprov/oqsprov_keys.c`, `d2i_PrivateKey` is used to verify that we the key we've just generated is sound. However, the returned object wasn't freed. ASan trace that helped me finding this one: ``` Indirect leak of 96 byte(s) in 4 object(s) allocated from: #0 0x5651eef1428e in malloc (/home/pawn/work/oqs-provider/build/test/oqs_test_signatures+0xb928e) (BuildId: 6cc7ebd4bc73a0c78b2ab8d7b9dcb8b165d88e74) open-quantum-safe#1 0x7fb7a23a946b in CRYPTO_malloc /home/pawn/work/openssl/crypto/mem.c:190:12 open-quantum-safe#2 0x7fb7a23a94a2 in CRYPTO_zalloc /home/pawn/work/openssl/crypto/mem.c:197:11 open-quantum-safe#3 0x7fb7a209cdcd in BN_new /home/pawn/work/openssl/crypto/bn/bn_lib.c:247:16 open-quantum-safe#4 0x7fb7a2224a00 in ossl_ec_GFp_mont_group_set_curve /home/pawn/work/openssl/crypto/ec/ecp_mont.c:169:11 open-quantum-safe#5 0x7fb7a220f5fa in EC_GROUP_set_curve /home/pawn/work/openssl/crypto/ec/ec_lib.c:562:12 open-quantum-safe#6 0x7fb7a2202d75 in EC_GROUP_new_curve_GFp /home/pawn/work/openssl/crypto/ec/ec_cvt.c:61:10 open-quantum-safe#7 0x7fb7a220184d in ec_group_new_from_data /home/pawn/work/openssl/crypto/ec/ec_curve.c:3179:22 open-quantum-safe#8 0x7fb7a2200ff9 in EC_GROUP_new_by_curve_name_ex /home/pawn/work/openssl/crypto/ec/ec_curve.c:3287:19 open-quantum-safe#9 0x7fb7a2201ed9 in EC_GROUP_new_by_curve_name /home/pawn/work/openssl/crypto/ec/ec_curve.c:3303:12 open-quantum-safe#10 0x7fb7a21f84e2 in EC_GROUP_new_from_ecpkparameters /home/pawn/work/openssl/crypto/ec/ec_asn1.c:863:20 open-quantum-safe#11 0x7fb7a21f8f28 in d2i_ECPrivateKey /home/pawn/work/openssl/crypto/ec/ec_asn1.c:956:22 open-quantum-safe#12 0x7fb7a270b4b5 in der2key_decode /home/pawn/work/openssl/providers/implementations/encode_decode/decode_der2key.c:220:19 open-quantum-safe#13 0x7fb7a226eae8 in decoder_process /home/pawn/work/openssl/crypto/encode_decode/decoder_lib.c:971:14 open-quantum-safe#14 0x7fb7a226d106 in OSSL_DECODER_from_bio /home/pawn/work/openssl/crypto/encode_decode/decoder_lib.c:82:10 open-quantum-safe#15 0x7fb7a226f007 in OSSL_DECODER_from_data /home/pawn/work/openssl/crypto/encode_decode/decoder_lib.c:157:9 open-quantum-safe#16 0x7fb7a200e123 in d2i_PrivateKey_decoder /home/pawn/work/openssl/crypto/asn1/d2i_pr.c:74:11 open-quantum-safe#17 0x7fb7a200dbba in d2i_PrivateKey_ex /home/pawn/work/openssl/crypto/asn1/d2i_pr.c:162:11 open-quantum-safe#18 0x7fb7a200e3d2 in d2i_PrivateKey /home/pawn/work/openssl/crypto/asn1/d2i_pr.c:172:12 open-quantum-safe#19 0x7fb79dc6a22e in oqsx_key_gen_evp_key /home/pawn/work/oqs-provider/oqsprov/oqsprov_keys.c:1036:25 open-quantum-safe#20 0x7fb79dc68f3f in oqsx_key_gen /home/pawn/work/oqs-provider/oqsprov/oqsprov_keys.c:1077:16 open-quantum-safe#21 0x7fb79dc7164a in oqsx_genkey /home/pawn/work/oqs-provider/oqsprov/oqs_kmgmt.c:496:9 open-quantum-safe#22 0x7fb79dc6d9f8 in oqsx_gen /home/pawn/work/oqs-provider/oqsprov/oqs_kmgmt.c:509:12 open-quantum-safe#23 0x7fb7a23373c7 in evp_keymgmt_gen /home/pawn/work/openssl/crypto/evp/keymgmt_meth.c:366:12 open-quantum-safe#24 0x7fb7a2334b58 in evp_keymgmt_util_gen /home/pawn/work/openssl/crypto/evp/keymgmt_lib.c:517:20 open-quantum-safe#25 0x7fb7a235a775 in EVP_PKEY_generate /home/pawn/work/openssl/crypto/evp/pmeth_gn.c:189:13 open-quantum-safe#26 0x5651eef4f2bd in test_oqs_signatures /home/pawn/work/oqs-provider/test/oqs_test_signatures.c:39:48 open-quantum-safe#27 0x5651eef4ee3d in main /home/pawn/work/oqs-provider/test/oqs_test_signatures.c:114:17 open-quantum-safe#28 0x7fb7a1b671c9 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 ``` ## 3. a `EVP_PKEY_CTX` that wasn't freed. In `oqsx_key_free`, `key->oqsx_provider_ctx.oqsx_evp_ctx->ctx` is freed if the keytype is `ECP_HYB_KEM` or `ECX_HYB_KEM`. However, `key->oqsx_provider_ctx.oqsx_evp_ctx->ctx` is also used for signatures. In that case, a call to `EVP_PKEY_CTX_free` was missing. ASan trace that helped me finding this one: ``` Indirect leak of 7 byte(s) in 2 object(s) allocated from: #0 0x55b6fc8d427e in malloc (/home/pawn/work/oqs-provider/build/test/oqs_test_signatures+0xb927e) (BuildId: 3a8bff15d3315ba7ac1746fdc7b67eb464c607a1) open-quantum-safe#1 0x7ff34dfa946b in CRYPTO_malloc /home/pawn/work/openssl/crypto/mem.c:190:12 open-quantum-safe#2 0x7ff34dfac50a in CRYPTO_strndup /home/pawn/work/openssl/crypto/o_str.c:43:11 open-quantum-safe#3 0x7ff34df9d152 in ossl_algorithm_get1_first_name /home/pawn/work/openssl/crypto/core_algorithm.c:195:11 open-quantum-safe#4 0x7ff34df353a1 in keymgmt_from_algorithm /home/pawn/work/openssl/crypto/evp/keymgmt_meth.c:50:31 open-quantum-safe#5 0x7ff34df15e44 in construct_evp_method /home/pawn/work/openssl/crypto/evp/evp_fetch.c:219:14 open-quantum-safe#6 0x7ff34df9e3ce in ossl_method_construct_this /home/pawn/work/openssl/crypto/core_fetch.c:109:19 open-quantum-safe#7 0x7ff34df9d740 in algorithm_do_map /home/pawn/work/openssl/crypto/core_algorithm.c:77:13 open-quantum-safe#8 0x7ff34df9ce88 in algorithm_do_this /home/pawn/work/openssl/crypto/core_algorithm.c:122:15 open-quantum-safe#9 0x7ff34dfcce3e in ossl_provider_doall_activated /home/pawn/work/openssl/crypto/provider_core.c:1423:14 open-quantum-safe#10 0x7ff34df9ca64 in ossl_algorithm_do_all /home/pawn/work/openssl/crypto/core_algorithm.c:162:9 open-quantum-safe#11 0x7ff34df9dd64 in ossl_method_construct /home/pawn/work/openssl/crypto/core_fetch.c:153:5 open-quantum-safe#12 0x7ff34df13b6f in inner_evp_generic_fetch /home/pawn/work/openssl/crypto/evp/evp_fetch.c:312:23 open-quantum-safe#13 0x7ff34df13226 in evp_generic_fetch /home/pawn/work/openssl/crypto/evp/evp_fetch.c:364:14 open-quantum-safe#14 0x7ff34df36a42 in EVP_KEYMGMT_fetch /home/pawn/work/openssl/crypto/evp/keymgmt_meth.c:220:12 open-quantum-safe#15 0x7ff34df5d6e4 in int_ctx_new /home/pawn/work/openssl/crypto/evp/pmeth_lib.c:280:23 open-quantum-safe#16 0x7ff34df5f0a9 in EVP_PKEY_CTX_new_id /home/pawn/work/openssl/crypto/evp/pmeth_lib.c:469:12 open-quantum-safe#17 0x7ff349865bcb in oqsx_hybsig_init /home/pawn/work/oqs-provider/oqsprov/oqsprov_keys.c:578:20 open-quantum-safe#18 0x7ff349865157 in oqsx_key_new /home/pawn/work/oqs-provider/oqsprov/oqsprov_keys.c:791:16 open-quantum-safe#19 0x7ff3498715a3 in oqsx_genkey /home/pawn/work/oqs-provider/oqsprov/oqs_kmgmt.c:487:16 open-quantum-safe#20 0x7ff34986da08 in oqsx_gen /home/pawn/work/oqs-provider/oqsprov/oqs_kmgmt.c:509:12 open-quantum-safe#21 0x7ff34df373c7 in evp_keymgmt_gen /home/pawn/work/openssl/crypto/evp/keymgmt_meth.c:366:12 open-quantum-safe#22 0x7ff34df34b58 in evp_keymgmt_util_gen /home/pawn/work/openssl/crypto/evp/keymgmt_lib.c:517:20 open-quantum-safe#23 0x7ff34df5a775 in EVP_PKEY_generate /home/pawn/work/openssl/crypto/evp/pmeth_gn.c:189:13 open-quantum-safe#24 0x55b6fc90f2ad in test_oqs_signatures /home/pawn/work/oqs-provider/test/oqs_test_signatures.c:39:48 open-quantum-safe#25 0x55b6fc90ee2d in main /home/pawn/work/oqs-provider/test/oqs_test_signatures.c:114:17 open-quantum-safe#26 0x7ff34d8461c9 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 ``` ## 4. Missing `EVP_PKEY_CTX_free` in case of error. In `oqsx_hybsig_init`, a `EVP_PKEY_CTX` is initialized using `EVP_PKEY_CTX_new_id`. However, later in the code, some extra work is done, and in case of error, the `EVP_PKEY_CTX` wasn't freed. (I couldn't recover the ASan message for this one…) ## 5. `OPENSSL_free` instead of `EVP_PKEY_free`. In `oqsx_key_gen`, when a classical private key is generated, the field `classical_pkey` of a `OQSX_KEY` is set with a value of type `EVP_PKEY`. However, in `oqsx_key_free`, `OPENSSL_free` was used on `classical_pkey` instead of `EVP_PKEY_free`. ASan trace that helped me finding this one: ``` #0 0x555a6b1b728e in malloc (/home/pawn/work/oqs-provider/build/test/oqs_test_signatures+0xb928e) (BuildId: 6cc7ebd4bc73a0c78b2ab8d7b9dcb8b165d88e74) open-quantum-safe#1 0x7f59a93a946b in CRYPTO_malloc /home/pawn/work/openssl/crypto/mem.c:190:12 open-quantum-safe#2 0x7f59a93a94a2 in CRYPTO_zalloc /home/pawn/work/openssl/crypto/mem.c:197:11 open-quantum-safe#3 0x7f59a909cdcd in BN_new /home/pawn/work/openssl/crypto/bn/bn_lib.c:247:16 open-quantum-safe#4 0x7f59a920b955 in ossl_ec_group_new_ex /home/pawn/work/openssl/crypto/ec/ec_lib.c:61:25 open-quantum-safe#5 0x7f59a920dd55 in EC_GROUP_dup /home/pawn/work/openssl/crypto/ec/ec_lib.c:273:14 open-quantum-safe#6 0x7f59a9207774 in EC_KEY_set_group /home/pawn/work/openssl/crypto/ec/ec_key.c:733:18 open-quantum-safe#7 0x7f59a975b3b5 in ec_gen_assign_group /home/pawn/work/openssl/providers/implementations/keymgmt/ec_kmgmt.c:1236:12 open-quantum-safe#8 0x7f59a9758bdb in ec_gen /home/pawn/work/openssl/providers/implementations/keymgmt/ec_kmgmt.c:1274:11 open-quantum-safe#9 0x7f59a93373c7 in evp_keymgmt_gen /home/pawn/work/openssl/crypto/evp/keymgmt_meth.c:366:12 open-quantum-safe#10 0x7f59a9334b58 in evp_keymgmt_util_gen /home/pawn/work/openssl/crypto/evp/keymgmt_lib.c:517:20 open-quantum-safe#11 0x7f59a935a775 in EVP_PKEY_generate /home/pawn/work/openssl/crypto/evp/pmeth_gn.c:189:13 open-quantum-safe#12 0x7f59a935b404 in EVP_PKEY_keygen /home/pawn/work/openssl/crypto/evp/pmeth_gn.c:274:12 open-quantum-safe#13 0x7f59a4c69a51 in oqsx_key_gen_evp_key /home/pawn/work/oqs-provider/oqsprov/oqsprov_keys.c:1012:12 open-quantum-safe#14 0x7f59a4c68faf in oqsx_key_gen /home/pawn/work/oqs-provider/oqsprov/oqsprov_keys.c:1084:16 open-quantum-safe#15 0x7f59a4c716ca in oqsx_genkey /home/pawn/work/oqs-provider/oqsprov/oqs_kmgmt.c:496:9 open-quantum-safe#16 0x7f59a4c6da78 in oqsx_gen /home/pawn/work/oqs-provider/oqsprov/oqs_kmgmt.c:509:12 open-quantum-safe#17 0x7f59a93373c7 in evp_keymgmt_gen /home/pawn/work/openssl/crypto/evp/keymgmt_meth.c:366:12 open-quantum-safe#18 0x7f59a9334b58 in evp_keymgmt_util_gen /home/pawn/work/openssl/crypto/evp/keymgmt_lib.c:517:20 open-quantum-safe#19 0x7f59a935a775 in EVP_PKEY_generate /home/pawn/work/openssl/crypto/evp/pmeth_gn.c:189:13 open-quantum-safe#20 0x555a6b1f2890 in test_oqs_signatures /home/pawn/work/oqs-provider/test/oqs_test_signatures.c:68:44 open-quantum-safe#21 0x555a6b1f1e3d in main /home/pawn/work/oqs-provider/test/oqs_test_signatures.c:114:17 open-quantum-safe#22 0x7f59a8c461c9 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 ``` ## 6. Some `OPENSSL_free` were missing in `oqs_test_kems.c` ``` ==569479==ERROR: LeakSanitizer: detected memory leaks Direct leak of 346035 byte(s) in 42 object(s) allocated from: #0 0x559c76ef524e in malloc (/home/pawn/work/oqs-provider/build/test/oqs_test_kems+0xb924e) (BuildId: 76a98f5859ba3c90334cef4286e89aa76436b1ae) open-quantum-safe#1 0x7f04407a946b in CRYPTO_malloc /home/pawn/work/openssl/crypto/mem.c:190:12 open-quantum-safe#2 0x559c76f30389 in test_oqs_kems /home/pawn/work/oqs-provider/test/oqs_test_kems.c:44:26 open-quantum-safe#3 0x559c76f2fdfd in main /home/pawn/work/oqs-provider/test/oqs_test_kems.c:93:17 open-quantum-safe#4 0x7f04400461c9 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 Direct leak of 2690 byte(s) in 42 object(s) allocated from: #0 0x559c76ef524e in malloc (/home/pawn/work/oqs-provider/build/test/oqs_test_kems+0xb924e) (BuildId: 76a98f5859ba3c90334cef4286e89aa76436b1ae) open-quantum-safe#1 0x7f04407a946b in CRYPTO_malloc /home/pawn/work/openssl/crypto/mem.c:190:12 open-quantum-safe#2 0x559c76f304a5 in test_oqs_kems /home/pawn/work/oqs-provider/test/oqs_test_kems.c:47:29 open-quantum-safe#3 0x559c76f2fdfd in main /home/pawn/work/oqs-provider/test/oqs_test_kems.c:93:17 open-quantum-safe#4 0x7f04400461c9 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 Direct leak of 2690 byte(s) in 42 object(s) allocated from: #0 0x559c76ef524e in malloc (/home/pawn/work/oqs-provider/build/test/oqs_test_kems+0xb924e) (BuildId: 76a98f5859ba3c90334cef4286e89aa76436b1ae) open-quantum-safe#1 0x7f04407a946b in CRYPTO_malloc /home/pawn/work/openssl/crypto/mem.c:190:12 open-quantum-safe#2 0x559c76f303e4 in test_oqs_kems /home/pawn/work/oqs-provider/test/oqs_test_kems.c:45:29 open-quantum-safe#3 0x559c76f2fdfd in main /home/pawn/work/oqs-provider/test/oqs_test_kems.c:93:17 open-quantum-safe#4 0x7f04400461c9 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 ``` ## 7. Bonus point: fix a potential double-free. In `oqsx_key_gen`, when the keytype isn't `KEY_TYPE_HYB_SIG`, the generated `pkey` is freed, and the flow goes down to the `err` label: https://github.com/open-quantum-safe/oqs-provider/blob/cc3895f208730abfe1efc15f7c3ff4358ac6e4c0/oqsprov/oqsprov_keys.c#L1089-L1104 There is a potential double-free bug here: if `oqsx_key_gen_oqs` fails, then `EVP_PKEY_free` will be called twice on the same `pkey` pointer. Note that it is very unlikely that `oqsx_key_gen_oqs` fails, since it only calls `OQS_KEM_keypair`/`OQS_SIG_keypair`.

This commit fixes various memory leaks that have been found using ASan. Note: I will soon add a CI job for testing oqs-provider against memory leaks with ASan. # 1. Memory leak in `oqs_test_signatures`. In `oqs_test_signatures`, [`OPENSSL_free`] was used with `EVP_PKEY_CTX`. However, the right destructor function for `EVP_PKEY_CTX` is [`EVP_PKEY_CTX_free`]. This commmit fixes this minor bug. [`OPENSSL_free`](https://www.openssl.org/docs/man3.0/man3/OPENSSL_free.html) [`EVP_PKEY_CTX_free`](https://www.openssl.org/docs/man3.0/man3/EVP_PKEY_CTX_free.html) ## 2. a selftest key that wasn't freed. In `oqsprov/oqsprov_keys.c`, `d2i_PrivateKey` is used to verify that we the key we've just generated is sound. However, the returned object wasn't freed. ASan trace that helped me finding this one: ``` Indirect leak of 96 byte(s) in 4 object(s) allocated from: #0 0x5651eef1428e in malloc (/home/pawn/work/oqs-provider/build/test/oqs_test_signatures+0xb928e) (BuildId: 6cc7ebd4bc73a0c78b2ab8d7b9dcb8b165d88e74) open-quantum-safe#1 0x7fb7a23a946b in CRYPTO_malloc /home/pawn/work/openssl/crypto/mem.c:190:12 open-quantum-safe#2 0x7fb7a23a94a2 in CRYPTO_zalloc /home/pawn/work/openssl/crypto/mem.c:197:11 open-quantum-safe#3 0x7fb7a209cdcd in BN_new /home/pawn/work/openssl/crypto/bn/bn_lib.c:247:16 open-quantum-safe#4 0x7fb7a2224a00 in ossl_ec_GFp_mont_group_set_curve /home/pawn/work/openssl/crypto/ec/ecp_mont.c:169:11 open-quantum-safe#5 0x7fb7a220f5fa in EC_GROUP_set_curve /home/pawn/work/openssl/crypto/ec/ec_lib.c:562:12 open-quantum-safe#6 0x7fb7a2202d75 in EC_GROUP_new_curve_GFp /home/pawn/work/openssl/crypto/ec/ec_cvt.c:61:10 open-quantum-safe#7 0x7fb7a220184d in ec_group_new_from_data /home/pawn/work/openssl/crypto/ec/ec_curve.c:3179:22 open-quantum-safe#8 0x7fb7a2200ff9 in EC_GROUP_new_by_curve_name_ex /home/pawn/work/openssl/crypto/ec/ec_curve.c:3287:19 open-quantum-safe#9 0x7fb7a2201ed9 in EC_GROUP_new_by_curve_name /home/pawn/work/openssl/crypto/ec/ec_curve.c:3303:12 open-quantum-safe#10 0x7fb7a21f84e2 in EC_GROUP_new_from_ecpkparameters /home/pawn/work/openssl/crypto/ec/ec_asn1.c:863:20 open-quantum-safe#11 0x7fb7a21f8f28 in d2i_ECPrivateKey /home/pawn/work/openssl/crypto/ec/ec_asn1.c:956:22 open-quantum-safe#12 0x7fb7a270b4b5 in der2key_decode /home/pawn/work/openssl/providers/implementations/encode_decode/decode_der2key.c:220:19 open-quantum-safe#13 0x7fb7a226eae8 in decoder_process /home/pawn/work/openssl/crypto/encode_decode/decoder_lib.c:971:14 open-quantum-safe#14 0x7fb7a226d106 in OSSL_DECODER_from_bio /home/pawn/work/openssl/crypto/encode_decode/decoder_lib.c:82:10 open-quantum-safe#15 0x7fb7a226f007 in OSSL_DECODER_from_data /home/pawn/work/openssl/crypto/encode_decode/decoder_lib.c:157:9 open-quantum-safe#16 0x7fb7a200e123 in d2i_PrivateKey_decoder /home/pawn/work/openssl/crypto/asn1/d2i_pr.c:74:11 open-quantum-safe#17 0x7fb7a200dbba in d2i_PrivateKey_ex /home/pawn/work/openssl/crypto/asn1/d2i_pr.c:162:11 open-quantum-safe#18 0x7fb7a200e3d2 in d2i_PrivateKey /home/pawn/work/openssl/crypto/asn1/d2i_pr.c:172:12 open-quantum-safe#19 0x7fb79dc6a22e in oqsx_key_gen_evp_key /home/pawn/work/oqs-provider/oqsprov/oqsprov_keys.c:1036:25 open-quantum-safe#20 0x7fb79dc68f3f in oqsx_key_gen /home/pawn/work/oqs-provider/oqsprov/oqsprov_keys.c:1077:16 open-quantum-safe#21 0x7fb79dc7164a in oqsx_genkey /home/pawn/work/oqs-provider/oqsprov/oqs_kmgmt.c:496:9 open-quantum-safe#22 0x7fb79dc6d9f8 in oqsx_gen /home/pawn/work/oqs-provider/oqsprov/oqs_kmgmt.c:509:12 open-quantum-safe#23 0x7fb7a23373c7 in evp_keymgmt_gen /home/pawn/work/openssl/crypto/evp/keymgmt_meth.c:366:12 open-quantum-safe#24 0x7fb7a2334b58 in evp_keymgmt_util_gen /home/pawn/work/openssl/crypto/evp/keymgmt_lib.c:517:20 open-quantum-safe#25 0x7fb7a235a775 in EVP_PKEY_generate /home/pawn/work/openssl/crypto/evp/pmeth_gn.c:189:13 open-quantum-safe#26 0x5651eef4f2bd in test_oqs_signatures /home/pawn/work/oqs-provider/test/oqs_test_signatures.c:39:48 open-quantum-safe#27 0x5651eef4ee3d in main /home/pawn/work/oqs-provider/test/oqs_test_signatures.c:114:17 open-quantum-safe#28 0x7fb7a1b671c9 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 ``` ## 3. a `EVP_PKEY_CTX` that wasn't freed. In `oqsx_key_free`, `key->oqsx_provider_ctx.oqsx_evp_ctx->ctx` is freed if the keytype is `ECP_HYB_KEM` or `ECX_HYB_KEM`. However, `key->oqsx_provider_ctx.oqsx_evp_ctx->ctx` is also used for signatures. In that case, a call to `EVP_PKEY_CTX_free` was missing. ASan trace that helped me finding this one: ``` Indirect leak of 7 byte(s) in 2 object(s) allocated from: #0 0x55b6fc8d427e in malloc (/home/pawn/work/oqs-provider/build/test/oqs_test_signatures+0xb927e) (BuildId: 3a8bff15d3315ba7ac1746fdc7b67eb464c607a1) open-quantum-safe#1 0x7ff34dfa946b in CRYPTO_malloc /home/pawn/work/openssl/crypto/mem.c:190:12 open-quantum-safe#2 0x7ff34dfac50a in CRYPTO_strndup /home/pawn/work/openssl/crypto/o_str.c:43:11 open-quantum-safe#3 0x7ff34df9d152 in ossl_algorithm_get1_first_name /home/pawn/work/openssl/crypto/core_algorithm.c:195:11 open-quantum-safe#4 0x7ff34df353a1 in keymgmt_from_algorithm /home/pawn/work/openssl/crypto/evp/keymgmt_meth.c:50:31 open-quantum-safe#5 0x7ff34df15e44 in construct_evp_method /home/pawn/work/openssl/crypto/evp/evp_fetch.c:219:14 open-quantum-safe#6 0x7ff34df9e3ce in ossl_method_construct_this /home/pawn/work/openssl/crypto/core_fetch.c:109:19 open-quantum-safe#7 0x7ff34df9d740 in algorithm_do_map /home/pawn/work/openssl/crypto/core_algorithm.c:77:13 open-quantum-safe#8 0x7ff34df9ce88 in algorithm_do_this /home/pawn/work/openssl/crypto/core_algorithm.c:122:15 open-quantum-safe#9 0x7ff34dfcce3e in ossl_provider_doall_activated /home/pawn/work/openssl/crypto/provider_core.c:1423:14 open-quantum-safe#10 0x7ff34df9ca64 in ossl_algorithm_do_all /home/pawn/work/openssl/crypto/core_algorithm.c:162:9 open-quantum-safe#11 0x7ff34df9dd64 in ossl_method_construct /home/pawn/work/openssl/crypto/core_fetch.c:153:5 open-quantum-safe#12 0x7ff34df13b6f in inner_evp_generic_fetch /home/pawn/work/openssl/crypto/evp/evp_fetch.c:312:23 open-quantum-safe#13 0x7ff34df13226 in evp_generic_fetch /home/pawn/work/openssl/crypto/evp/evp_fetch.c:364:14 open-quantum-safe#14 0x7ff34df36a42 in EVP_KEYMGMT_fetch /home/pawn/work/openssl/crypto/evp/keymgmt_meth.c:220:12 open-quantum-safe#15 0x7ff34df5d6e4 in int_ctx_new /home/pawn/work/openssl/crypto/evp/pmeth_lib.c:280:23 open-quantum-safe#16 0x7ff34df5f0a9 in EVP_PKEY_CTX_new_id /home/pawn/work/openssl/crypto/evp/pmeth_lib.c:469:12 open-quantum-safe#17 0x7ff349865bcb in oqsx_hybsig_init /home/pawn/work/oqs-provider/oqsprov/oqsprov_keys.c:578:20 open-quantum-safe#18 0x7ff349865157 in oqsx_key_new /home/pawn/work/oqs-provider/oqsprov/oqsprov_keys.c:791:16 open-quantum-safe#19 0x7ff3498715a3 in oqsx_genkey /home/pawn/work/oqs-provider/oqsprov/oqs_kmgmt.c:487:16 open-quantum-safe#20 0x7ff34986da08 in oqsx_gen /home/pawn/work/oqs-provider/oqsprov/oqs_kmgmt.c:509:12 open-quantum-safe#21 0x7ff34df373c7 in evp_keymgmt_gen /home/pawn/work/openssl/crypto/evp/keymgmt_meth.c:366:12 open-quantum-safe#22 0x7ff34df34b58 in evp_keymgmt_util_gen /home/pawn/work/openssl/crypto/evp/keymgmt_lib.c:517:20 open-quantum-safe#23 0x7ff34df5a775 in EVP_PKEY_generate /home/pawn/work/openssl/crypto/evp/pmeth_gn.c:189:13 open-quantum-safe#24 0x55b6fc90f2ad in test_oqs_signatures /home/pawn/work/oqs-provider/test/oqs_test_signatures.c:39:48 open-quantum-safe#25 0x55b6fc90ee2d in main /home/pawn/work/oqs-provider/test/oqs_test_signatures.c:114:17 open-quantum-safe#26 0x7ff34d8461c9 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 ``` ## 4. Missing `EVP_PKEY_CTX_free` in case of error. In `oqsx_hybsig_init`, a `EVP_PKEY_CTX` is initialized using `EVP_PKEY_CTX_new_id`. However, later in the code, some extra work is done, and in case of error, the `EVP_PKEY_CTX` wasn't freed. (I couldn't recover the ASan message for this one…) ## 5. `OPENSSL_free` instead of `EVP_PKEY_free`. In `oqsx_key_gen`, when a classical private key is generated, the field `classical_pkey` of a `OQSX_KEY` is set with a value of type `EVP_PKEY`. However, in `oqsx_key_free`, `OPENSSL_free` was used on `classical_pkey` instead of `EVP_PKEY_free`. ASan trace that helped me finding this one: ``` #0 0x555a6b1b728e in malloc (/home/pawn/work/oqs-provider/build/test/oqs_test_signatures+0xb928e) (BuildId: 6cc7ebd4bc73a0c78b2ab8d7b9dcb8b165d88e74) open-quantum-safe#1 0x7f59a93a946b in CRYPTO_malloc /home/pawn/work/openssl/crypto/mem.c:190:12 open-quantum-safe#2 0x7f59a93a94a2 in CRYPTO_zalloc /home/pawn/work/openssl/crypto/mem.c:197:11 open-quantum-safe#3 0x7f59a909cdcd in BN_new /home/pawn/work/openssl/crypto/bn/bn_lib.c:247:16 open-quantum-safe#4 0x7f59a920b955 in ossl_ec_group_new_ex /home/pawn/work/openssl/crypto/ec/ec_lib.c:61:25 open-quantum-safe#5 0x7f59a920dd55 in EC_GROUP_dup /home/pawn/work/openssl/crypto/ec/ec_lib.c:273:14 open-quantum-safe#6 0x7f59a9207774 in EC_KEY_set_group /home/pawn/work/openssl/crypto/ec/ec_key.c:733:18 open-quantum-safe#7 0x7f59a975b3b5 in ec_gen_assign_group /home/pawn/work/openssl/providers/implementations/keymgmt/ec_kmgmt.c:1236:12 open-quantum-safe#8 0x7f59a9758bdb in ec_gen /home/pawn/work/openssl/providers/implementations/keymgmt/ec_kmgmt.c:1274:11 open-quantum-safe#9 0x7f59a93373c7 in evp_keymgmt_gen /home/pawn/work/openssl/crypto/evp/keymgmt_meth.c:366:12 open-quantum-safe#10 0x7f59a9334b58 in evp_keymgmt_util_gen /home/pawn/work/openssl/crypto/evp/keymgmt_lib.c:517:20 open-quantum-safe#11 0x7f59a935a775 in EVP_PKEY_generate /home/pawn/work/openssl/crypto/evp/pmeth_gn.c:189:13 open-quantum-safe#12 0x7f59a935b404 in EVP_PKEY_keygen /home/pawn/work/openssl/crypto/evp/pmeth_gn.c:274:12 open-quantum-safe#13 0x7f59a4c69a51 in oqsx_key_gen_evp_key /home/pawn/work/oqs-provider/oqsprov/oqsprov_keys.c:1012:12 open-quantum-safe#14 0x7f59a4c68faf in oqsx_key_gen /home/pawn/work/oqs-provider/oqsprov/oqsprov_keys.c:1084:16 open-quantum-safe#15 0x7f59a4c716ca in oqsx_genkey /home/pawn/work/oqs-provider/oqsprov/oqs_kmgmt.c:496:9 open-quantum-safe#16 0x7f59a4c6da78 in oqsx_gen /home/pawn/work/oqs-provider/oqsprov/oqs_kmgmt.c:509:12 open-quantum-safe#17 0x7f59a93373c7 in evp_keymgmt_gen /home/pawn/work/openssl/crypto/evp/keymgmt_meth.c:366:12 open-quantum-safe#18 0x7f59a9334b58 in evp_keymgmt_util_gen /home/pawn/work/openssl/crypto/evp/keymgmt_lib.c:517:20 open-quantum-safe#19 0x7f59a935a775 in EVP_PKEY_generate /home/pawn/work/openssl/crypto/evp/pmeth_gn.c:189:13 open-quantum-safe#20 0x555a6b1f2890 in test_oqs_signatures /home/pawn/work/oqs-provider/test/oqs_test_signatures.c:68:44 open-quantum-safe#21 0x555a6b1f1e3d in main /home/pawn/work/oqs-provider/test/oqs_test_signatures.c:114:17 open-quantum-safe#22 0x7f59a8c461c9 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 ``` ## 6. Some `OPENSSL_free` were missing in `oqs_test_kems.c` ``` ==569479==ERROR: LeakSanitizer: detected memory leaks Direct leak of 346035 byte(s) in 42 object(s) allocated from: #0 0x559c76ef524e in malloc (/home/pawn/work/oqs-provider/build/test/oqs_test_kems+0xb924e) (BuildId: 76a98f5859ba3c90334cef4286e89aa76436b1ae) open-quantum-safe#1 0x7f04407a946b in CRYPTO_malloc /home/pawn/work/openssl/crypto/mem.c:190:12 open-quantum-safe#2 0x559c76f30389 in test_oqs_kems /home/pawn/work/oqs-provider/test/oqs_test_kems.c:44:26 open-quantum-safe#3 0x559c76f2fdfd in main /home/pawn/work/oqs-provider/test/oqs_test_kems.c:93:17 open-quantum-safe#4 0x7f04400461c9 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 Direct leak of 2690 byte(s) in 42 object(s) allocated from: #0 0x559c76ef524e in malloc (/home/pawn/work/oqs-provider/build/test/oqs_test_kems+0xb924e) (BuildId: 76a98f5859ba3c90334cef4286e89aa76436b1ae) open-quantum-safe#1 0x7f04407a946b in CRYPTO_malloc /home/pawn/work/openssl/crypto/mem.c:190:12 open-quantum-safe#2 0x559c76f304a5 in test_oqs_kems /home/pawn/work/oqs-provider/test/oqs_test_kems.c:47:29 open-quantum-safe#3 0x559c76f2fdfd in main /home/pawn/work/oqs-provider/test/oqs_test_kems.c:93:17 open-quantum-safe#4 0x7f04400461c9 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 Direct leak of 2690 byte(s) in 42 object(s) allocated from: #0 0x559c76ef524e in malloc (/home/pawn/work/oqs-provider/build/test/oqs_test_kems+0xb924e) (BuildId: 76a98f5859ba3c90334cef4286e89aa76436b1ae) open-quantum-safe#1 0x7f04407a946b in CRYPTO_malloc /home/pawn/work/openssl/crypto/mem.c:190:12 open-quantum-safe#2 0x559c76f303e4 in test_oqs_kems /home/pawn/work/oqs-provider/test/oqs_test_kems.c:45:29 open-quantum-safe#3 0x559c76f2fdfd in main /home/pawn/work/oqs-provider/test/oqs_test_kems.c:93:17 open-quantum-safe#4 0x7f04400461c9 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 ``` ## 7. Missing `X509_PUBKEY_free` in `oqsx_d2i_PUBKEY`. This was probably the hardest one for me to track down. In `oqsx_d2i_PUBKEY`, a call is done to `oqsx_d2i_X509_PUBKEY_INTERNAL`. However, the returned value (of type `X509_PUBKEY`) wasn't freed. ASan: ``` Direct leak of 48 byte(s) in 1 object(s) allocated from: #0 0x55610d68132e in malloc (/home/pawn/work/oqs-provider/build/test/oqs_test_endecode+0xb932e) (BuildId: 53bfaa0a3b850edc7328f30be8b047ab6675ba3c) open-quantum-safe#1 0x7f68999a946b in CRYPTO_malloc /home/pawn/work/openssl/crypto/mem.c:190:12 open-quantum-safe#2 0x7f68999a94a2 in CRYPTO_zalloc /home/pawn/work/openssl/crypto/mem.c:197:11 open-quantum-safe#3 0x7f6895295897 (<unknown module>) open-quantum-safe#4 0x7f6895295ce4 (<unknown module>) open-quantum-safe#5 0x7f6895296a6c (<unknown module>) open-quantum-safe#6 0x7f689986eae8 in decoder_process /home/pawn/work/openssl/crypto/encode_decode/decoder_lib.c:971:14 open-quantum-safe#7 0x7f6899d13383 in spki2typespki_decode /home/pawn/work/openssl/providers/implementations/encode_decode/decode_spki2typespki.c:111:10 open-quantum-safe#8 0x7f689986eae8 in decoder_process /home/pawn/work/openssl/crypto/encode_decode/decoder_lib.c:971:14 open-quantum-safe#9 0x7f6899d114d3 in pem2der_decode /home/pawn/work/openssl/providers/implementations/encode_decode/decode_pem2der.c:204:14 open-quantum-safe#10 0x7f689986eae8 in decoder_process /home/pawn/work/openssl/crypto/encode_decode/decoder_lib.c:971:14 open-quantum-safe#11 0x7f689986d106 in OSSL_DECODER_from_bio /home/pawn/work/openssl/crypto/encode_decode/decoder_lib.c:82:10 open-quantum-safe#12 0x55610d6bd407 in decode_EVP_PKEY_prov /home/pawn/work/oqs-provider/test/oqs_test_endecode.c:151:10 open-quantum-safe#13 0x55610d6bc75b in test_oqs_encdec /home/pawn/work/oqs-provider/test/oqs_test_endecode.c:188:14 open-quantum-safe#14 0x55610d6bbf37 in main /home/pawn/work/oqs-provider/test/oqs_test_endecode.c:240:17 open-quantum-safe#15 0x7f68991671c9 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 Indirect leak of 1312 byte(s) in 1 object(s) allocated from: #0 0x55610d68132e in malloc (/home/pawn/work/oqs-provider/build/test/oqs_test_endecode+0xb932e) (BuildId: 53bfaa0a3b850edc7328f30be8b047ab6675ba3c) open-quantum-safe#1 0x7f68999a946b in CRYPTO_malloc /home/pawn/work/openssl/crypto/mem.c:190:12 open-quantum-safe#2 0x7f68995d6b99 in ossl_c2i_ASN1_BIT_STRING /home/pawn/work/openssl/crypto/asn1/a_bitstr.c:117:13 open-quantum-safe#3 0x7f689961f659 in asn1_ex_c2i /home/pawn/work/openssl/crypto/asn1/tasn_dec.c:885:14 open-quantum-safe#4 0x7f689961bbc9 in asn1_d2i_ex_primitive /home/pawn/work/openssl/crypto/asn1/tasn_dec.c:818:10 open-quantum-safe#5 0x7f6899618054 in asn1_item_embed_d2i /home/pawn/work/openssl/crypto/asn1/tasn_dec.c:217:16 open-quantum-safe#6 0x7f689961e02f in asn1_template_noexp_d2i /home/pawn/work/openssl/crypto/asn1/tasn_dec.c:682:15 open-quantum-safe#7 0x7f689961a7b4 in asn1_template_ex_d2i /home/pawn/work/openssl/crypto/asn1/tasn_dec.c:558:16 open-quantum-safe#8 0x7f689961986a in asn1_item_embed_d2i /home/pawn/work/openssl/crypto/asn1/tasn_dec.c:422:19 open-quantum-safe#9 0x7f68996175c8 in asn1_item_ex_d2i_intern /home/pawn/work/openssl/crypto/asn1/tasn_dec.c:118:10 open-quantum-safe#10 0x7f68996177da in ASN1_item_d2i_ex /home/pawn/work/openssl/crypto/asn1/tasn_dec.c:144:9 open-quantum-safe#11 0x7f6895295926 (<unknown module>) open-quantum-safe#12 0x7f6895295ce4 (<unknown module>) open-quantum-safe#13 0x7f6895296a6c (<unknown module>) open-quantum-safe#14 0x7f689986eae8 in decoder_process /home/pawn/work/openssl/crypto/encode_decode/decoder_lib.c:971:14 open-quantum-safe#15 0x7f6899d13383 in spki2typespki_decode /home/pawn/work/openssl/providers/implementations/encode_decode/decode_spki2typespki.c:111:10 open-quantum-safe#16 0x7f689986eae8 in decoder_process /home/pawn/work/openssl/crypto/encode_decode/decoder_lib.c:971:14 open-quantum-safe#17 0x7f6899d114d3 in pem2der_decode /home/pawn/work/openssl/providers/implementations/encode_decode/decode_pem2der.c:204:14 open-quantum-safe#18 0x7f689986eae8 in decoder_process /home/pawn/work/openssl/crypto/encode_decode/decoder_lib.c:971:14 open-quantum-safe#19 0x7f689986d106 in OSSL_DECODER_from_bio /home/pawn/work/openssl/crypto/encode_decode/decoder_lib.c:82:10 open-quantum-safe#20 0x55610d6bd407 in decode_EVP_PKEY_prov /home/pawn/work/oqs-provider/test/oqs_test_endecode.c:151:10 open-quantum-safe#21 0x55610d6bc75b in test_oqs_encdec /home/pawn/work/oqs-provider/test/oqs_test_endecode.c:188:14 open-quantum-safe#22 0x55610d6bbf37 in main /home/pawn/work/oqs-provider/test/oqs_test_endecode.c:240:17 open-quantum-safe#23 0x7f68991671c9 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 Indirect leak of 24 byte(s) in 1 object(s) allocated from: #0 0x55610d68132e in malloc (/home/pawn/work/oqs-provider/build/test/oqs_test_endecode+0xb932e) (BuildId: 53bfaa0a3b850edc7328f30be8b047ab6675ba3c) open-quantum-safe#1 0x7f68999a946b in CRYPTO_malloc /home/pawn/work/openssl/crypto/mem.c:190:12 open-quantum-safe#2 0x7f68999a94a2 in CRYPTO_zalloc /home/pawn/work/openssl/crypto/mem.c:197:11 open-quantum-safe#3 0x7f68995fde90 in ASN1_STRING_type_new /home/pawn/work/openssl/crypto/asn1/asn1_lib.c:350:11 open-quantum-safe#4 0x7f689962a7dd in ASN1_BIT_STRING_new /home/pawn/work/openssl/crypto/asn1/tasn_typ.c:31:1 open-quantum-safe#5 0x7f68995d6a0f in ossl_c2i_ASN1_BIT_STRING /home/pawn/work/openssl/crypto/asn1/a_bitstr.c:98:20 open-quantum-safe#6 0x7f689961f659 in asn1_ex_c2i /home/pawn/work/openssl/crypto/asn1/tasn_dec.c:885:14 open-quantum-safe#7 0x7f689961bbc9 in asn1_d2i_ex_primitive /home/pawn/work/openssl/crypto/asn1/tasn_dec.c:818:10 open-quantum-safe#8 0x7f6899618054 in asn1_item_embed_d2i /home/pawn/work/openssl/crypto/asn1/tasn_dec.c:217:16 open-quantum-safe#9 0x7f689961e02f in asn1_template_noexp_d2i /home/pawn/work/openssl/crypto/asn1/tasn_dec.c:682:15 open-quantum-safe#10 0x7f689961a7b4 in asn1_template_ex_d2i /home/pawn/work/openssl/crypto/asn1/tasn_dec.c:558:16 open-quantum-safe#11 0x7f689961986a in asn1_item_embed_d2i /home/pawn/work/openssl/crypto/asn1/tasn_dec.c:422:19 open-quantum-safe#12 0x7f68996175c8 in asn1_item_ex_d2i_intern /home/pawn/work/openssl/crypto/asn1/tasn_dec.c:118:10 open-quantum-safe#13 0x7f68996177da in ASN1_item_d2i_ex /home/pawn/work/openssl/crypto/asn1/tasn_dec.c:144:9 open-quantum-safe#14 0x7f6895295926 (<unknown module>) open-quantum-safe#15 0x7f6895295ce4 (<unknown module>) open-quantum-safe#16 0x7f6895296a6c (<unknown module>) open-quantum-safe#17 0x7f689986eae8 in decoder_process /home/pawn/work/openssl/crypto/encode_decode/decoder_lib.c:971:14 open-quantum-safe#18 0x7f6899d13383 in spki2typespki_decode /home/pawn/work/openssl/providers/implementations/encode_decode/decode_spki2typespki.c:111:10 open-quantum-safe#19 0x7f689986eae8 in decoder_process /home/pawn/work/openssl/crypto/encode_decode/decoder_lib.c:971:14 open-quantum-safe#20 0x7f6899d114d3 in pem2der_decode /home/pawn/work/openssl/providers/implementations/encode_decode/decode_pem2der.c:204:14 open-quantum-safe#21 0x7f689986eae8 in decoder_process /home/pawn/work/openssl/crypto/encode_decode/decoder_lib.c:971:14 open-quantum-safe#22 0x7f689986d106 in OSSL_DECODER_from_bio /home/pawn/work/openssl/crypto/encode_decode/decoder_lib.c:82:10 open-quantum-safe#23 0x55610d6bd407 in decode_EVP_PKEY_prov /home/pawn/work/oqs-provider/test/oqs_test_endecode.c:151:10 open-quantum-safe#24 0x55610d6bc75b in test_oqs_encdec /home/pawn/work/oqs-provider/test/oqs_test_endecode.c:188:14 open-quantum-safe#25 0x55610d6bbf37 in main /home/pawn/work/oqs-provider/test/oqs_test_endecode.c:240:17 open-quantum-safe#26 0x7f68991671c9 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 Indirect leak of 16 byte(s) in 1 object(s) allocated from: #0 0x55610d68132e in malloc (/home/pawn/work/oqs-provider/build/test/oqs_test_endecode+0xb932e) (BuildId: 53bfaa0a3b850edc7328f30be8b047ab6675ba3c) open-quantum-safe#1 0x7f68999a946b in CRYPTO_malloc /home/pawn/work/openssl/crypto/mem.c:190:12 open-quantum-safe#2 0x7f68999a94a2 in CRYPTO_zalloc /home/pawn/work/openssl/crypto/mem.c:197:11 open-quantum-safe#3 0x7f6899625ed4 in asn1_item_embed_new /home/pawn/work/openssl/crypto/asn1/tasn_new.c:136:21 open-quantum-safe#4 0x7f689962620e in ossl_asn1_item_ex_new_intern /home/pawn/work/openssl/crypto/asn1/tasn_new.c:52:12 open-quantum-safe#5 0x7f68996191a5 in asn1_item_embed_d2i /home/pawn/work/openssl/crypto/asn1/tasn_dec.c:366:21 open-quantum-safe#6 0x7f689961e02f in asn1_template_noexp_d2i /home/pawn/work/openssl/crypto/asn1/tasn_dec.c:682:15 open-quantum-safe#7 0x7f689961a7b4 in asn1_template_ex_d2i /home/pawn/work/openssl/crypto/asn1/tasn_dec.c:558:16 open-quantum-safe#8 0x7f689961986a in asn1_item_embed_d2i /home/pawn/work/openssl/crypto/asn1/tasn_dec.c:422:19 open-quantum-safe#9 0x7f68996175c8 in asn1_item_ex_d2i_intern /home/pawn/work/openssl/crypto/asn1/tasn_dec.c:118:10 open-quantum-safe#10 0x7f68996177da in ASN1_item_d2i_ex /home/pawn/work/openssl/crypto/asn1/tasn_dec.c:144:9 open-quantum-safe#11 0x7f6895295926 (<unknown module>) open-quantum-safe#12 0x7f6895295ce4 (<unknown module>) open-quantum-safe#13 0x7f6895296a6c (<unknown module>) open-quantum-safe#14 0x7f689986eae8 in decoder_process /home/pawn/work/openssl/crypto/encode_decode/decoder_lib.c:971:14 open-quantum-safe#15 0x7f6899d13383 in spki2typespki_decode /home/pawn/work/openssl/providers/implementations/encode_decode/decode_spki2typespki.c:111:10 open-quantum-safe#16 0x7f689986eae8 in decoder_process /home/pawn/work/openssl/crypto/encode_decode/decoder_lib.c:971:14 open-quantum-safe#17 0x7f6899d114d3 in pem2der_decode /home/pawn/work/openssl/providers/implementations/encode_decode/decode_pem2der.c:204:14 open-quantum-safe#18 0x7f689986eae8 in decoder_process /home/pawn/work/openssl/crypto/encode_decode/decoder_lib.c:971:14 open-quantum-safe#19 0x7f689986d106 in OSSL_DECODER_from_bio /home/pawn/work/openssl/crypto/encode_decode/decoder_lib.c:82:10 open-quantum-safe#20 0x55610d6bd407 in decode_EVP_PKEY_prov /home/pawn/work/oqs-provider/test/oqs_test_endecode.c:151:10 open-quantum-safe#21 0x55610d6bc75b in test_oqs_encdec /home/pawn/work/oqs-provider/test/oqs_test_endecode.c:188:14 open-quantum-safe#22 0x55610d6bbf37 in main /home/pawn/work/oqs-provider/test/oqs_test_endecode.c:240:17 open-quantum-safe#23 0x7f68991671c9 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 SUMMARY: AddressSanitizer: 1400 byte(s) leaked in 4 allocation(s). ``` ## 8. Bonus point: fix a potential double-free. In `oqsx_key_gen`, when the keytype isn't `KEY_TYPE_HYB_SIG`, the generated `pkey` is freed, and the flow goes down to the `err` label: https://github.com/open-quantum-safe/oqs-provider/blob/cc3895f208730abfe1efc15f7c3ff4358ac6e4c0/oqsprov/oqsprov_keys.c#L1089-L1104 There is a potential double-free bug here: if `oqsx_key_gen_oqs` fails, then `EVP_PKEY_free` will be called twice on the same `pkey` pointer. Note that it is very unlikely that `oqsx_key_gen_oqs` fails, since it only calls `OQS_KEM_keypair`/`OQS_SIG_keypair`.

baentsch mentioned this issue Jun 4, 2021

Correct claimed NIST level for Kyber768_90s open-quantum-safe/liboqs#1012

Closed

baentsch mentioned this issue Jun 4, 2021

Validate claimed NIST levels open-quantum-safe/openssl#315

Closed

bhess mentioned this issue Jun 4, 2021

Update to ensure hybrid kex interop, updates for OpenSSL master #24

Merged

baentsch closed this as completed in #24 Jun 4, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

2 hybrid KEMs failing #23

2 hybrid KEMs failing #23

baentsch commented Jun 3, 2021

baentsch commented Jun 3, 2021

bhess commented Jun 3, 2021

baentsch commented Jun 3, 2021

bhess commented Jun 3, 2021

baentsch commented Jun 4, 2021 •

edited

Loading

baentsch commented Jun 4, 2021

bhess commented Jun 4, 2021 •

edited

Loading

baentsch commented Jun 4, 2021

2 hybrid KEMs failing #23

2 hybrid KEMs failing #23

Comments

baentsch commented Jun 3, 2021

baentsch commented Jun 3, 2021

bhess commented Jun 3, 2021

baentsch commented Jun 3, 2021

bhess commented Jun 3, 2021

baentsch commented Jun 4, 2021 • edited Loading

baentsch commented Jun 4, 2021

bhess commented Jun 4, 2021 • edited Loading

baentsch commented Jun 4, 2021

baentsch commented Jun 4, 2021 •

edited

Loading

bhess commented Jun 4, 2021 •

edited

Loading