digital envelope routines:evp_pkey_ctx_ctrl_int

[than1]

After setting SoftHSM with OpenSSL 3.0.1, I tried to generate an RSA key pair by the following command:

openssl genrsa -engine pkcs11 2048

It give me the following error.

Engine "pkcs11" set.
Error setting RSA length
04520000:error:03000093:digital envelope routines:evp_pkey_ctx_ctrl_int:command not supported:crypto\evp\pmeth_lib.c:1321:

It seems the libp11 has some compatibility issue with OpenSSL 3.0.x. When I change the version to 1.x.x, the above error disappeared and the key pair generation is finished successfully.

Also, the above problem is confirmed when using a physical HSM.

Do anyone knows is it really a compatibility issue with libp11? if is , Do we have the plan to fix it?

[mtrojnar]

What exactly do you intend to achieve with openssl genrsa -engine pkcs11 2048? Is it to generate an RSA key pair on a PKCS#11 device and then retrieve the private key? PKCS#11 was specifically designed to allow for private key operations without retrieving the private key.

My understanding is that older versions of OpenSSL silently ignored the request to use an engine, while OpenSSL 3.0 attempts to invoke an engine control command instead. It may be useful for other engines, but not for PKCS#11.

[rodrigo-speller]

@than1 I have the same scenario as you.

But the error also ocurrs if I don't specify "-engine" argument: openssl genrsa 2048.

Do you found any solution for that?