Add PKCS#11 provider support for key generation

[olszomal]

Pull Request Type

• Bug fix
• New feature
• Code style / formatting / renaming
• Refactoring (no functional or API changes)
• Build / CI related changes
• Documentation
• Other (please describe):

Related Issue

Closed #677

Current Behavior

The PKCS#11 provider does not implement KEYMGMT key generation, so keys cannot be generated on a token using openssl genpkey.

New Behavior

The provider implements KEYMGMT key generation for the supported PKCS#11 key types.
The target token and generated key attributes are selected using a PKCS#11 URI.

Scope of Changes

• Add provider KEYMGMT generation callbacks.
• Add PKCS11_generate_key_ext() to return the generated private key.
• Add PKCS#11 provider support for openssl genpkey.
• Add SoftHSM tests for supported key types.

Testing

• Existing tests
• New tests added
• Manual testing

Additional Notes

License Declaration

• I hereby agree to license my contribution under the project's license.

[mtrojnar]

Reviewed at 683aeee against master (01c21ac). One blocking issue found: the advertised openssl genpkey command still exits unsuccessfully.

1. openssl genpkey reports failure after permanently creating the key

tests/provider-genpkey.softhsm:63-74 — every genpkey failure is suppressed with || true, and the test declares success merely when an object with the expected label exists. On OpenSSL 3.6.3, all exercised algorithms create their token objects but then fail while serializing the non-extractable private key.

openssl genpkey \
  -provider default -provider pkcs11prov \
  -propquery '?provider=pkcs11prov' \
  -algorithm RSA \
  -pkeyopt 'pkcs11_uri:pkcs11:token=libp11-0;object=repro;id=%01;pin-value=1234' \
  -pkeyopt rsa_keygen_bits:2048
echo $?

Decisive output:

Error writing key(s)
error:068000DE:asn1 encoding routines:asn1_template_ex_i2d:illegal zero content
error:1C88000D:Provider routines:key_to_p8info:ASN1 lib
1

Result: master exits 1 with “operation not supported” and creates no objects; this branch also exits 1, but leaves both public and private objects on the token. A caller or script will reasonably treat this as failure and may retry, creating duplicate persistent keys.

Suggestion: either provide an encoding/reference path that lets genpkey complete successfully, or narrow the advertised feature to provider EVP_PKEY_generate support and document the CLI limitation. The test should exercise EVP_PKEY_generate directly, require a successful return, and use the returned key for an operation instead of suppressing the command status.

Minor

• src/p11_key.c:557-564 — C_GenerateKeyPair returns the exact public and private handles, but generation retains only the private handle. EC and raw-key construction later searches for the public object solely by CKA_ID (src/p11_key.c:425-435, src/p11_rawkey.c:292-313). Because the URI permits an empty or duplicate ID and PKCS#11 does not define search ordering, the returned key can acquire public material from a different object. This is especially plausible after retrying the failing CLI command with the same URI. SoftHSM happened to return the newest matching object in my probe, so it did not reproduce there. Preserve the generated public handle when constructing the returned key, rather than performing an ambiguous lookup.

Behavioural changes worth mentioning in the PR description

openssl genpkey now creates persistent, sensitive, non-extractable token objects, but still reports failure because it cannot output the resulting private key.

What I ran

Environment	Result
GCC, OpenSSL 3.6.3, ./bootstrap && ./configure --enable-strict && make -j4	Passed on base and PR
OpenSSL 3.6.3 + SoftHSM2, make check	Base: 46 pass, 5 skip; PR: 47 pass, 5 skip
Direct RSA openssl genpkey comparison	Base: exit 1, 0 objects; PR: exit 1, 2 objects
git diff --check	Passed

Not covered: OpenSSL 3.0/3.5/4.x, Windows, hardware HSMs, and PQ mechanisms unavailable in SoftHSM.

[olszomal]

Ad 1.
openssl genpkey currently attempts to serialize the generated private key, which fails for non-exportable token keys even when key generation succeeds. The test therefore verifies that the key was created on the token instead of relying on the genpkey exit status.

OpenSSL PR 32483 adds a -noout option to address this issue. Until that change is accepted and merged, the test uses this workaround.

Ad 2. (Minor)
PKCS11_generate_key_ext() intentionally returns the generated private-key object. The public handle returned by C_GenerateKeyPair() is not needed here.

PKCS11_get_private_key() later constructs the EVP_PKEY from that private object and retrieves public material through the existing libp11 code when required.

The CKA_ID-based public-object lookup is part of the existing EVP_PKEY construction path and is unchanged by this patch.

[mtrojnar]

Thanks for the clarification.

Ad 1. Agreed. The provider’s KEYMGMT generation succeeds; the failure occurs afterward when openssl genpkey attempts to serialize the non-exportable private key. I was conflating the command’s final exit status with the provider feature itself. Given the pending OpenSSL -noout change, the current token-object check is a reasonable workaround. I withdraw the blocking finding.

Ad 2. Returning the private-key object is appropriate. My concern is specifically that the exact pub_key_obj returned by C_GenerateKeyPair() is discarded, after which public material is resolved by CKA_ID. Empty or duplicate IDs can therefore select a different public object.

I recommend preserving the pairing internally without changing the public API:

1. Add an optional associated-public-object reference to PKCS11_OBJECT_private.
2. After C_GenerateKeyPair(), initialize the private object and an internal public object directly from priv_key_obj and pub_key_obj.
3. Store the public-object reference on the generated private object.
4. Have pkcs11_object_from_object() prefer this exact association when a public object is requested, falling back to the existing CKA_ID lookup for ordinarily loaded keys.
5. Release the reference from pkcs11_object_free().

This would let the existing EC and raw-key construction paths use the exact generated public object while keeping PKCS11_generate_key_ext() returning the private key.

Since I could not reproduce a mismatch with SoftHSM, I consider this non-blocking. With the first finding withdrawn, I have no blocking objections to this PR.

[olszomal]

Implemented the suggested fix by preserving the exact public-key handle returned by C_GenerateKeyPair().

Generated private keys now use this handle when resolving the corresponding public object, with the existing CKA_ID lookup kept as a fallback. The stored handle is cleared when the object is reloaded after fork.

[mtrojnar]

Thanks. I reviewed 8966574d.

Preserving the public handle is safe even after the generation session is returned to the pool. PKCS#11 v2.40 §3.4 states that an object handle is assigned “for that application’s sessions to use”; the usage guide also demonstrates a handle obtained through one session being used through another. Since these generated public keys are token objects, retaining the originating session is unnecessary.

There is one remaining gap: pkcs11_object_from_object() uses public_object only when passed a valid session (src/p11_key.c:434-438). PKCS11_get_public_key() reaches it through pkcs11_get_key(), which passes CK_INVALID_HANDLE (src/p11_key.c:1324-1325). That path therefore still falls back to the potentially ambiguous CKA_ID lookup.

I suggest acquiring a temporary session when session == CK_INVALID_HANDLE, resolving public_object through that session, releasing it, and retaining the existing CKA_ID lookup as fallback. A regression test using duplicate or empty IDs and PKCS11_get_public_key() would cover this.

[olszomal]

Updated as suggested and added a regression test for duplicate CKA_ID values.