Cannot decrypt RSA OAEP?

[mouse07410]

Current master:

$ openssl rsautl -engine pkcs11 -keyform ENGINE -encrypt -pubin -inkey "pkcs11:manufacturer=piv_II;object=KEY%20MAN%20pubkey;type=public" -oaep -in t256.dat -out t256.dat.enc
engine "pkcs11" set.
$ ls -l t256.dat.enc 
-rw-r--r--  1 mouse   256 Apr 10 17:34 t256.dat.enc
$ openssl rsautl -engine pkcs11 -keyform ENGINE -decrypt -inkey "pkcs11:manufacturer=piv_II;object=KEY%20MAN%20key;type=private" -oaep -in t256.dat.enc -out t256.dat.dec
engine "pkcs11" set.
PKCS#11 token PIN: 
PKCS#11: Unsupported padding type
RSA operation error
$ 

Not sure if the log would be of use:
libp11-decr.txt

[mtrojnar]

What is the output of openssl version -a?

Did it work before commit d216164, for example with commit 0206efb?

[mouse07410]

Did it work before commit d216164, for example with commit 0206efb?

It worked before... To be honest, I don't recall. Not sure about 0206efb.

What is the output of openssl version -a?

OpenSSL 1.0.2k  26 Jan 2017
built on: reproducible build, date unspecified
platform: darwin64-x86_64-cc
options:  bn(64,64) rc4(ptr,int) des(idx,cisc,16,int) idea(int) blowfish(idx) 
compiler: /usr/bin/clang -I. -I.. -I../include  -fPIC -fno-common -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -arch x86_64 -O3 -DL_ENDIAN -Wall -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM -DECP_NISTZ256_ASM
OPENSSLDIR: "/opt/local/etc/openssl"
$ 

This code is suspect (file src/p11_rsa.c around line 73), as it is the one responsible for the error message:

/* Setup PKCS#11 mechanisms for encryption/decryption */
static int pkcs11_mechanism(CK_MECHANISM *mechanism, const int padding)
{
        memset(mechanism, 0, sizeof(CK_MECHANISM));
        switch (padding) {
        case RSA_PKCS1_PADDING:
                mechanism->mechanism = CKM_RSA_PKCS;
                break;
        case RSA_NO_PADDING:
                mechanism->mechanism = CKM_RSA_X_509;
                break;
        case RSA_X931_PADDING:
                mechanism->mechanism = CKM_RSA_X9_31;
                break;
        default:
                fprintf(stderr, "PKCS#11: Unsupported padding type\n");
                return -1;
        }
        return 0;
}

[mtrojnar]

Yes, I also suspect that. This is why it is essential to check whether it worked before, and which commit broke it. You may use git bisect.

[mouse07410]

I just checked. Commit 0206efb does not work with this either.

[dengert]

With OpenSSL-1.0.2, opensc master and libp11 master:

running gdb --args openssl rsautl -engine pkcs11 -keyform ENGINE -decrypt -inkey "pkcs11:manufacturer=piv_II;object=KEY%20MAN%20key;type=private" -oaep -in t256.dat.enc -out t256.dat.dec

#0 pkcs11_mechanism (mechanism=0x7fffffffd4f0, padding=4) at ../../src/src/p11_rsa.c:73

#1 0x00007ffff6f35d55 in pkcs11_private_decrypt (flen=256, from=0x6f4380 "ϟ$5;\213\336\376Μ\314KԊw!U\237\355q\227\071\223\026\246WX$\250\004|\301§\257\335}#\201P\211(\351\206\001\276\304JN\340\233~\347x[\304#\217\030\245\025\247\225\267\300\033\001\231\242\034\372'\036\361K\333\021\372\320B\225\027\372M\"7\357\222y4\032\353\314T\326\064q=\272\023\274.\264u\fP\306t2\023\312k\336\352K9\206;\231\226\067D]\335\327-\317NnJ\215\253\022Z\213\003I\370\036\375\302\201\032\207Ry]4\364ء\346\020\036\267YW+*c\347\263O\220\025\357\302J\244J\003\217-\310\366:W\260Mk~\212P\017\311\316\361t\257]$xh\311~(8m8"..., to=0x6f4590 "xkp\367\377\177", key=0x6f3ef0, padding=4) at ../../src/src/p11_rsa.c:139`

#2 0x00007ffff6f399aa in PKCS11_private_decrypt (flen=256, from=0x6f4380 "ϟ$5;\213\336\376Μ\314KԊw!U\237\355q\227\071\223\026\246WX$\250\004|\301§\257\335}#\201P\211(\351\206\001\276\304JN\340\233~\347x[\304#\217\030\245\025\247\225\267\300\033\001\231\242\034\372'\036\361K\333\021\372\320B\225\027\372M\"7\357\222y4\032\353\314T\326\064q=\272\023\274.\264u\fP\306t2\023\312k\336\352K9\206;\231\226\067D]\335\327-\317NnJ\215\253\022Z\213\003I\370\036\375\302\201\032\207Ry]4\364ء\346\020\036\267YW+*c\347\263O\220\025\357\302J\244J\003\217-\310\366:W\260Mk~\212P\017\311\316\361t\257]$xh\311~(8m8"..., to=0x6f4590 "xkp\367\377\177", key=0x6f3ef0, padding=4) at ../../src/src/p11_front.c:442`

#3 0x00007ffff6f3639c in pkcs11_rsa_priv_dec_method (flen=256, from=0x6f4380 "ϟ$5;\213\336\376Μ\314KԊw!U\237\355q\227\071\223\026\246WX$\250\004|\301§\257\335}#\201P\211(\351\206\001\276\304JN\340\233~\347x[\304#\217\030\245\025\247\225\267\300\033\001\231\242\034\372'\036\361K\333\021\372\320B\225\027\372M\"7\357\222y4\032\353\314T\326\064q=\272\023\274.\264u\fP\306t2\023\312k\336\352K9\206;\231\226\067D]\335\327-\317NnJ\215\253\022Z\213\003I\370\036\375\302\201\032\207Ry]4\364ء\346\020\036\267YW+*c\347\263O\220\025\357\302J\244J\003\217-\310\366:W\260Mk~\212P\017\311\316\361t\257]$xh\311~(8m8"..., to=0x6f4590 "xkp\367\377\177", rsa=0x6f1850, padding=4) at ../../src/src/p11_rsa.c:341`

#4 0x000000000042e4a7 in rsautl_main (argc=<optimized out>, argv=<optimized out>) at rsautl.c:308

#5 0x000000000041a448 in do_cmd (prog=prog@entry=0x6a8700, argc=argc@entry=13, argv=0x7fffffffdc90) at openssl.c:477

#6 0x000000000041a103 in main (Argc=13, Argv=<optimized out>) at openssl.c:371

This shows that the rsautl is expecting the PKCS11 module to support OAEP padding.
But OpenSC pkcs11 does not support it.

If the intent was to have OpenSSL do padding removal after calling PKCS#11 CKM_RSA_X_509 to do the raw RSA operation, this looks like it is not going to work.

[mouse07410]

...This shows that the rsautl is expecting the PKCS11 module to support OAEP padding.

I see.

But OpenSC pkcs11 does not support it.

I see. I'd love to see that support added.

If the intent was to have OpenSSL do padding removal after calling PKCS#11 CKM_RSA_X_509 to do the raw RSA operation, this looks like it is not going to work.

Yes, I think the intent described above would be the most logical solution, as few (if any) tokens support fancy padding directly - and most of them (all of them?) support raw RSA.

From your explanation I see why it cannot work as is - OpenSSL and OpenSC/libp11 implementations "point finger at each other" (figuratively speaking) regarding who should remove the padding, and neither one actually does it.

I do not know what is simpler, or likelier to happen (especially for the 1.0.2 branch):

• Get OpenSSL perform the padding removal and pass only raw RSA to the PKCS11 module;
• Add the OAEP capability to libp11 (or OpenSC?).

What's your opinion? (I'm not asking which path is more elegant - but rather which path has a better chance to be actually implemented and deployed.)

Update. I checked my test-program (C code) that programmatically invokes OpenSSL and PKCS#11 engine to "wrap" and "unwrap" (aka encrypt and decrypt) a symmetric key using RSA-OAEP. It still works correctly with the current libp11 master. Which probably means - the bug is somewhere in openssl/apps/rsautl.c. Still, if p11_rsa.c could accommodate OAEP, it would probably be a quicker solution...

This is the code sample that works now (error checks and recovery removed for compactness):

        const char *KeyManPrivKey = "pkcs11:manufacturer=piv_II;id=%03;type=private";
	/* Loaded pkcs11 engine - e is a pointer to it */

	privkey = ENGINE_load_private_key(e, KeyManPrivKey, NULL, &cb_data);
	ctx = EVP_PKEY_CTX_new(privkey, NULL);
	EVP_PKEY_free(privkey);

	rv = EVP_PKEY_decrypt_init(ctx);
	if (rv <= 0) goto end;
	rv = EVP_PKEY_CTX_set_rsa_padding(ctx, PADDING);

	*olen = 0;
	rv = EVP_PKEY_decrypt(ctx, NULL, olen, in, inlen);

	*out = OPENSSL_malloc(*olen);
	rv = EVP_PKEY_decrypt(ctx, *out, olen, in, inlen);

 end:
 . . . . .

[dengert]

The OpenSSL-1.1 does the same thing.

From your https://github.com/OpenSC/libp11/files/911548/libp11-decr.txt it has:

100: C_DecryptInit
2017-04-10 17:11:53.817
[in] hSession = 0x7f89d9f0b980
pMechanism->type=CKM_RSA_PKCS
[in] hKey = 0x7f89d9f044d0
Returned: 99 CKR_KEY_TYPE_INCONSISTENT

And I don't see how the OpenSSL rsautil or the libp11 would select CKM_RSA_PKCS?
And I don't see why OpenSC would not accept the above id it was really PKCS1.
pkcs11-tool -l -t appear to test C_Decrypt with CKM_RSA_PKCS to do PKCS1.5 padding removal.

@mouse07410 you asked: "I do not know what is simpler, or likelier to happen"

Its complicated because PKCS#11 2.40 Section 2.1.8 "PKCS #1 RSA OAEP" says:
"It has a parameter, a CK_RSA_PKCS_OAEP_PARAMS structure."
This needs to be provided in the CK_MECHANISM as the , CK_VOID_PTR pParameter; and CK_ULONG ulParameterLen;

So based on what rsautl does for parameters, it needs to provide them to PKCS#11.
It is not clear if there is a default for the parameters or if the default would be the same as in PKCS11.

OpenSSL using software, .crypto/rsa/rsa_pmeth.c pkey_rsa_decrypt checks for
rctx->pad_mode == RSA_PKCS1_OAEP_PADDING
then does a RSA_private_decrypt(inlen, in, rctx->tbuf, ctx->pkey->pkey.rsa, RSA_NO_PADDING);
The equivalent of a CKM_RSA_X_509 raw operation, then RSA_padding_check_PKCS1_OAEP_mgf1

But in rsautl it looks like it just jumps to doing assuming PKCS11 can do it but does not pass in any parameters. So it looks like rsautl is the problem. And it is not clear what parameters are the default, or if they can be provided. (I have not looked at the OpenSSL code to see if there is a default. )

If you wanted to have your own version of rsautil, that relied on the lower level engine code, and not pkcs11 you might get it to work with out changes to libp11 or opensc.

If there is a default for the parameters, then OpenSC mods to support CKM_RSA_PKCS_OAEP
could be added.
Eventially OpenSC pkcs11 should support OAEP. Sounds like a big project. Windows supports it,
as the minidiver.c has a "TODO" comment, and in ./libopensc/authentic.h
there is" #define AUTHENTIC_ALGORITHM_RSA_OAEP 0x13
so some card may even support it on the card. But the OpenSC code has not been written.

[mouse07410]

Its complicated because PKCS#11 2.40 Section 2.1.8 "PKCS #1 RSA OAEP" says:
"It has a parameter, a CK_RSA_PKCS_OAEP_PARAMS structure."
This needs to be provided in the CK_MECHANISM as the , CK_VOID_PTR pParameter; and CK_ULONG ulParameterLen;

Ah, but that assumes that the PKCS#11 device would process OAEP? I'd like to constrain the PKCS11 part to the raw RSA if at all possible - because (a) OpenSC seems to rely upon OpenSSL in any case, and (b) it looks like OpenSSL can process OAEP perfectly well (as my code sample above proves).

So based on what rsautl does for parameters, it needs to provide them to PKCS#11.
It is not clear if there is a default for the parameters or if the default would be the same as in PKCS11.

Again, I'd rather that OpenSSL took care of OAEP by itself, and passed only the raw RSA down to the PKCS11 end. And that's what it must be doing with the encryption, otherwise both -encrypt and -decrypt would fail.

But in rsautl it looks like it just jumps to doing assuming PKCS11 can do it but does not pass in any parameters. So it looks like rsautl is the problem.

It looks like that. So rsautl needs to be fixed, right?

BTW, here's what I get from pkcs11-tool -l -t:

$ pkcs11-tool -l -t
Using slot 1 with a present token (0x4)
Logging in to "PIV Card Holder pin (PIV_II)".
Please enter User PIN: 
C_SeedRandom() and C_GenerateRandom():
  seeding (C_SeedRandom) not supported
  seems to be OK
Digests:
  all 4 digest functions seem to work
  MD5: OK
  SHA-1: OK
  RIPEMD160: OK
Signatures (currently only for RSA)
  testing key 0 (PIV AUTH key) 
  all 4 signature functions seem to work
  testing signature mechanisms:
    RSA-X-509: ERR: verification failed
    RSA-PKCS: OK
    SHA1-RSA-PKCS: OK
    MD5-RSA-PKCS: OK
    RIPEMD160-RSA-PKCS: OK
    SHA256-RSA-PKCS: OK
  testing key 1 (2048 bits, label=SIGN key) with 1 signature mechanism
Logging in to "PIV Card Holder pin (PIV_II)".
Please enter context specific PIN: 
    RSA-X-509: OK
  testing key 2 (2048 bits, label=KEY MAN key) with 1 signature mechanism -- can't be used to sign/verify, skipping
  testing key 3 (2048 bits, label=CARD AUTH key) with 1 signature mechanism
    RSA-X-509: OK
Verify (currently only for RSA)
  testing key 0 (PIV AUTH key)    RSA-X-509:   ERR: C_Verify() returned CKR_GENERAL_ERROR (0x5)
  testing key 1 (SIGN key) with 1 mechanism    RSA-X-509: Logging in to "PIV Card Holder pin (PIV_II)".
Please enter context specific PIN: 
  ERR: C_Verify() returned CKR_GENERAL_ERROR (0x5)
  testing key 2 (KEY MAN key) with 1 mechanism -- can't be used to sign/verify, skipping
  testing key 3 (CARD AUTH key) with 1 mechanism    RSA-X-509:   ERR: C_Verify() returned CKR_GENERAL_ERROR (0x5)
Unwrap: not implemented
Decryption (currently only for RSA)
  testing key 0 (PIV AUTH key) 
    RSA-X-509: OK
    RSA-PKCS: OK
  testing key 1 (SIGN key) 
    RSA-X-509: OK
    RSA-PKCS: OK
  testing key 2 (KEY MAN key) 
    RSA-X-509: OK
    RSA-PKCS: OK
  testing key 3 (CARD AUTH key)  -- can't be used to decrypt, skipping
4 errors
$

[mouse07410]

Turns out the problem is in openssl/apps/rsautl.c. Replacing the command in the script with openssl pkeyutl with -pkeyopt rsa_padding_mode:oaep resolved the issue.

Hopefully, eventually we'll see both: rsautl.c will be fixed, and OpenSC will support OAEP. ;-)