Malformed data error when signing SHA3-384 with yh_util_sign_pkcs1v1_5

[ktrinh-anduril]

I'm running into an issue signing a SHA3-384 hash of one of our images(SHA3-384 is required by the ROM of our device so we can't change what's used). Since yubihsm doesn't support SHA3, I opted to do the hash as well as encoding before signing and while it works if the asn1 ID part of the digest is for SHA2-384, it doesn't work if it's SHA3-384 even when the hash is the same but the asn1 ID is different.

For example, this is a hexdump of the SHA2-384 that was signed:

0000 - 00 01 ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0010 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0020 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0030 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0040 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0050 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0060 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0070 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0080 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0090 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
00a0 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
00b0 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
00c0 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
00d0 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
00e0 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
00f0 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0100 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0110 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0120 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0130 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0140 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0150 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0160 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0170 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0180 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0190 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
01a0 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
01b0 - ff ff ff ff ff ff ff ff-ff ff ff ff 00 30 41 30   .............0A0
01c0 - 0d 06 09 60 86 48 01 65-03 04 02 02 05 00 04 30   ...`.H.e.......0
01d0 - c7 7e e5 d3 f7 06 9d f8-aa c7 53 57 fc b7 95 4b   .~........SW...K
01e0 - 5d 92 e6 81 f6 6c a7 18-a9 b8 bd 02 9c 67 e3 17   ]....l.......g..
01f0 - c3 96 c9 ed 48 4d 9a 55-59 22 0e c3 32 d5 89 ec   ....HM.UY"..2...

As you can see at 0x01c0 where the ASN1 ID portion is located, it's specified to be 0x60,0x86,0x48,0x01,0x65,0x03,0x04,0x02,0x02 which specifies OID for SHA2-384. However, if I change this part to be 0x60,0x86,0x48,0x01,0x65,0x03,0x04,0x02,0x09 (differ the last byte), then yubihsm would complain about malformed data

This confuses me because I didn't think yubihsm would care about the ASN1 ID part, just that the data is correctly padded and of correct length. And weirdly, even if I gave it sha3-384 but specified in the asn1 part that it's sha2-384, yubihsm still signs it successfully so it looks like it doesn't really care if it's actually sha3-384 or sha2-384

In summary, is there a way to get the yubihsm to RSA sign a sha3-384 hash padded using pkcs1.5?

[qpernil]

SHA3 is currently not a supported algorithm. The YubiHSM checks what hash is used for two purposes - to determine if it is enabled, and secondly to add the proper OID if a raw hash is given, in which case the length of the hash is used to determine the hash algorithm. That is needed because the OID is part of a pkcs#1 signature. See https://developers.yubico.com/YubiHSM2/Commands/Sign_Pkcs1.html for info on what algorithms are supported.
So even if supplying a SHA2-384 OID works the signature will be incorrect when verified as a pkcs#1-sha3-384 signature (because the oid part of the signed data will be different), and also incorrect when verified as a pkcs#1-sha2-384 signature (because the hash part of the signed data will be different).