intermittent cms test failure

[jamuir]

Tomas M pointed out the following intermittent CI failure against the 3.1 branch:

https://github.com/openssl/openssl/actions/runs/7184322214/job/19565167165

Here is a screenshot:

The failure is in a cms test.

It can be reproduced against openssl-3.1 by running those cms-encrypt / cms-decrypt commands in a loop (script attached below).

$ cd issues/ci-cms-fail
$ ./doit.bash
Error decrypting CMS structure
40A7E6F88E7F0000:error:1C800064:Provider routines:ossl_cipher_unpadblock:bad decrypt:providers/implementations/ciphers/ciphercommon_block.c:124:
$ cat iters.txt
59437

It can take several minutes to reproduce the error, but it is reproducible.

I ran the script three times and observed the following iteration counts:

59437
29304
29927

(i.e. the test passed 59436 times, then failed after the 59437 time, etc.)

doit.bash.txt

[jamuir]

Here are more iteration counts (against 3.1):

73665
26700
84039
 5073
 2910

[jamuir]

I was able to reproduce this against master as well, but it took a large number of iterations:

$ ./doit.bash
Error decrypting CMS structure
80CB7692A17F0000:error:1C800064:Provider routines:ossl_cipher_unpadblock:bad decrypt:providers/implementations/ciphers/ciphercommon_block.c:107:
$ cat iters.txt 
167763

[jamuir]

The encrypt part of the test targets three recipients:

openssl cms \
               -provider-path ../../providers -provider default \
               -encrypt -in ../../test/smcont.txt -stream -out cms2cms-1-16.cms \
               ../../test/smime-certs/smrsa1.pem \
               ../../test/smime-certs/smrsa2.pem \
               ../../test/smime-certs/smrsa3.pem

The decrypt part of the test is done using only the private key inside smrsa3.pem.

When the test fails, it does so b/c the tdes decryption for recipient 3 fails (usually due to a bad unpad operation):

$ . dec3.bash
1e28cd46cd4e3982c49c94ab2dcb2c43  ../../test/smime-certs/smrsa3.pem
b656f31b53618f2df424ddb927d5fea4  cms2cms-1-16.cms
176 = pad
08 = blocksize
Error decrypting CMS structure
40375A20F77F0000:error:1C800064:Provider routines:ossl_cipher_unpadblock:bad decrypt:providers/implementations/ciphers/ciphercommon_block.c:126:
$ . dec3.bash
1e28cd46cd4e3982c49c94ab2dcb2c43  ../../test/smime-certs/smrsa3.pem
b656f31b53618f2df424ddb927d5fea4  cms2cms-1-16.cms
146 = pad
08 = blocksize
Error decrypting CMS structure
40E72669187F0000:error:1C800064:Provider routines:ossl_cipher_unpadblock:bad decrypt:providers/implementations/ciphers/ciphercommon_block.c:126:
$ . dec3.bash
1e28cd46cd4e3982c49c94ab2dcb2c43  ../../test/smime-certs/smrsa3.pem
b656f31b53618f2df424ddb927d5fea4  cms2cms-1-16.cms
13 = pad
08 = blocksize
Error decrypting CMS structure
4087139F7D7F0000:error:1C800064:Provider routines:ossl_cipher_unpadblock:bad decrypt:providers/implementations/ciphers/ciphercommon_block.c:126:

The decrypt attempts above are all done against the same ciphertext. The printed value of pad is supposed to be in the range 1..8. It is interesting that we see different values for pad with each decrypt attempt.

Decryption using smrsa1.pem and smrsa2.pem succeeds:

$ . dec1.bash
69cd8bbb4f5dca8c253a7dc12b389030  ../../test/smime-certs/smrsa1.pem
b656f31b53618f2df424ddb927d5fea4  cms2cms-1-16.cms
$ . dec2.bash
84336fb24001deab25cb40c22e31aa3e  ../../test/smime-certs/smrsa2.pem
b656f31b53618f2df424ddb927d5fea4  cms2cms-1-16.cms

This suggests to me that it is actually the RSA ciphertext for recipient 3 that is faulty (i.e. the RSA ciphertext that holds the encrypted tdes key for recipient 3).

[jamuir]

Quoting from man openssl-cms:

If the -decrypt option is used without a recipient certificate then an attempt is made to locate the recipient by trying each potential recipient in turn using the supplied private key.

That is the cause of the intermittent failure.

Let R3 denote recipient 3.

We do a cms-decrypt operation without specifying R3's cert -- we only use R3's private RSA key. Each of the three RSA ciphertexts is decrypted using R3's private key, and, unluckily, more than one of those RSAES-PKCS-v1_5 decryption ops succeeds. The result is that the wrong content encryption key is recovered, and that results in a failed content decryption.

We can avoid the intermittent failures by providing R3's cert during decryption using the -recip option.

[jamuir]

Attached is a copy of one of the "bad" cms encrypted messages.

It can be used to reproduce the failure and verify that specifying -recip avoids it.

cms2cms-1-16.cms.txt

[jamuir]

It is interesting that we see different values for pad with each decrypt attempt.

I wanted to understand why repeatedly running the decrypt op against one of the bad cms-encrypted messages resulted in different output.

During a key-only decrypt, we loop over each RecipientInfo object and decrypt its RSA ciphertext (to attempt to get the content-encryption key).

A pointer to the last successfully decrypted RSA ciphertext is saved. It may potentially be used to decrypt the content, but note that all of the RSA decryptions might fail, and more than one might succeed.

The intermittent failures we encountered happened because the last successfully decrypted RSA ciphertext was not the right one.

In the cms code referenced below, before ec->key is used to decrypt the content, it is checked.

If it is NULL, then it is replaced with tkey (a random key).

If it does not have the correct length, then it is also replaced with tkey.

This last case explains the different output I saw: the same potential key was considered each time, but it did not have the right length (it was 175 bytes, whereas a TDES key is 24 bytes), so a random content-encryption key was used to do the decryption.

https://github.com/openssl/openssl/blob/e4542332fa36eab6d6bbf33815bde433ade3b547/crypto/cms/cms_enc.c#L116-L160

[beldmit]

Probably this discussion makes sense: openssl/openssl#13817 (comment)