Add CMS AuthEnvelopedData with AES-GCM support #8024

bukka · 2019-01-15T17:28:46Z

This PR adds support for CMS AuthEnvelopedData as defined in RFC 5083 with AES-GCM parameter as defined in RFC 5084.

I think it useful to support GCM as it allows the tag to be stored with the message. Especially if the cms is now recommended tool for AEAD and there was a decision not to introduce special format for apps enc to handle AEAD modes IIRC.

It works already with use of apps cms when selecting for example -aes-128-gcm cipher. It updates CMS_encrypt to automatically use AuthEnvelopedData for GCM modes so the application doesn't require new API. As there is now new special API, it means that there is currently no support for AAD but that's something that I would like to look in the future as part of another PR. Also CCM mode is not supported as it's a bit more tricky (mainly because the length needs to be known in advance) but I plan to look into it in the future too.

The documentation is partially updated which means that I updated CMS_encrypt and CMS_decrypt. The reason why I left the rest out is that I'm not sure if the new public functions should be really public. Currently I added them as a public for consistency. For example EVP_CIPHER_set_asn1_aead_params is an AEAD variant for EVP_CIPHER_set_asn1_iv so it might be useful to expose it possibly for engines use. Similarly ASN1_TYPE_set_octetstring_int is consistent with public ASN1_TYPE_set_int_octetstring (also not documented). In addition the new CMS_AuthEnvelopedData_create is like exposed CMS_EnvelopedData_create (also not documented) but for AuthEnvelopedData of course. Considering that all the existing mentioned functions are not documented, it might be better to leave the documentation for another PR and do it together with the existing once. I would be happy to do that later.

Checklist

documentation is added or updated
tests are added or updated

bukka · 2019-01-15T17:29:32Z

FYI I have sent ICLA this morning.

crypto/cms/cms_env.c

doc/man3/CMS_encrypt.pod

crypto/evp/evp_lib.c

russhousley · 2019-01-16T21:47:01Z

Thanks for all of this very valuable work. The S/MIME 4.0 specifications are in the final stages, and they make use of AuthEnvelopedData!

Ideally, in the future, AAD support for the authenticated attributes in AuthEnvelopedData would be the next step.

bukka · 2019-01-18T20:20:56Z

@russhousley Yes I plan to look on adding AAD support after this gets reviewed and hopefully merged. I'm still thinking about the best API for that but the implementation should be fairly easy. Although I think it will require some improvements in the CMS testing as it's quite limited atm. I would like to look on adding ChaCha20-Poly1305 (thank you for creating RFC 8103 btw.) as well and I also thought about CCM support but it will probably require loading the whole plaintext to the memory BIO to get the length before it's processed by cipher BIO. That needs a bit more thinking though!

crypto/cms/cms_env.c

crypto/cms/cms_pwri.c

wolftobias · 2019-06-12T09:43:48Z

@levitte Could you pls take a look at this? I also would need the support for 'AuthEnvelopData'.

wolftobias · 2019-06-13T06:42:16Z

@bukka @russhousley @FdaSilvaYY @mattcaswell Hi guys, I think here is a 2nd reviewer needed! Who can support us?

wolftobias · 2019-06-13T06:43:51Z

@bukka You developed this on the future openssl version 3.0.0, right?

crypto/cms/cms_enc.c

crypto/cms/cms_env.c

crypto/evp/evp_lib.c

include/openssl/cms.h

include/openssl/evp.h

crypto/asn1/evp_asn1.c

mattcaswell · 2019-06-14T14:35:35Z

test/recipes/80-test_cms.t

+      [ "-decrypt", "-in", "test.cms", "-out", "smtst.txt", "-inform", "PEM",
+	"-secretkey", "000102030405060708090A0B0C0D0E0F",
+	"-secretkeyid", "C0FEE0" ]
+    ],


I know your just following the way the other CMS tests are done, but really I'd like to see stronger tests that this. Rather than just testing things through the command line we should really be testing the API directly...e.g. with some negative tests too.

I agree! I guess it might make more sense if I create a new with some tests for encryption and then just extend it in this PR. What do you think?

Again I'd be delighted if you want to do that. I would accept more minimal testing which just tests the newly added functions in order to approve this PR. I'll leave it with you to decide.

I plan to add those tests and already started looking into it. It's basically next thing that I want to do before I start looking to the CCM and ChaCha20-Poly1305 in CMS as it's currently quite a pain to debug issues and API tests would really help.

I'm just thinking that it would be helpful for me if it doesn't have to be requirement for this PR? The reason is that I'm spending quite a bit of time just on rebasing it and integrating changes into it (e.g. the GOST related one took a bit of time). If it doesn't need to wait for adding a new way of testing in CMS, that would be great as it would save me some time!

I just extended the cmsapitest.c to at least cover the gcm mode. It's not a great test but hope it should do for this PR.

I was thinking about improvements in CMS testing and I think that the main issue it that the actual ASN.1 elements are not verified in any way. It's basically just encrypted envelope can be decrypted and the result is the same. But if there was some structural bug in both, then it could be easily missed. Maybe some kind of match for asn1parse as part of cli test..? Anyway that's more for a different PR but thoughts on this would be appreciated.

bukka · 2019-06-16T19:06:01Z

@bukka You developed this on the future openssl version 3.0.0, right?

@wolftobias Yes it targets 3.0.0. I just rebased it on latest master.

wolftobias · 2019-06-17T07:03:09Z

I need this new feature on the current v1.1.1xxx, maybe I have to integrate it manually in my local build, version 1.1.1 has long time support till 2023 and version 3 has not target release date yet. Do you or anybody else see a chance to make this available for v1.1.1?

mattcaswell · 2019-06-17T10:18:42Z

Do you or anybody else see a chance to make this available for v1.1.1?

Unfortunately not. 1.1.1 is a stable release. We don't add new features to stable releases (and this is counted as a new feature).

wolftobias · 2019-06-17T11:05:44Z

Do you or anybody else see a chance to make this available for v1.1.1?

Unfortunately not. 1.1.1 is a stable release. We don't add new features to stable releases (and this is counted as a new feature).

Yes I understand your point. Is there already an idea about the target release date of v3?

mattcaswell · 2019-06-17T11:22:02Z

Yes I understand your point. Is there already an idea about the target release date of v3?

We hope to be code complete by the end of this year, i.e. beta release around that point. With a possible release around early Q2 2020 (FIPS validation would come some time later).

wolftobias · 2019-07-01T13:42:38Z

@bukka @mattcaswell What's the current status?

mattcaswell · 2019-07-01T15:33:42Z

Oh...sorry, it looks like @bukka previously responded to earlier review comments and is waiting on me to respond to some questions. I'll try and answer those now.

mattcaswell · 2019-07-01T15:54:23Z

I've responded to the various comments above...over to @bukka. Looks like this PR needs a rebase due to conflicts with master

bukka · 2019-07-21T17:22:02Z

I have been bit busy lately. Will try to address all the outstanding bits when I have got a bit more free time.

wolftobias · 2019-07-22T07:32:44Z

I have been bit busy lately. Will try to address all the outstanding bits when I have got a bit more free time.

time is always against us! But it would be great if it would be possible for you to finish the open tasks, otherwise we're risking not to go in the next release and that will be sad. Also the reviewers took there time to work on this.

wolftobias · 2019-07-23T10:14:51Z

@bukka I tried to test your unauthenticated attribute feature you developed, but the result is the error below. Could you please help me further?

C:\download\openssl-bukka\win64\bin>openssl cms -encrypt -in test.txt -out test.txt.enc -aes-128-gcm -secretkey 000102030405060708090A0B0C0D0E0F -secretkeyid C0FEE0
35672:error:060CD0E4:digital envelope routines:EVP_CIPHER_param_to_asn1:reason(228):crypto/evp/evp_lib.c:46:
35672:error:2E078066:CMS routines:cms_EncryptedContent_init_bio:cipher parameter initialisation error:crypto/cms/cms_enc.c:142:
35672:error:2E07F068:CMS routines:CMS_final:cms lib:crypto/cms/cms_smime.c:764:

crypto/asn1/evp_asn1.c

bukka · 2020-09-07T11:34:40Z

@slontis Addressed the last 2 comments.

slontis · 2020-09-07T11:40:59Z

Unless you really need to rebase could you please start using commit fixups..
It makes it a lot harder to review your changes on something large if you do it via force pushing..

To do this you do

git log
To view your commits and get the hash of the commit you want to add a fixup to..
git commit --fixup hashid
git push

Note you can also do this with the generated files (e.g libcrypto.num)..

slontis · 2020-09-07T11:45:07Z

Have a look at something simple like
https://github.com/openssl/openssl/pull/12795/commits
And you be able to see the change fixups..
At the moment you are doing monolithic blobs - which means I have to scan everything each time (not just what you changed).

slontis · 2020-09-07T11:47:23Z

NOTE: when we merge the committers can easily squash these fixups as part of the process.

slontis

Approved assuming the CI loop passes.

I wish we had something to interop against though - your code is likely to become the thing that is interoped against in the future :)

bukka · 2020-09-07T12:01:33Z

@slontis Thanks for the approval.

The reason why I kept it in a single commit is because I needed to rebase it quite a few times and there were lots of conflicts. If I had lots of commits, it would be really a pain to rebase it. Although I guess I could just merge master in if it's something that you are fine with? I can certainly do that next time. It works for me as soon as I don't need to rebase multiple commits...

slontis · 2020-09-07T12:06:26Z

The reason why I kept it in a single commit is because I needed to rebase it quite a few times and there were lots of

I understand.. What I do is collapse it when it is ready to start getting reviews.. Then once people start reviewing just do the fixups to make it easy to review afte the initial review.. If it is going to sit there for a long time and cause you lots of conflicts later then collapse it again and start the fixups process again when it starts getting reviewed again.

t8m · 2020-09-07T12:26:11Z

Approved, if the CI passes.

bukka · 2020-09-07T20:42:08Z

@slontis @t8m @mattcaswell The CI is green so hopefully it's good to be merged.

openssl-machine · 2020-09-08T13:00:26Z

24 hours has passed since 'approval: done' was set, but as this PR has been updated in that time the label 'approval: ready to merge' is not being automatically set. Please review the updates and set the label manually.

Add the AuthEnvelopedData as defined in RFC 5083 with AES-GCM parameter as defined in RFC 5084.

Add the AuthEnvelopedData as defined in RFC 5083 with AES-GCM parameter as defined in RFC 5084. Reviewed-by: Shane Lontis <shane.lontis@oracle.com> Reviewed-by: Tomas Mraz <tmraz@fedoraproject.org> (Merged from #8024)

t8m · 2020-09-08T13:45:56Z

In the force push I've rebased and resolved trivial conflict in libcrypto.num.
Merged to master.
Thank you for the contributed PR and for the patience with the reviews and merging.

elear · 2020-09-15T09:23:41Z

@bukka does this require a PR for PHP?

bukka · 2020-09-17T10:45:22Z

@elear No, it should just work.

openssl-machine added the hold: cla required The contributor needs to submit a license agreement label Jan 15, 2019

FdaSilvaYY reviewed Jan 15, 2019

View reviewed changes

crypto/cms/cms_env.c Outdated Show resolved Hide resolved

FdaSilvaYY reviewed Jan 15, 2019

View reviewed changes

crypto/cms/cms_env.c Show resolved Hide resolved

FdaSilvaYY reviewed Jan 15, 2019

View reviewed changes

doc/man3/CMS_encrypt.pod Outdated Show resolved Hide resolved

FdaSilvaYY reviewed Jan 15, 2019

View reviewed changes

crypto/evp/evp_lib.c Outdated Show resolved Hide resolved

bukka closed this Jan 16, 2019

bukka reopened this Jan 16, 2019

openssl-machine removed the hold: cla required The contributor needs to submit a license agreement label Jan 16, 2019

bukka force-pushed the cms-auth-env-gcm branch from c31d6e7 to de3efb3 Compare January 16, 2019 14:40

FdaSilvaYY reviewed Jan 18, 2019

View reviewed changes

bukka force-pushed the cms-auth-env-gcm branch from de3efb3 to 10778e5 Compare January 23, 2019 18:18

mattcaswell requested changes Jun 14, 2019

View reviewed changes

bukka force-pushed the cms-auth-env-gcm branch from 10778e5 to b74eca6 Compare June 16, 2019 18:23

bukka force-pushed the cms-auth-env-gcm branch from fda86bd to ef31de8 Compare September 7, 2020 10:17

slontis reviewed Sep 7, 2020

View reviewed changes

crypto/asn1/evp_asn1.c Outdated Show resolved Hide resolved

slontis reviewed Sep 7, 2020

View reviewed changes

crypto/asn1/evp_asn1.c Outdated Show resolved Hide resolved

bukka force-pushed the cms-auth-env-gcm branch from ef31de8 to 7ecbfcc Compare September 7, 2020 11:33

slontis approved these changes Sep 7, 2020

View reviewed changes

slontis added approval: review pending This pull request needs review by a committer and removed approval: otc review pending This pull request needs review by an OTC member labels Sep 7, 2020

slontis added this to In progress in 3.0 New Core + FIPS via automation Sep 7, 2020

slontis removed this from In progress in 3.0 New Core + FIPS Sep 7, 2020

t8m approved these changes Sep 7, 2020

View reviewed changes

t8m added approval: done This pull request has the required number of approvals and removed approval: review pending This pull request needs review by a committer labels Sep 7, 2020

Add CMS AuthEnvelopedData with AES-GCM support

a9eac52

Add the AuthEnvelopedData as defined in RFC 5083 with AES-GCM parameter as defined in RFC 5084.

t8m force-pushed the cms-auth-env-gcm branch from 7ecbfcc to a9eac52 Compare September 8, 2020 13:42

t8m closed this Sep 8, 2020

gnarea mentioned this pull request Sep 24, 2020

GCMParams in EnvelopedData required to be OCTET STRING instead of SEQUENCE PeculiarVentures/PKI.js#287

Open

bukka mentioned this pull request Mar 7, 2021

Update CHANGES with info about AuthEnvelopedData addition #14456

Closed

Add CMS AuthEnvelopedData with AES-GCM support #8024

Add CMS AuthEnvelopedData with AES-GCM support #8024

bukka commented Jan 15, 2019

bukka commented Jan 15, 2019

russhousley commented Jan 16, 2019

bukka commented Jan 18, 2019 •

edited

wolftobias commented Jun 12, 2019

wolftobias commented Jun 13, 2019

wolftobias commented Jun 13, 2019

mattcaswell Jun 14, 2019

bukka Jun 16, 2019

mattcaswell Jul 1, 2019

bukka Apr 12, 2020

bukka Apr 13, 2020

bukka commented Jun 16, 2019 •

edited

wolftobias commented Jun 17, 2019

mattcaswell commented Jun 17, 2019

wolftobias commented Jun 17, 2019

mattcaswell commented Jun 17, 2019

wolftobias commented Jul 1, 2019

mattcaswell commented Jul 1, 2019

mattcaswell commented Jul 1, 2019

bukka commented Jul 21, 2019

wolftobias commented Jul 22, 2019

wolftobias commented Jul 23, 2019

bukka commented Sep 7, 2020

slontis commented Sep 7, 2020 •

edited

slontis commented Sep 7, 2020

slontis commented Sep 7, 2020 •

edited

slontis left a comment

bukka commented Sep 7, 2020

slontis commented Sep 7, 2020

t8m commented Sep 7, 2020

bukka commented Sep 7, 2020

openssl-machine commented Sep 8, 2020

t8m commented Sep 8, 2020

elear commented Sep 15, 2020

bukka commented Sep 17, 2020

Add CMS AuthEnvelopedData with AES-GCM support #8024

Add CMS AuthEnvelopedData with AES-GCM support #8024

Conversation

bukka commented Jan 15, 2019

Checklist

bukka commented Jan 15, 2019

russhousley commented Jan 16, 2019

bukka commented Jan 18, 2019 • edited

wolftobias commented Jun 12, 2019

wolftobias commented Jun 13, 2019

wolftobias commented Jun 13, 2019

mattcaswell Jun 14, 2019

Choose a reason for hiding this comment

bukka Jun 16, 2019

Choose a reason for hiding this comment

mattcaswell Jul 1, 2019

Choose a reason for hiding this comment

bukka Apr 12, 2020

Choose a reason for hiding this comment

bukka Apr 13, 2020

Choose a reason for hiding this comment

bukka commented Jun 16, 2019 • edited

wolftobias commented Jun 17, 2019

mattcaswell commented Jun 17, 2019

wolftobias commented Jun 17, 2019

mattcaswell commented Jun 17, 2019

wolftobias commented Jul 1, 2019

mattcaswell commented Jul 1, 2019

mattcaswell commented Jul 1, 2019

bukka commented Jul 21, 2019

wolftobias commented Jul 22, 2019

wolftobias commented Jul 23, 2019

bukka commented Sep 7, 2020

slontis commented Sep 7, 2020 • edited

slontis commented Sep 7, 2020

slontis commented Sep 7, 2020 • edited

slontis left a comment

Choose a reason for hiding this comment

bukka commented Sep 7, 2020

slontis commented Sep 7, 2020

t8m commented Sep 7, 2020

bukka commented Sep 7, 2020

openssl-machine commented Sep 8, 2020

t8m commented Sep 8, 2020

elear commented Sep 15, 2020

bukka commented Sep 17, 2020

bukka commented Jan 18, 2019 •

edited

bukka commented Jun 16, 2019 •

edited

slontis commented Sep 7, 2020 •

edited

slontis commented Sep 7, 2020 •

edited