Chunk 11 of CMP contribution to OpenSSL: CMP command-line interface

[DDvO]

This PR adds the CMP app and its documentation.
CLI-based tests will be the topic of the final chunk 12.

Update: List of open spin-off PRs (including current state) on which this PR has been depending:

• Generalize setup_engine of apps/apps.c to allow specific crypto methods #4277 (generalize setup_engine) - merged
• Various fixes regarding PKCS#12 input and related cleanup of apps, doc, and tests #4930 (improve PKCS#12 loading) - to be extended and review to be finished
• fix mem leaks in try_decode_PKCS12() of crypto/store/ #11733 (fix mem leaks loading PKCS#12) - review to be finished
• Generalize HTTP server code from apps/ocsp.c to apps/lib/http_server.c #11736 (generalize HTTP server) - merged
• Use OSSL_STORE in apps, generalizing and simplifying load_key() etc. #11755 (use OSSL_STORE in apps.c) - merged

The CLI enables everyone to easily try out typical certificate management use cases
with the demo configuration added to apps/openssl.cnf, which refers to the Insta Demo CA.
For instance, as described in the EXAMPLES section of doc/man1/openssl-cmp.pod:

  export OPENSSL_CONF=apps/openssl.cnf
  export LD_LIBRARY_PATH=.
  wget 'http://pki.certificate.fi:8080/install-ca-cert.html/ca-certificate.crt\
        ?ca-id=632&download-certificate=1' -O insta.ca.crt
  openssl genrsa -out insta.priv.pem

  apps/openssl cmp -section insta,ir                 # initial cert request
  apps/openssl x509 -noout -text -in insta.cert.pem  # view the enrolled cert

  apps/openssl cmp -section insta,cr                 # subsequent cert request
  apps/openssl cmp -section insta,kur                # cert update
  apps/openssl cmp -section insta,rr,signature       # cert revocation

[DDvO]

This is a pretty large body of code and documentation,
mostly due to the flexibility of CMP and the many options implemented.

[DDvO]

About 1000 of the 3500 lines of code in apps/cmp.c are of more generic nature,
for example implementing more flexible and convenient loading of credentials etc.
Most of those functions could be moved to apps/lib/, as indicated by the TODOs given.

I had tried contributing such changes long ago in the PRs #4277, #4930, and #4940,
but that got stuck due to lack of general interest.
I could revive that as part of the present PR, but I don't think it's strictly needed.
Let's see what comments will come up on this topic.

[DDvO]

Travis CI took too long on one of the test runs as usual; everything else went fine.
BTW, how about increasing that 50 mins timeout value to, e.g., 60 mins?

[richsalz]

It already is 60 minutes. @t8m is working on a PR to remove that build

[DDvO]

Rebased to the latest master (now including the fixes of #11448).

[DDvO]

Has anyone already tried out the CMP app? For instance:

OPENSSL_CONF=apps/openssl.cnf apps/openssl cmp -section insta

[DDvO]

Currently shown Travis CI issues are unrelated -
still the usual timeout on one build, and this time also some krb5 test failure.
Rebasing...

[mattcaswell]

Some review comments so far. I have not yet looked at apps/cmp.c - but I've looked at everything else.

[DDvO]

Some review comments so far. I have not yet looked at apps/cmp.c - but I've looked at everything else.

Thanks @mattcaswell for these comments - I've answered/handled all of them.
Makes much sense to start by having a look at the documentation first.

[DDvO]

A bit awkward that changes to apps/openssl.cnf must be kept in sync with apps/openssl-vms.cnf.
I just found that this is done by make generate_apps, as part of make update.

[levitte]

I had tried contributing such changes long ago in the PRs #4277, #4930, and #4940, but that got stuck due to lack of general interest.

I'll regard that as a huge PING!
It would be better to review those separately, rather that have them gobbled up in this PR

[DDvO]

I'll regard that as a huge PING!
It would be better to review those separately, rather that have them gobbled up in this PR

@levitte I'm glad that you picked this up.
As I wrote above the present PR also works without those three PRs, but it's going to become a bit shorter and cleaner (with the respective TODOs I included in apps/cmp.c having been resolved).
So I've dusted off those PRs (i.e.., rebased them etc.) and started handling your new comments there.

@mattcaswell, this consolidation activity should not have much effect on your review of apps/cmp.c in the sense that there is enough other code to look at.
I wonder if you already started reviewing this file?

[mattcaswell]

I wonder if you already started reviewing this file?

I started to look and then asked @levitte if he could have a look at the engine related stuff. I will try to get back to it...but I'm a little distracted by the looming alpha1 at the moment. Probably it will be later the week before I can look.

[DDvO]

Rebased on current master, which includes the newly merged #4940.

[DDvO]

@slontis, let's do the adaptation of the CMP & CRMF lib modules for providers in a separate PR.
When this PR #11470 been merged we can rebase on it and adapt also the CLI as far as needed.

Who should be the one setting up that PR and maintaining the branch with the code changes?
I think I can start contributing to this topic tomorrow.

[mattcaswell]

LGTM

[slontis]

Who should be the one setting up that PR and maintaining the branch with the code changes?
I think I can start contributing to this topic tomorrow.

I am happy for you to be in charge. Talking to matt he thought that it should be approached from the view of changing the tests to use providers and then get that working.. I can start looking at the test setup side of it if you like.

[DDvO]

@slontis, sounds good.
So I'm going to set up today a WIP branch starting from changes to the lib components, while you will focus on the test aspects.
Can we somehow work on the same branch (on which repo?), or would you have test-oriented branch based on the former?

[slontis]

Can we somehow work on the same branch (on which repo?), or would you have test-oriented branch based on the former?

Just set up a branch like you normally would on master and I will add some commits to it
Should be no problem if we just say it is a WIP.
force pushing is a bit deadly though :)

[DDvO]

Normally I set up branches on the Siemens repo or the one by @mpeylo where I have write access, but you won't be able to push to either of those, and I since I'm not the owner I cannot give others write access. Is there a repo where we both can push to?

[DDvO]

Or is it sufficient if you have read access to my WIP branch?

[slontis]

See if you can do

>git fetch git@github.com:slontis/openssl.git  cmp_provider_support:{yourbranchname}

[DDvO]

This did not work directly, but as follows:

git clone git@github.com:slontis/openssl.git slontis
cd slontis
git fetch origin cmp_provider_support:cmp_provider_libext

[slontis]

I will stop polluting this PR with comments. We can pollute the new PR instead :)

[openssl-machine]

24 hours has passed since 'approval: done' was set, but as this PR has been updated in that time the label 'approval: ready to merge' is not being automatically set. Please review the updates and set the label manually.

[mattcaswell]

Looks like this is good to go @DDvO

[DDvO]

Pushed - hooray!
Thanks a lot @mattcaswell!

[yangyangtiantianlonglong]

@DDvO Hi, When will the cmp and crmf feature plans be fully integrated into the openssl master ？ I want to make my openssl version plan based on your plan. https://github.com/mpeylo/cmpossl/wiki, I see that the ossl-cmp branch plan is already inaccurate, I hope you can update the plan

[DDvO]

Hi @yangyangtiantianlonglong,

When will the cmp and crmf feature plans be fully integrated into the openssl master I want to make my openssl version plan based on your plan. https://github.com/mpeylo/cmpossl/wiki, I see that the ossl-cmp branch plan is already inaccurate, I hope you can update the plan

I updated the plan on https://github.com/mpeylo/cmpossl/wiki yesterday, so it is up-to-date.

As of yesterday, the code of the CMP implementation in OpenSSL is complete.
Yet we have some test extensions and various (mostly internal) improvements in the pipeline.
I presume that most of this will be included in the OpenSSL dev master around end of May.

Over the next months I do not expect CMP API changes (and if so, they will be minimal).
The official release of OpenSSL 3.0, including CMP, is planned for October, as you can see here:
https://www.openssl.org/policies/releasestrat.html