Start migrating to OpenSSL 3.0

[dstebila]

New APIs that we'll be able to make use of, including an EVP API for KEMs and forthcoming pluggable KEMs in TLS 1.3 (#233, openssl#13018).

[baentsch]

Initial functional integration of all NIST PQC KEM candidates available in https://github.com/open-quantum-safe/openssl/tree/OQS-OpenSSL3; with tip of the hat to @romen for openssl#13018 : All new KEM APIs worked perfectly. Status and limitations of this WIP documented in README; suggested ToDos in oqsprovider code; PQC-KEM interoperability with OpenSSL_1_1_1 branch validated (required code-generator in oqs-template/generate.sh).

[baentsch]

One could argue to close the issue as we have started the migration :-) Question(s) open to discussion (creation of other issues or maintaining this issue for documentation/decision-making purposes?):

How do we want to continue after this start? Below's a first list of options. @christianpaquin , @xvzcf , @dstebila Please edit/amend/comment as you see fit.

1. Add CI
2. Add OQS-test suite
3. ensure OpenSSL make test passes 100%
4. Add hybrid KEMs
5. Add full OQS-signature support (incl. hybrids)
6. Add OQS-CMS support
7. Add OQS-speed/s_test profiling support

Initial thoughts:
1-3 would not deliver value before we decide to support this branch for interested external parties: Let's decide (when to do) that. Initial idea: The moment OpenSSL3 goes Beta. Looking at this, this might take some time still.
4-7 would not deliver incremental value over what already is provided in the OQS-OpenSSL_1_1_1 branch.
Doing 5 with an EVP integration as in OQS-OpenSSL_1_1_1 would be work lost as and when the issue documented here gets implemented upstream. We could offer to work on this upstream with higher priority than replicating our OQS-EVP code in OQS-OpenSSL1.1.1: This would enhance OpenSSL and make an OQS provider uptake upstream easier. Also, if OSSL-issue 10512 is implemented perfectly, CMS integration may be "automatically" available.
Doing 7 would also probably preferably be done with upstream feature requests/"direct PRs" as groups now are a first-order, EC-independent concept and one could make a case that they should be made selectable via -groups in all openssl programs (speed, s_time for profiling).

Edit: 3 done. 2 done by way of extending OSSL testing.

[baentsch]

8. Extract new provider code from openssl entirely, i.e., create stand-alone OQS-Provider build environment

This should conceptually be doable, speed up build time and make code updates completely independent from upstream changes (unless APIs change there, of course). This would speak for an entirely separate OQS subproject instead of a branch. Drawback: Effort required to create a cross-platform build environment. Alternative: Straight integration of this code into liboqs (which already has such build env but no OSSL3 dependency... hmmm... Discussion solicited). In any case, arguably only really necessary when OSSL3 goes beta.

Edit: Done (see below with kudos to @levitte)

[mattcaswell]

Extract new provider code from openssl entirely, i.e., create stand-alone OQS-Provider build environment

I would strongly recommend that you take this step. All the provider side code should be implementable without forking OpenSSL at all. This gives you libcrypto integration for all the algorithms, as well as libssl integration for all KEMs.

AFAICT the main piece that you need an OpenSSL fork for is the libssl integration of signature algs. Am I right? This is mainly because I've simply not had the bandwidth to implement the pluggable signature support into libssl yet and unfortunately its unlikely to happen before 3.0 :-(

[baentsch]

Thanks very much for taking the time to give feedback. Yes, we also identified OSSL issue 10512 as the one that would be worth while resolving most urgently from "our" perspective (of adding PQC to OpenSSL with the least pain to everyone, i.e., without the need to retain/maintain an OpenSSL fork). I'd like to try to give 10512 a go, unless you think this is way too complex or too late for getting into a stable OSSL3.0 (@mattcaswell, @romen)?

[romen]

I think such a PR would be highly desirable, but we cannot commit to hold beta1 release in case the PR is not mergeable when we reach the moment to release beta1.

At the moment beta1 does not seem too close (the list of open issues for the beta1 milestone can give some idea of the amount of work still left to do).

So I would personally commit to try and prioritize my reviews to help shape it up and bring such a PR to a point where it is ready, but we can't make guarantees regarding it being included in 3.0 if it misses the cutoff date.

There might be the need for an OTC or OMC decision on the matter (as we did for the pluggable KEM feature, against feature creep).
I could raise the issue at the next OTC meeting to get the ball rolling on this and see if there is consensus on merging it for 3.0 if it becomes ready in time.

[mattcaswell]

Pluggable signatures is definitely going to be complex to implement. I did pluggable kex because the changes I needed to make to enable that to work weren't that much more than the changes I needed to do to make libssl provider aware for kex. The kem support that @romen implemented was simply an extension of the existing kex work.

Signatures OTOH are going to require more extensive changes. I haven't looked too closely at it yet, but one issue that springs to mind is that, until now, we've always had fixed "slots" for different certificate types. So you get one slot for an RSA cert, one for an ECDSA cert, one for DSA, and so on. With pluggable signatures we're going to have to support a varying number of algorithms and so we'll have to dynamically allocate the slots in some way.

Another thing that springs to mind is that we're going to have to support X509 certs with arbitrary signature algs. I haven't looked at what would be required to support that. It might be straightforward....but it might not.

With all that said I don't want to put you off having a go. I'd be delighted if you could get it working and I'd be more than happy to offer what advice and assistance I can given my limited bandwidth.

In terms of the timescale, that is unclear. At the moment there seems like a long list of things we need to fix before beta1 is ready, so we certainly have some time. But we haven't got a definitive target date at the moment. Strictly speaking beta1 is the feature freeze point, but for a feature as significant as this I think @romen is right, there would have to at least be an OTC decision to approve it. Partly it might depend on how invasive and extensive the changes end up being.

[baentsch]

Thanks for your willingness to help this along! Clearly understood there are no guarantees. Your caveats scare me a bit as you know much better what's involved, but I think we'd give it a shot (or two or three)... I implemented the OQS-CMS integration so have some idea what may be "lurking" there. Also, we managed to get OQS-x509 certs working -- and extended the cert type slots (but that wasn't me). So let's see whether we have enough collective insight to get this going in time. Thanks again for all your pointers and help (so far and possibly in the future)!

[levitte]

I got interested in the OQS provider, and wanted to see how hard it would be to turn it into an external provider.
It turns out it wasn't very difficult, it was a fairly simple file extraction, adding the proper CMake files, and a new variant of the signature tests.

I have the result in a personal repository, https://github.com/levitte/tmp-oqsprov
If that is of interest, feel free to clone and do whatever you like with it.

[baentsch]

Thank you very much! This also again confirms the utility of the effort @mattcaswell put into guiding us to "dis-entangle" internal APIs.

We'll probably put this into its own open-quantum-safe repo and maintain it there at latest when OpenSSL3 goes beta to immediately "PQC-enable" OSSL3: OK for you?

At first glance, I fail to see the KEM tests, though: Was there a reason to not bring those over or do I overlook them?

Final question: What'd be your recommendation as to where to maintain OQS provider code for "the long run"? Background: We'd eventually be interested in achieving all the same OQS-OSSL functionality in 3.0 that the 1.1.1 fork has (incl. certs, CMS, etc.). The present state of things means that we probably best maintain the provider in an OSSL3 fork until pluggable signatures are available and we can make do with a fully external provider and don't need "upstream patches" any more. The latter clearly would be our preference.

I guess the real question is: How soon do you see the latter (full provider support for keys, X509, signatures, ...) come to pass? @mattcaswell already stated the OTC decision to not rush this before 3.0 release; but maybe there's a guesstimate as to the timeline you'd (all together) be looking at for this? In turn, would you'd consider work on our side to replicate in OSSL3 core what we did in 1.1.1 to be futile and quickly superfluous as full provider-enablement/"legacy-disablement" is your goal probably?

[mattcaswell]

Final question: What'd be your recommendation as to where to maintain OQS provider code for "the long run"? Background: We'd eventually be interested in achieving all the same OQS-OSSL functionality in 3.0 that the 1.1.1 fork has (incl. certs, CMS, etc.). The present state of things means that we probably best maintain the provider in an OSSL3 fork until pluggable signatures are available and we can make do with a fully external provider and don't need "upstream patches" any more. The latter clearly would be our preference.

If it was me, I would separate the provider from your 3.0 fork. The provider is useful standalone with standard OpenSSL 3.0. It gives you full integration to the "EVP" level crypto - so users could start using OQS algorithms in their code immediately just by dropping in the OQS provider without having to have a "custom" OpenSSL. Separate to that are the additional changes you need to OpenSSL to support full integration for things like TLS/CMS/whatever. Users that want that can download your fork. By having the two things separate it gives people a higher degree of flexibility depending on what they want to achieve.

How soon do you see the latter (full provider support for keys, X509, signatures, ...) come to pass?

I personally would like to see TLS pluggable signature support in 3.1. I expect and hope for X509 support in 3.0. I think the main thing preventing it at the moment is openssl#13893. Not sure what you mean by "keys" and "signatures" support...those things should be possible now. TLS pluggable signature support is of course missing, but you can do libcrypto signatures now. Not sure what is required for CMS.

a guesstimate as to the timeline you'd (all together) be looking at for this?

3.0 won't be out until later this year. No timeline has yet been discussed for 3.1, but I'd be surprised if it was quicker than a year after the 3.0 release so we're talking 2022 at the earliest.

In turn, would you'd consider work on our side to replicate in OSSL3 core what we did in 1.1.1 to be futile and quickly superfluous as full provider-enablement/"legacy-disablement" is your goal probably?

It would be nice if any work you did could be used as the basis for adding the missing features upstream once we start accepting patches for 3.1, e.g. rather than just hacking in OQS TLS signature support - actually adding pluggable signatures which could be used as the basis for that feature in mainline OpenSSL.

[romen]

@mattcaswell ninjaed me: I was writing exactly the same recommendations!

[levitte]

If it was me, I would separate the provider from your 3.0 fork. The provider is useful standalone with standard OpenSSL 3.0. It gives you full integration to the "EVP" level crypto - so users could start using OQS algorithms in their code immediately just by dropping in the OQS provider without having to have a "custom" OpenSSL.

If it were me, I'd go one step further and have the oqsprovider as a separately installable package... that's where I was going with what I did. Then, applications would only have to load it and use it. I don't see why it would need to be dropped into another application's source code (I assume that's what @mattcaswell meant).

[mattcaswell]

I don't see why it would need to be dropped into another application's source code (I assume that's what @mattcaswell meant).

No. I meant what you said.

[levitte]

At first glance, I fail to see the KEM tests, though: Was there a reason to not bring those over or do I overlook them?

Er, KEM tests? In test/oqs_test.c, I see group tests and signature tests, is one of those the KEM tests?
I avoided the groups test, because it depends too heavily on an OpenSSL internal test helper, and it was simply too much of an effort to refactor that for a demo that I intended to take max half a day.

[levitte]

I guess the real question is: How soon do you see the latter (full provider support for keys, X509, signatures, ...) come to pass?

Keys have support, at least in principle. There are corners where we haven't quite fulfilled that yet, but provider backed are meant to be possible to use (almost) everywhere, and transparently.
(I say almost, because there are spots where we can only support legacy EVP_PKEYs)

X509 have some corners that we need to look at.

"signatures"... I'm honestly mystified by this, because you already have that, per se. I assume it's this "pluggable signatures", which I've honestly not dived into, that you're talking about.

[baentsch]

In test/oqs_test.c, I see group tests and signature tests, is one of those the KEM tests?

Yes: That one's testing all OQS-KEMs.

it depends too heavily on an OpenSSL internal test helper

ACK: create_ssl_objects and create_ssl_connection -- should be possible to create an "externalized" version. Will do that.

I say almost, because there are spots where we can only support legacy EVP_PKEYs

I know -- I already banged my head into/over those.

I assume it's this "pluggable signatures", which I've honestly not dived into, that you're talking about.

Yes.

[baentsch]

Thanks for this rather unanimous vote, @mattcaswell @romen @levitte : We'll then move this work to an OSSL-external project. Should also make work on a provider-encoder/decoder "strictly separate" from the start.

@dstebila Would you please create an open-quantum-safe/oqs-provider repo? I'll manage it, clone https://github.com/levitte/tmp-oqsprov/ into it, add templating and KEM testing as per the above as well as CI-enablement, too (this one without docker auth to also allow non-OQS-core PRs).

[dstebila]

@dstebila Would you please create an open-quantum-safe/oqs-provider repo? I'll manage it, clone https://github.com/levitte/tmp-oqsprov/ into it, add templating and KEM testing as per the above as well as CI-enablement, too (this one without docker auth to also allow non-OQS-core PRs).

Done. Once there's a .circleci/config.yml I can enable the Circle CI integration as well.

[levitte]

@baentsch, I ended up adding a groups test too. It's a bit of a hack, as it requires copying stuff from the OpenSSL source, and has patches for ssltestlib.c and ssltestlib.h. It builds, though, but doesn't test properly.

[baentsch]

@levitte , much appreciated. I'll take it from there.

[baentsch]

All work items as per the above complete with open-quantum-safe/oqs-provider#1.

Further discussions regarding OQS-OpenSSL provider to occur in https://github.com/open-quantum-safe/oqs-provider/issues