Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add EVP/KDF API #6674

Closed
Closed

Conversation

@davidmakepeace
Copy link
Contributor

@davidmakepeace davidmakepeace commented Jul 9, 2018

This PR adds a new EVP/KDF API that collects all KDFs and PRFs under the one API.

#include <openssl/kdf.h>

Currently some KDFs are available directly (PBKDF2 and scrypt) while others are available through the PKEY API (scrypt, TLS1 PRF and HKDF) even though the PKEY API was originally intended for Public Key algorithms. Note that the low level KDF functions (PKCS5_PBKDF2_HMAC and EVP_PBE_scrypt) are defined in openssl/evp.h even though EVP is the high level interface.

The new API is modeled after the current KDF support under the PKEY API in order to minimize the work required to update applications to use the new API. Support is included for PBKDF2, scrypt, TLS1 PRF and HKDF. PBKDF1 and PKCS12 KDF are not implemented.

The low level KDF functions and PKEY KDF support have been changed to wrap the new API. Other parts of OpenSSL have not yet been updated to call the new API.

Documentation to follow.

  • documentation is added or updated
  • tests are added or updated
@davidmakepeace davidmakepeace changed the title WIP Add EVP/KDF API WIP: Add EVP/KDF API Jul 9, 2018
@davidmakepeace davidmakepeace force-pushed the add-evp-kdf-api branch 2 times, most recently from a4b5940 to 1615c66 Jul 10, 2018
include/openssl/evp_kdf.h Outdated Show resolved Hide resolved
@davidmakepeace davidmakepeace force-pushed the add-evp-kdf-api branch 4 times, most recently from 868ad00 to 2a1efeb Jul 13, 2018
@mattcaswell mattcaswell added this to the Post 1.1.1 milestone Jul 13, 2018
@paulidale
Copy link
Contributor

@paulidale paulidale commented Jul 24, 2018

We've got a single step KDF (SP 800-56) implemented and passing test vectors but it will need some cleanup to fit into this framework. The intention is to submit this it after this PR has been accepted or at least after the final form of these changes are determined.

@paulidale
Copy link
Contributor

@paulidale paulidale commented Jul 26, 2018

Thought from today: this interface should deal with NULL result buffers better. Some of the algorithms use this to request the size of the result. The rest are variably sized and don't support this functionality. Uniformity would be better here.

My suggestion is that a new API is introduced that requests the size of the result (which matches digests) and that this returns 0 or -1 for variable size outcomes (I favour zero). Then, the NULL checking can be removed from the new APIs and handed off to the legacy wrappers.

Thoughts?

@t-j-h
Copy link
Member

@t-j-h t-j-h commented Aug 27, 2018

Do we really need the uint64 in the scrypt interface?

@levitte
Copy link
Member

@levitte levitte commented Aug 27, 2018

The scrypt page on tarsnap seems to indicate that at least N can be a large enough integer to warrant 64 bits...

@t-j-h
Copy link
Member

@t-j-h t-j-h commented Aug 27, 2018

Sure it can technically go that high - but pragmatically would anyone ever exceed 2^32 - that I seriously doubt is realistic in any context

@levitte
Copy link
Member

@levitte levitte commented Sep 20, 2018

@paulidale, it would be great if you could rebase and fix the conflicts.
(re crypto/err/openssl.txt, crypto/evp/evp_err.c, include/openssl/evperr.h and util/libcrypto.num, I would just toss them and do make update again)

crypto/evp/evp_kdf.c Outdated Show resolved Hide resolved
include/openssl/evp_kdf.h Outdated Show resolved Hide resolved
@paulidale
Copy link
Contributor

@paulidale paulidale commented Sep 20, 2018

I'll ask @davidmakepeace to update tomorrow.

@levitte
Copy link
Member

@levitte levitte commented Sep 20, 2018

Oh oops, I mixed you guys up... sorry :-/

@paulidale
Copy link
Contributor

@paulidale paulidale commented Sep 20, 2018

I'm fatter and uglier. I figured it would have been hard to mix us up :)

crypto/evp/pkey_kdf.c Outdated Show resolved Hide resolved
@simo5
Copy link
Contributor

@simo5 simo5 commented Sep 20, 2018

@davidmakepeace any chance you will rebase this PR soonish, it's been indicated as a base to be used to implement SSHKDF and it'd be nice to start on a fresh rebase that does not conflict with master

@davidmakepeace davidmakepeace force-pushed the add-evp-kdf-api branch 2 times, most recently from cd71808 to bfa7eaa Sep 21, 2018
include/openssl/evp_kdf.h Outdated Show resolved Hide resolved
Copy link
Contributor

@paulidale paulidale left a comment

Approved once the two nits are addressed (both should be non-code impacting) and the merge conflict resolved (ditto, it is the changes file).

@paulidale
Copy link
Contributor

@paulidale paulidale commented Feb 12, 2019

The build failures are unrelated to this.

Copy link
Contributor

@paulidale paulidale left a comment

Reaffirmed after nit fixes and rebase.

Changed PKEY/KDF API to call the new API.
Added wrappers for PKCS5_PBKDF2_HMAC() and EVP_PBE_scrypt() to call the new EVP KDF APIs.
Documentation updated.
@davidmakepeace
Copy link
Contributor Author

@davidmakepeace davidmakepeace commented Feb 13, 2019

I just rebased, now that master has been fixed.
We should get a clean build now.

Copy link
Member

@levitte levitte left a comment

Reaffirmed

@levitte
Copy link
Member

@levitte levitte commented Feb 13, 2019

Merged.

5a285ad Added new EVP/KDF API. Changed PKEY/KDF API to call the new API. Added wrappers for PKCS5_PBKDF2_HMAC() and EVP_PBE_scrypt() to call the new EVP KDF APIs. Documentation updated.

@levitte levitte closed this Feb 13, 2019
@levitte
Copy link
Member

@levitte levitte commented Feb 13, 2019

This was great work, @davidmakepeace

levitte added a commit that referenced this issue Feb 13, 2019
Changed PKEY/KDF API to call the new API.
Added wrappers for PKCS5_PBKDF2_HMAC() and EVP_PBE_scrypt() to call the new EVP KDF APIs.
Documentation updated.

Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from #6674)
@paulidale
Copy link
Contributor

@paulidale paulidale commented Feb 13, 2019

219 days in coming. A great effort and a very worthwhile contribution. Thanks to all who assisted and contributed.

The various follow on KDFs can now be tapped onto this work. @slontis has single step KDF, @romen has one too I believe. No doubt there are others. Let the floodgates open to the pressure of the 617th.

@levitte
Copy link
Member

@levitte levitte commented Feb 13, 2019

This search shows all the KDF related PRs. I'll have a quick run through, as I'm sure some can just be closed by now

@davidmakepeace davidmakepeace deleted the add-evp-kdf-api branch Feb 18, 2019
@tomato42
Copy link
Contributor

@tomato42 tomato42 commented Apr 15, 2019

I must be missing something, but I don't see a way to run PBKDF2 using command line tools (see #4021), just the TLS KDF, HKDF and scrypt...

@mattcaswell
Copy link
Member

@mattcaswell mattcaswell commented Apr 15, 2019

I must be missing something, but I don't see a way to run PBKDF2 using command line tools (see #4021), just the TLS KDF, HKDF and scrypt...

It seems that PBKDF2 does not appear in crypto/evp/pkey_kdf.c like the others do. @davidmakepeace is this an accidental omission or was there a reason for that?

@t8m
Copy link
Member

@t8m t8m commented Apr 15, 2019

I suppose only KDF algorithms that were previously implemented in the PKEY interface were ported via wrapping the KDF API. Anything newly implemented (PBKDF2, SSHKDF, ...) does not have this wrapping as the PKEY interface should be seen as legacy for kdfs?

@mattcaswell
Copy link
Member

@mattcaswell mattcaswell commented Apr 15, 2019

I suppose only KDF algorithms that were previously implemented in the PKEY interface were ported via wrapping the KDF API. Anything newly implemented (PBKDF2, SSHKDF, ...) does not have this wrapping as the PKEY interface should be seen as legacy for kdfs?

Yes, perhaps that is the reason. That has the unfortunate side-effect that they're not available to pkeyutl. That might suggest that pkeyutl should be ported to use the new API - which is itself problematic since the whole structure of pkeyutl is focused around handling EVP_PKEY objects (hence the name of the application). Or alternatively we decide to retrofit the PKEY interface for these "new" KDFs.

@levitte
Copy link
Member

@levitte levitte commented Apr 15, 2019

Didn't someone submit a PR to create a `openssl kdf' command? I think I remember seeing that, but can't find it right now

@davidmakepeace
Copy link
Contributor Author

@davidmakepeace davidmakepeace commented Apr 15, 2019

Yes, the PBKDF2 algorithm was not previously supported by the PKEY interface, which is now considered legacy with respect to KDF algorithms.
The intention was to eventually change everything that was using PKEY for KDFs to using the KDF API but there is no hurry due to the wrappers.

We could change the pkeyutl app to use the KDF API for KDFs. Alternatively a new kdf app could be added specifically for KDFs.

@mattcaswell
Copy link
Member

@mattcaswell mattcaswell commented Apr 15, 2019

A KDF app seems like a good option to me.

@slontis
Copy link
Contributor

@slontis slontis commented Apr 16, 2019

I had added a mac app, but not a kdf one.
I will submit a PR for this very soon.
Not sure that it is a good idea to use a utility like this to do anything other than do testing.
Deriving a key via a command-line tool seems a bit shady to me..

@tomato42
Copy link
Contributor

@tomato42 tomato42 commented Apr 16, 2019

there are two purposes, primary is to verify if a different implementation implements the KDF correctly[1], and secondary is for tools like https://testssl.sh/

so yes, diagnostics/debugging

1 - including a situation where the KDF is implemented in an engine or pkcs11 module

@slontis
Copy link
Contributor

@slontis slontis commented Apr 16, 2019

Added KDF app PR: #8762

@slontis
Copy link
Contributor

@slontis slontis commented Apr 22, 2019

Added EVP_KDF PR: #8808

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet