libp11 does not support ECDH key derivation is OpenSSL versions < 1.1 #49
Comments
|
While that may be a feature for the sake of completeness, ECDH is not really used by real world TLS. Its deployment is compares with static DH (i.e. 0). So I'd expect if anyone is interested on that feature for a particular application to contribute the code rather than adding code just for the sake of being "complete" in supported protocols. |
http://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-56Ar2.pdf $ openssl pkeyutl -engine pkcs11 -peerform engine -derive -inkey 384priv.pem -peerkey id_03 -hexdump Using reader with a card: Yubico Yubikey 4 OTP+U2F+CCID Douglas E. Engert DEEngert@gmail.com |
OpenSC would also have to be rewritten? Because it seems to support ECDH1-COFACTOR-DERIVE as is: P.S. Happy New Year, guys! |
Extracting public key from the token... Generating ephemeral ECC key pair on secp384r1... Generating random 250 bytes of Base64-encoded data... Deriving shared key from ephemeral private and token public keys... Encrypting data file with derived symmetric key in /tmp/derive.31436.shared1 and AES-CFB... Deriving shared symmetric key on the token, using ephemeral public key... Decrypting data file with derived symmetric key in /tmp/derive.31436.shared2 and AES-CFB... Derived keys matched. Douglas E. Engert DEEngert@gmail.com |
engine "pkcs11" set. 0 warnings, 0 errors. Exception Type: EXC_BAD_ACCESS (SIGSEGV) VM Regions Near 0x1021d1fc0: Thread 0 Crashed:: Dispatch queue: com.apple.main-thread Douglas E. Engert DEEngert@gmail.com |
|
This method does not seem to do anything but calling PKCS11_ecdsa_method_free(): And this method does not seem to be our likely culprit, because it does not take any parameters (that could unexpectedly be NULL), and checks for I don't know... All I can say is that adding a simple check for a NULL-pointer where I added it, alleviates the problem. Of course, if the experts in the code can find the exact |
static int pkcs11_engine_destroy(ENGINE * e) #ifndef OPENSSL_NO_EC } { Douglas E. Engert DEEngert@gmail.com |
|
Note that OpenSSL 1.1 combines ECDSA_METHOD and ECDH_METHOD into EC_KEY_METHOD Adding ECDH could wait till the cleanup of libp11 and engine is complete. 1.1 also has some other unexpected changes, like moving struct x509_st from x509.h to internal/x509_int.h |
No objections from me, especially since this cleanup work seems to have already begun. |
|
What are the practical applications for using ECDH key derivation instead of ECDHE? |
Off-hand, first thing that comes to mind: ability to send (and receive!) encrypted email, in a way similar to how it is done with RSA. Except that in this case the sender generates a random ECC key pair instead of a random symmetric key (for obvious reasons). But the receiver only needs to publish his ECC "encryption" (or rather "derivation") key, no negotiation (and round-trips) involved. |
|
RFC 5753 discribes how EC is used with a CMS that is used in E-mail. See the URLs earlier in this discusion. Sections 3 and 3.1 discribe how (method C(1, 1, ECC CDH)) in 800-56 can be used in CMS. The engine and/or the libp11 do not need to generate or do any crypto with the ephemeral EC private key. The engine and/or libp11 only need to support ECC CDH when using the sendor's and/or receivers's EC private key on a card. At the requset of a NIST developer, who was working on getting patches in Mozilla NSS to do ECDH: His response was: His Mozilla patches and my OpenSC patches were then added to NSS and OpenSC. |
Correct (I did not imply they would).
Correct. Receiver's key for decrypting, and sender's key for signing. |
Indeed. Thank you. |
|
#60 implemented this feature as a side effect of porting to OpenSSL 1.1.0. |
|
I assumed that it will. @dengert is the right person to answer this question, as he currently develops EC functionality. I currently spend most of my time on the engine_pkcs11 release planned for the next week. The next step, planned for the beginning of February, will be merging engine_pkcs11 into libp11. |
|
I would say no. but... OpenSSL-1.0.2 only added the functions need by an enging to hook the ECDSA_METHOD. They did not add the hooks for ECDH_METHOD. the "but..." is with access to OpenSSL source and internal OpenSSL header files it might be possible to hook the ECDH_METHOD. In 1.1 OpenSSL combined the ECDSA_METHOD and ECDH_METHOD into one EC_KEY_METHOD OpenSSL never exposed the internal strucuture of EC routines, and does not in 1.1 either. OpenSC code prior to the recent changes for 1.1 had a way to compile engine using the internal OpenSSL header files. This same approach could be used again for 1.0.2 only to access ECDH. I don't even want to look at the 1.0.2 code to see if could be done untill @mtrojnar combines engine and libp11, and 1.1 is working. There are just to many changes going on. And even then any 1.0.2 ECDH changes would be not standard changes that no distro would even consider implementing. Even if you got the OpenSC side working, there may still be issues with OpenSSL 1.0.2 using ECDH. You have been on the OPenSSL dev list, and look at the responses you have gotten about changes to 1.0.2. You might have to have local code changes to OpenSSL too. |
What exactly do you mean by "hooks for ECDH_METHOD"? int ENGINE_set_ECDH(ENGINE *e, const ECDH_METHOD *ecdh_meth);
int ENGINE_set_ECDSA(ENGINE *e, const ECDSA_METHOD *ecdsa_meth);
I completely fail to understand how my work on merging engine_pkcs11 into lib11 affects your work on EC. There will still be two separate libraries. The changes in engine_pkcs11 are just 1 (one) line of code, namely, the invocation of
Could you elaborate on the required changes to OpenSSL 1.0.2? Some links to the related emails in the OpenSSL mailing list archive would be sufficient... |
|
@mouse07410, do you have output showing how the compilicate failed? @mtrojnar you said: I am interpreting your work to merge engine_pkcs11 and libp11 to be they will end up in one package. PKCS11_get_ecdsa_method is only used with 1.0.2. PKCS11_get_ec_key_method is only used with 1.1 and above. Even though the functions start with PKCS11_ they are only intended to be used from the engine. If there is only one package, the package will be built with one version of OpenSSL. |
This will indeed be one package, just like OpenSSL is one package. Also, just like the OpenSSL produces two separate libraries: libcrypto and libssl, libp11 will also produce two libraries: libp11 (installed with other libraries) and libpkcs11 (installed with other OpenSSL engines). Just like in OpenSSL, there is absolutely no impact on the exported symbols: libcrypto still needs to export symbols only intended for libssl. |
Apologies - please see the next comment.
Yes of course. You really should have more faith in me. :-) But to bring hard evidence: If the above is not conclusive - please let me know what kind of evidence you'd like. |
|
@mtrojnar @dengert here's the |
|
As expected, 1.0.2 will fail to use an engine to to ECDH. I have told you that libp11 and opensc_engine when using OpenSSL 1.0.2 does not support engine use of ECDH because OpenSSL does not export a way to set the hooks needed for the engine to get control to do ECDH. Also as I said I did not want to look at this problem untill at least libp11 and opensc_engine were working with 1.1. I have started looking at what it would take to get then working with 1.0.2. |
:-(
Being unfamiliar with how exactly the interface between OpenSSL and libp11/engine_pkcs11 works, I can't properly appreciate this info. And looking at the Alexander Gostrer's changes, he basically uncommented
That is great - but mostly useless for the majority of users for (probably) a long period of time. Because now there are two kinds of users: those who need the applications they use to support OpenSSL (aka production environment, Web servers, etc. etc.), and those who don't. The first group is likely to be stuck with 1.0.2 for a while, so whatever good 1.1 brings up won't help them until the mainstream moves to 1.1. And the second group could just use OpenSC-based key derivation and be done with it (it works, as you can see above). I estimate the number/amount of those who can switch their production environment to 1.1 soon (within a few months) as nil to negligible.
Yes. But I personally think that the priorities should be reversed - because you're working first on something that isn't likely to be usable for some time, and postponing something that could have value right now. (Also, if what Alexander Gostrer did actually works - his changes to OpenSSL are literally miniscule, almost nothing. And we already have a working engine_pkcs11, so unlike him we don't need to write the entire engine from scratch.) |
|
Here are the patches to allow ECDH to work with OpenSSL-1.0.2. It is based on routines that work Testing has been done with 1.0.2d, and the cms decrypt command. I consider these patches to be a hack, and should not end up in the master branch of libp11 or engine_opensc. Libp11 needs to be compiled with the OpenSSL-1.0.2 internal source file crypto/ecdh/ech_locl.h See the commit messages for more information. |
|
Thank you! I will begin testing as soon as a working/build-able libp11 is pushed to Github (see #65 :-( |
|
First, thank you!
Of course. I didn't see any notice of 1.0.2 discontinuance impending any time soon, did you? :-) And when 1.0.2 disappears (probably around 2020) - all the code that relies on it would be either changed, or retired. But for the coming years I think it is safe to assume that these patches will be perfectly usable. I've applied and tested your patches against the latest OpenSSL_1_0_2-stable in Github (safe to assume it's 1.0.2f). It appears to work fine:
I am not sure I agree with such a harsh statement. 1.0.2 is going to be around, like it or not. ECDH capability should also be around. So in my opinion, as long as we care to be 1.0.2-compatible, it makes sense to go all the way, and support full functionality - including ECDH.
Yes, this is a disadvantage. I still hope that OpenSSL dev team might be persuaded to make the needed interface visible, to allow us using stock Doug, could you explain why you think that your perfectly working fix should not be in the master branch? P.S. OpenSSL Release Strategy says: |
|
Now - for the problem. ;) Before, pkeyutl failed to do ECDH key derivation on the token, but with openssl/openssl#557 could use public key on the token to EC-derive from a private key in a file. Now, thankfully pkeyutl successfully does ECDH derivation on the token - but somehow lost the ability to derive from a private key in a file and public key on the token: I checked, and the PR is in the openssl-1.0.2, so it doesn't seem like it was lost along the way. Any comments? Any place you'd recommend me to start looking at? Update For RSA there are two methods for each key pair: Are there two methods for ECDH-DERIVE, one that uses private key on the token, and the other one that uses public key? Because if not - then it would explain why ECDH-DERIVE works either with private key and fails with public, or works with public but fails with private. If the above diagnosis is correct, then for a one-time command line invocation (like what pkeyutl is doing) we could probably fill that Can we address it for an application that might want repeated operations, some with public key and some with private..? |
|
When I install the older code ( Re-installing the It would be great if we could keep this newly-acquired ECDH private key-based derivation, and fix the public key-based... Must add that all of the above works as shown with openssl-1.0.2f, and 1.0.2g-dev. Update ECDSA still works fine: |
|
Oh, and in case it matters, I'm getting these warnings compiling with the patches on: |
|
Does not really matter, they are warnings. But see update of the ecdh-1.0.2 patch |
|
@mtrojnar I would like to see this issue closed. The code for OpenSSL-1.1 has been added. If its not closed, at least rename it to: |
|
@dengert since 1.0.2 isn't going to disappear from the mainstream until at least the end of 2019, I don't think it is appropriate to close this issue simply because it may be resolved in the version that would become the mainstream who knows when. (I say "may" because the code has been added for 1.0.2 also. There's no reason that I see to expect that the code currently in 1.1 would work better than the one we have in 1.0.2.) Yes, I think it's fine to rename it the way suggested. P.S. I'd much appreciate if we could spend a bit more effort on finding the problem, rather than on what's the proper title for this issue. ;) |
|
What is "the problem"? OpenSSL is evolving, as is OpenSC, its just not at the rate you would like. I though the OpenSSL developers said to you they would not add new functionality to 1.0.2 which is what is needed to hook the method by external engines without using internal OpenSSL header files. I don't think OpenSC or libp11 should be trying to use internal OpenSSL header files in our releases. That will never end up in destros. So you are left with local source code changes and compiling yourself. I have given you the basis for this with the ECDH-1.0.2. mods. |
That currently I can get working either one of the following, but not both: or
And that is fine, especially since it's not in my power to affect in any way.
Yes (though it's still a question of whether exposing an existing interface without any changes to it means adding functionality :). But I'm perfectly OK to use that internal header file (luckily, only one seems necessary). I also hope that
Yes. That is perfectly OK.
Yes, thank you! Now I'm trying to cross that "last bridge" and make both kinds of derivation work (see above). It may be not possible - I don't know. In that case perhaps you could explain what makes it impossible... P.S. So far I've been unable to test openssl-1.1, because on Mac OS X I seem unable to provide the appropriate shared libraries path (yes I know about DYLD_LIBRARY_PATH :), and am unable to wipe my current openssl (1.0.2) and break everything else to install 1.1 over it for testing. :-( Has |
mtrojnar
changed the title
|
The best way to support ECDH key derivation in OpenSSL 1.0.2 would be to submit a pull request with the missing functions against OpenSSL 1.0.2. Adding a few missing functions to the API of a stable release should not break anything. In fact, it happened before. If the OpenSSL maintainers refuse to merge this pull request then I am okay with including a local definition of ECDH_METHOD for OpenSSL 1.2 in p11_ec.c. |
I concur. @dengert you know both codebases (your patches to libp11, and openssl) better than me, and probably better than @mtrojnar. Would you be agreeable to create such a PR?
Again, I concur. However, the following dialog from and Other Dev Team members neither confirmed nor denied the above statements. ;) |
|
ECDH_METHOD is definitely non-functional without API functions to instantiate it and access its data. I'm positive implementing these missing functions fixes a bug (design flaw?) rather than implements a new feature. |
I concur (and had stated that same thing in the past). The trick is convincing the OpenSSL Dev Team. ;) In the meanwhile, I'd really love to get ECDH-DERIVE completely fixed in An application (or a user) may want to: a. Derive ECDH key on the token - which did not work until you fixed it, thanks! or b. Derive ECDH key from the ephemeral private key it received or generated and the public key on the token. Here's the example of (a) (which I'm very grateful for fixing!): Here's the example of (b) (which got broken, probably because currently the code can't distinguish between where the private key for this operation is): The problem is - once the token gets involved/mentioned, the code automatically assumes that that's where the private key is going to be. Or at least tries to perform the operation as if that's where the private key was. With RSA, it does not make this mistake: when Decrypt is requested it goes to the token, but I can request Encrypt - in which case it just takes the public key from the token and uses it in software. @dengert, what is your opinion regarding how to address the above? Do/can we need to do something different within the method when it's |
|
In regards to pkeyutil, it the real world why would you have access to the token of the peer? The peer's pubkey is in the peer's certificate. In any protocol, the certificates or pubkey of an ephemeral EC key are what are sent to the other party. In your testing, you could simulate this be getting the peer certificate or pubkey in a separate step and pass it to pkeyutil as a file. All the OpenSSL utilities appear to be able to handle only a single engine at a time. As you point out there may be some confusion and any engine is considered usable for the private key operations. So you could submit a bug report to OpenSSL. Even if they could handle more then one engine at a time, could engine_pkcs11 and libp11 be invoked as two separate engines? PKCS#11 could have multiple tokens at the same time, but does this work correctly? It could also be a design issue. I believe engines were originally designed to replace crypto routines because one may not trust the OpenSSL implementation. So pkeyutil may be assuming that the availability of an engine that can do derive can be used. You need to look at the code. (You know I am retired, and am still on this project because of the PIV card and I wanted to see ECDH implemented in OpenSC and engine. Having ECDH in OpenSSL-1.1 satisfies that desire. I am not interested in strange ways to use pkeyutil and only marginally interested in any back ports of ECDH to 1.0.2.) |
|
@mtrojnar |
|
@dengert Could you please leave the ecdh-1.0.2 branch as it is? I'll be glad to merge your code after I finish the cleanup I'm working on right now. Thank you very much for your work on this feature! |
Good point.
Yes, that's what I've been doing. But there is certain elegance in not having to employ another package and another command-line tool. And everything else (RSA and ECDSA) already supports this kind of usage, so it would be nice to cover that "last mile". :-)
Do you happen to know whether this applies to applications linked with OpenSSL libraries? I want access to PKCS#11 tokens via engines work for "arbitrary" software linked with openssl libs, not only openssl CLI (dgst, pkeyutl, etc).
You see, the problem is that for RSA (sign/verify and encrypt/decrypt) and ECDSA (sign/verify) it works perfectly as is. I.e., I can sign using private key on the token, and verify using public key on the token without having to extract it first (same with RSA encrypt/decrypt). So I don't expect a favorable response from the OpenSSL team.
I don't think I understand - I thought that
I don't know - but I don't think so. Since there's no way to indicate which token (out of several available) you want, one would probably have to play with OpenSC configuration/parameters outside of OpenSSL. I haven't tried that (yet? as I have several tokens of three kinds).
Yes I do.
But you don't know if it works. And probably won't know for some considerable time.
Understood. |
|
@mtrojnar, request re-opening. Current/latest master branch from The previous version (manually merged @dengert's patches) worked: Please let me know what info you'd need to chase problem this down. P.S. It doesn't work the other way too, but as Doug pointed out, the following is not a practical scenario, so we can worry about it after the above is fixed: |
|
Fixed. Thank you. |
|
Thanks - it works now: |
Symptoms: trying to do encryption/decryption - which for ECC translates into shared key derivation - fails, because
p11_ops.c(and/orp11_ec.c) does not implement ECDH1-DERIVE (ECDH1-COFACTOR-DERIVE) method. It seems to only support ECDSA now.Example:
The problem is not with pkeyutl key derivation itself, because it works fine when the keys are in the filesystem:
The text was updated successfully, but these errors were encountered: