OopenSSL 1.1.0 compatibility

[Jakuje]

Update 2017-11-14: Resolves #104. Tested with latest Yubikey 4 with hardware tests, provisioning new key and selfsigning certificates.

[klali]

Have you tested if this works? The problem with X509_REQ_digest() is (used to be?) that it hashes the entire X509_REQ, ideally I guess there would be a X509_REQ_INFO_digest().

[Jakuje]

I didn't test this part, but I am not sure if there is an interface in the new OpenSSL to get the req_info from the structure. I will remove this chunk for now.
I planned to do all the changes, but some of the certificate handling was more complex than I was familiar with. Currently dumping what is available for others to have possibility to continue or I will finish that later.

[klali]

The approach of doing this with a compat layer is probably the only way to go ahead. I haven't looked it over to see how many of those would need to be added.
There is no direct plan or schedule for this work.

[Jakuje]

Thank you for prompt comment. There are really only few more changes, but they are probably harder to address than this part in this commit.
If there is no plan yet from Yubico, I will be hopefully able to continue on that (or somebody else), since many distributions are moving to openssl 1.1.0

[Jakuje]

I added few more changes I was able to handle. The embedded tests work except the hardware ones (failing even before the change).

What is missing is working with {X509,X509_REQ}_digest() and tweaking of {x509,req}->signature->flags and {req,x509}->{cert_info,sig_alg}->{parameter,algorithm} on several places in the code.

I believe that is a good start, but I don't have enough internal knowledge to test further changes to make sure they will not break some use cases. I will build the package against old OpenSSL so far, but having this fixed in future would be very nice.

[Jakuje]

I was tweaking with it little bit more added the straight-forward parts, but the certificates are still little bit more complicated.

From what I see, with opaque structures, it is not possible to do exactly what you were doing in functions request_certificate() and selfsign_certificate() and do_create_empty_cert(), such as modifying algorithms, writing signatures directly into X509_REQ and X509 structures and so on.

On the other hand, the X509_REQ_sign() and X509_sign() should get used to achieve the same using high-level interface, while the signature should be provided by using RSA_method and EC_KEY_METHOD structures to pass the signature request to the card itself.

I am pushing a WIP status, that builds with OpenSSL 1.1.0, but for older is still using the same approach. Unfortunately, currently I do not have any capable yubikey at hand to test the actual functionality, so the last commit is for further review/testing, but the idea should be clear.

[Jakuje]

Finally my new Yubikey arrived so before setting it up for myself I finished with this PR and made even the HW tests pass with OpenSSL 1.1.0 and hopefully also with the old version (including the rebase to the current master).

[mrmekon]

@Jakuje I updated this to work with the recent refactoring: PR #131