Support TLS 1.3 post handshake auth in trio.ssl

[njsmith]

This was recently added to the stdlib ssl APIs (in 3.8-dev, and also backported to the other -dev branches), so it should be a fairly straightforward matter of wrapping the new APIs: python/cpython#9460

One thing I'm not clear on is how you trigger the post-handshake auth cycle if you don't want to commit to writing something immediately. (e.g., because what you decide to write will depend on the success/failure of the authentication.) Maybe you call do_handshake again? (If so then we need to provide a way to call do_handshake a second time!) Or does failure kill the connection, so it doesn't matter?

[njsmith]

See also: https://bugs.python.org/issue34670#msg327932

[njsmith]

The API for this should be considered alongside #198, which is about the way renegotiation/post-handshake-auth create nasty issues for SSLStream cancellation. In particular, normally for trio streams, cancelling send_all leaves the stream in an indeterminate (basically unusable) state, but cancelling receive_some is safe. But renegotiation breaks that because it means that SSLStream.receive_some has to call transport_stream.send_all, which means cancelling SSLStream.receive_some cancels transport_stream.send_all, which means from the PoV of SSLStream's, user, cancelling receive_some can leave the stream in an indeterminate (basically unusable) state.

Also, this appears to be the only openssl man page that discusses post-handshake auth: https://www.openssl.org/docs/man1.1.1/man3/SSL_CTX_set_verify.html

Anyway, on further investigation, I suspect the way the ssl module's post-handshake auth API works is:

• sslobj.verify_client_post_handshake actually sends the CertificateRequest message immediately, without waiting for sslobj.send to be called (this contradicts the Python docs)
• when a CertificateRequest is received, openssl immediately does its normal pick-a-cert-to-send-and-send-it logic, which happens synchronously, and the ssl module doesn't really expose any way for us to affect this process
• after sending a CertificateRequest, openssl doesn't actually require that the next data to arrive contain the certificate data. RFC 8446 is very clear about this: "Because client authentication could involve prompting the user, servers MUST be prepared for some delay, including receiving an arbitrary number of other messages between sending the CertificateRequest and receiving a response." Again, this contradicts the Python documentation, but I'm hoping that's because the Python documentation is wrong.
• so, if we keep reading tls data from the peer, then eventually a new cert will arrive, or else we'll get a message from the client saying that it isn't going to send a new cert, but other things might arrive while we're waiting
• The most common (only?) use case for this is in HTTP, when the server wants to negotiate a client cert in between reading the request and sending the response. This means that after sending the CertificateRequest, the server wants to read until it gets a new cert, but not wait for new application-level data to arrive. I don't see any way to do this in Python if you're using the blocking API, which I think means the current API is effectively not usable at all in blocking mode? We should probably point this out to @tiran. In non-blocking mode, we should be able to kluge around it sometimes, in the cases where the client actually does send us a cert, because each time new data arrives on the underlying transport layer we can feed it to openssl and then check whether the cert state has changed. But if the client sends us a message saying that they won't be sending us a cert, I don't think we can detect that?

I'm not certain of anything I just said, so it should all be checked.

If it is correct, then I guess we want to do something like:

Client side: enabling post-handshake auth is generally an opt-in thing in TLS 1.3 and in the openssl and Python stdlib APIs, so it can/should be opt-in for us too. This is a flag you set in the context, saying "yep, I'm prepared to handle post-handshake auth if the server requests it" (which then sets an extension flag in the handshake, etc.)

Then when the server actually sends a post-handshake auth request: openssl/the stdlib don't provide any way for the client to examine the request and handle it in an async way. So, I guess we're just not going to support that, which means that SSLStream.receive_some will still sometimes call transport_stream.send_all, which means we have the cancellation issues. But! There is an exciting change, which is that now this is opt-in, so only people who have explicitly signed up to cope with tricky receive_some semantics have to worry about it!

We should strongly consider making renegotiation support opt-in as well. This is what BoringSSL and golang do, and it would help a lot with #198. It would also increase consistency between TLS 1.3 and TLS <1.3 – can we just have a single flag that controls both renegotiation support and post-handshake auth support?

Important thing to check: TLS 1.3 also supports "rekeying". The basic form of rekeying is harmless: it just lets one side say "everything I send after this will be encrypted with a new key", so it doesn't trigger any weird send/receive interlocking by itself. BUT, a rekey message can also request that the peer also rekey. The way this is intended to work is, when we receive this message, we don't have to do anything except an internal "rekey requested" flag, and then the next time we're going to send some application data, at that point we do a rekey and send both the rekey message and the application data (encrypted with the new key) together. So in theory this shouldn't cause any problems with SSLStream.receive_some → transport_stream.send_all. But we need to double-check that openssl handles this correctly, and doesn't try to "simplify" things by immediately sending a rekey message when it receives a rekey request. Or if it does, and we're in the don't-send-from-receive mode, we can work around that by simply queueing the data to be sent later.

Server side: The server needs a way to send a post-auth handshake request. Something like await ssl_stream.verify_client_post_handshake(). However, this can only send the request – it cannot wait for the response, because there might be arbitrary other data coming in. So then we need a way to keep receiving data, while also getting notification if/when a client actually sends us a cert.

I guess the simplest way to do this is: declare that after calling verify_client_post_handshake, it becomes legal for receive_some to return some kind of special PostHandshakeAuthCompleted object. In principle, this object should indicate (a) whether or not a new certificate was actually received, (b) some kind of identifier to let you match it up to a previous verify_client_post_handshake call. In practice, I suspect openssl doesn't actually let us detect when a new certificate isn't received, and doesn't let us match up responses to methods. So for now, maybe this object won't carry any data, and you figure out what the new cert is by introspecting the connection, and if we ever add support for having multiple CertificateRequests outstanding at the same time we can add a some kind of id field. ...Yeah, I think that would be OK. So we can do the simple thing now, and it won't paint us into a corner.

Server-side opt-in is a bit different from client-side opt-in. On the client, you have to include the extension right in the initial handshake, and then once you've set the flag, you have to be prepared to handle the CertificateRequest message at any moment, which changes the semantics of receive_some for the duration of the connection. So it's definitely an up-front opt-in thing.

On the server, there is an opt-in flag called SSL_VERIFY_POST_HANDSHAKE. IIUC its semantics are to disable client cert requests during the handshake, saying that we prefer to always use post-handshake auth whenever possible. I'm not sure what this does when the client is using TLS 1.3 but doesn't have the post-handshake auth extension enabled. I'm not sure why this is ever useful.

Warning: the stdlib uses a single flag that on the client means "opt-in to supporting post-handshake auth", and on the server means "don't request the client cert during the handshake". These are pretty different.

Anyway, so on the server you probably don't care about setting that opt-in flag. What actually matters is whether you call verify_client_post_handshake. If you don't, then you definitely don't have to deal with PHA; if you do, then you've opted in to receive_some returning weird sentinel values.

This makes interaction between PHA and renegotiation more complicated on the server-side.

If you don't want to deal with receive_some having weird cancellation semantics, then we still need to make renegotiation opt-in on the server side.

Also, since the ssl module doesn't expose any way to request a renegotiation, trio servers – and Python servers in general using the ssl module – don't currently have any way to request a client cert in the middle of a connection. The addition of verify_client_post_handshake makes it possible to support this feature on TLS 1.3, but it remains impossible to support on TLS <1.3. Given that, maybe we should just not worry about supporting PHA on the server side for now?

So....... I think that means:

• On client: make renegotiation and PHA opt-in, and toggled by the same flag.
• On server: make renegotiation opt-in (or maybe disable it unconditionally?), don't support PHA for now
• Run a bunch of tests to make sure

To actually implement opt-in renegotiation/PHA: we want to enable/disable OP_NO_RENEGOTIATION when it's available (depends on Python and openssl version). On the client, we want to enable/disable PHA when it's available (depends on Python and openssl version). On configurations where we can't do the enable/disable at the openssl level, we want to implement it directly inside Trio, by making receive_some error out if it finds data to send, and likewise for send_all I guess.

Together, I think this means that it needs to be an option passed to the SSLStream constructor (like https_compatible), and that we will have to mutate (!) the passed-in SSLContext object. Which is unfortunate, but I don't see any way around it.

---

Another confusing thing: I don't understand what this issue is about at all: openssl/openssl#6933. It sounds like the issue is that when openssl clients were enabling PHA by default, then this caused problems in the Python test suite because suddenly the client cert wasn't available on the server side after handshake. But AFAICT, that should only happen if the server was setting SSL_VERIFY_POST_HANDSHAKE, which doesn't appear to have ever been enabled by default. And also if the server does set SSL_VERIFY_POST_HANDSHAKE, then AFAICT openssl doesn't actually care whether the client has enabled PHA; if the client speaks TLS 1.3, then it skips requesting the client cert (Ref). (Maybe this is an openssl bug?)

[pquentin]

The most common (only?) use case for this is in HTTP

http://www.watersprings.org/pub/id/draft-sullivan-tls-post-handshake-auth-00.html#rfc.section.1 mentions those use cases:

• servers that have the ability to serve requests from multiple domains over the same connection but do not have a certificate that is simultaneously authoritative for all of them
• servers that have resources that require client authentication to access and need to request client authentication after the connection has started
• clients that want to assert their identity to a server after a connection has been established
• clients that want a server to re-prove ownership of their private key during a connection
• clients that wish to ask a server to authenticate for a new domain not covered by the certificate included in the initial handshake

[jab]

(From a past life where I was following this more closely,) that first bullet point in particular is a major use case for censorship circumvention through a CDN. (Note that Nick Sullivan (co-author of that post-handshake auth draft) works for Cloudflare, which stopped turning a blind eye to domain fronting a while ago.)