SslStream delayed client certificate negotiation

[Tratcher]

Background and Motivation

Related:
dotnet/aspnetcore#23948
#26210

Http servers like IIS, Http.Sys, HttpListener, etc. can conditionally request a client certificate after receiving the http request. Kestrel does not due to a lack of support in SslStream. Customers ask for this because they only want to require a client certificate for some pages/paths in their application.

Alternative strategies have proven too breaking for applications to adopt when moving from IIS to Kestrel.

Proposed API

namespace  System.Net.Security
{
    public class SslStream {

+   public Task<X509Certificate2?> GetClientCertificateAsync(CancellationToken cancellationToken)

No reads or writes would be allowed in parallel with this operation in order to avoid corrupting internal state.

Should successes and failures be cached? E.g. what happens if you call this API more than once per connection? What if a client cert was provided during the initial handshake?

What validation should happen on this certificate?

Risks

• POST Deadlocks: It's common for Http.Sys users of this feature to deadlock for POST request scenarios if the TCP window fills up with body data before the certificate is requested.
  • The app can mitigate this by draining the expected data before requesting the certificate.
  • 100-continue mechanics can also be used to delay the request body, but that is timing sensitive.
  • SslStream should fail the connection if any application data is received while the client certificate request is outstanding. That avoids all questions about buffering or parallelism.
  • What about HTTP pipelining? An app processing one request at a time wouldn't know about pipelined requests.
• In TLS 1.2 and lower the client certificate is requested by renegotiating the TLS context. This is called out as a security risk in the spec. Explore why.
• TLS 1.3 has removed the renegotiation feature and replaced it with a dedicated extension for requesting the client certificate. The same public API should work across protocols.
• HTTP/2 has explicitly banned renegotiation and the new TLS 1.3 extension due to head of line blocking. Kestrel will need to prevent this API from being called for HTTP/2 connections.
• HTTP/3: Edit: Quic requires TLS 1.3 and explicitly prohibits the new extension.

[ghost]

Tagging subscribers to this area: @dotnet/ncl, @vcsjones
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Background and Motivation

Related:
dotnet/aspnetcore#23948
#26210

Http servers like IIS, Http.Sys, HttpListener, etc. can conditionally request a client certificate after receiving the http request. Kestrel does not due to a lack of support in SslStream. Customers ask for this because they only want to require a client certificate for some pages/paths in their application.

Alternative strategies have proven too breaking for applications to adopt when moving from IIS to Kestrel.

Proposed API

namespace  System.Net.Security
{
    public class SslStream {

+   public Task<X509Certificate2?> GetClientCertificateAsync(CancellationToken cancellationToken)

No reads or writes would be allowed in parallel with this operation in order to avoid corrupting internal state.

Should successes and failures be cached? E.g. what happens if you call this API more than once per connection? What if a client cert was provided during the initial handshake?

What validation should happen on this certificate?

Risks

• POST Deadlocks: It's common for Http.Sys users of this feature to deadlock for POST request scenarios if the TCP window fills up with body data before the certificate is requested.
  • The app can mitigate this by draining the expected data before requesting the certificate.
  • 100-continue mechanics can also be used to delay the request body, but that is timing sensitive.
  • SslStream should fail the connection if any application data is received while the client certificate request is outstanding. That avoids all questions about buffering or parallelism.
  • What about HTTP pipelining? An app processing one request at a time wouldn't know about pipelined requests.
• In TLS 1.2 and lower the client certificate is requested by renegotiating the TLS context. This is called out as a security risk in the spec. Explore why.
• TLS 1.3 has removed the renegotiation feature and replaced it with a dedicated extension for requesting the client certificate. The same public API should work across protocols.
• HTTP/2 has explicitly banned renegotiation and the new TLS 1.3 extension due to head of line blocking. Kestrel will need to prevent this API from being called for HTTP/2 connections.
• HTTP/3: will the new TLS extension be allowed? Quic avoids most head of line issues.

Author:	Tratcher
Assignees:	-
Labels:	api-suggestion, area-System.Net.Security
Milestone:	-

[scalablecory]

No reads or writes would be allowed in parallel with this operation in order to avoid corrupting internal state.

This seems like a 99th percentile perf issue waiting to be discovered -- can you go into more detail on why this behavior is needed?

[scalablecory]

HTTP/3: will the new TLS extension be allowed? Quic avoids most head of line issues.

CC @nibanks any plans for MsQuic delayed client certs?

[nibanks]

Can you clarify what the ask is here?

[Tratcher]

No reads or writes would be allowed in parallel with this operation in order to avoid corrupting internal state.

This seems like a 99th percentile perf issue waiting to be discovered -- can you go into more detail on why this behavior is needed?

A) It's unclear if the TLS state would allow data to be encrypted or decrypted during a renegotiation since the ciphers could change, I'll defer to Tomas on that.
B) This avoids the deadlock issue discussed above.

[nibanks]

Ok. Read a bit more. No, we don't support that TLS extension, nor do we plan to expose that type of functionality in our API.

[Tratcher]

@nibanks Are there known blockers or is it a matter of priorities?

[nibanks]

TLS has to support it first. Then expose an API for it. Then we need to understand how it would would with QUIC (if at all; I don't know anything about the mentioned extension) and then we need to figure out how/if we can expose this in our API. And then there is the question of prioritization; which I'd expect this to fall fairly low on the list.

[Tratcher]

@nibanks nevermind, Quic (and HTTP/3) also prohibits the use of this extension.
https://quicwg.org/base-drafts/draft-ietf-quic-tls.html#section-4.4

A server MAY request that the client authenticate during the handshake. A server MAY refuse a connection if the client is unable to authenticate when requested. The requirements for client authentication vary based on application protocol and deployment.

A server MUST NOT use post-handshake client authentication (as defined in Section 4.6.2 of [TLS13]), because the multiplexing offered by QUIC prevents clients from correlating the certificate request with the application-level event that triggered it (see [HTTP2-TLS13]). More specifically, servers MUST NOT send post-handshake TLS CertificateRequest messages and clients MUST treat receipt of such messages as a connection error of type PROTOCOL_VIOLATION.

That limits the usefulness of this feature to HTTP/1.x (and non-HTTP protocols).

[wfurt]

It is part of base TLS 1.3 RFC https://tools.ietf.org/html/rfc8446#section-4.6 @nibanks. But as @Tratcher mentioned, it seems to be forbidden by QUICK.
I did some experiments and the client certificate comes in via EncryptedHandsahke message e.g. not in cleartext as usual. That may be interesting if privacy is concern regardless of the multiplexing.

This works with with TLS 1.3 - tested with openssl s_client and Windows Insider build. As long as client indicates willingneess via post_handshake_auth. Interestingly, neither Chrome, Safari nor HttpClient sets that. (by default?)
Perhaps HttpClient can do it if LocalCertificateSelectionCallback is set.

Setting RemoteCertificate impacts IsMutuallyAuthenticated. If we re-use (but we don't have to) handshake logic, standard verification logic would apply e.g. we would build a chain and if RemoteCertificateValidationCallback is set, it would be called as well. That may seems redundant but mimics current flow.

This generally seems like server feature. So on client, we would either throw or simply return RemoteCertificate. If we return RemoteCertificate, I would do the same on server e.g. if the certificate exist (was provided during handshake) then return it. Otherwise ask for it. This would effectively block option to ask multiple times, perhaps for different certificates. But I don't know if that is need or if IIS supports that. So I would probably go with one shot for simplicity unless there is indication that something else is needed.

cc: @blowdart @bartonjs @GrabYourPitchforks for any security concerns.

[karelz]

Triage: 6.0 until proven otherwise.

[wfurt]

I think this should be feasible with Windows and Linux. I have PoC for Windows with TLS 1.2 & 1.3 and Linux with TLS 1.2.
Since nobody from security group objected, I'm going to mark it as API review ready. Debate about behavior and error handling can continue as that does not impact API shape IMHO.

I think this should return existing remote certificate if that exist. e.g. it will not ask for yet another one during existing session and that will also cover cases when called on client.

If no remote certificate exist, SslStream will try to use means of given protocol version to obtain one.
It will return updated certificate if provided by client, null if new handshake happens and client does not provide one or it would throw exception if underling function report error in process of obtain new certificate. In that point, state of SslStream would be undefined. We can possibly make all function fail but in case of Tls 1.3 the attempt will fail with "clean" error if peer did not offer PHA and that does not impact state of the TLS e.g. the session is still usable without any problems.

Since this does not serve in handshake, the certificate will be returned without any validation. It would be up to the caller to do what ever needs to be done with it. That also prevents cases when SslStream tries to do too much and can for example block on resolving certificate chain.

[Tratcher]

I think this should return existing remote certificate if that exist. e.g. it will not ask for yet another one during existing session and that will also cover cases when called on client.

If no remote certificate exist, SslStream will try to use means of given protocol version to obtain one.

At most one client certificate negotiation should happen for the lifetime of an SslStream instance, even if a prior call returned null.

[bartonjs]

Video

• We changed "Get" to "Negotiate" to make it clear that complicated protocol stuff is happening.
• We removed the value from the return, the caller needs to just check the RemoteCertificate property again.
• Because we don't really want it to be called, we don't see the need for the synchronous version.
• It's virtual because all of the authenticate methods are virtual.
• Sounds like it'll need some SupportedOS attributes.

namespace System.Net.Security
{
    public partial class SslStream
    {
        public virtual Task NegotiateClientCertificateAsync(CancellationToken cancellationToken = default) => throw null;
    }
}

[geoffkizer]

Also, just to be clear... if the negotiation fails (i.e. no client cert) then does this throw? Or simply return and RemoteCertificate is still null?

[wfurt]

I don't think it needs to throw if everything end cleanly e.g. negotiation succeeds without certificate. We would throw if negotiation fails for whet ever reason.

[Tratcher]

At most one negotiation should happen. Subsequent calls to the API should return the original result (success or failure).

[wfurt]

Should we do validation and call RemoteCertificateValidationCallback on the certificate? Current code does it somewhat automatically on any renegotiation but we can prevent it if we want to. That would give caller more control as the validation can happen if needed outside of the SslStream.

[Tratcher]

What would happen if the cert failed validation?

• Would the SslStream abort?
• Would the cert still be available on the SslStream.RemoteCertificate?
• Would you be able to observe the failure without providing RemoteCertificateValidationCallback?
• Would NegotiateClientCertificateAsync throw?

[wfurt]

If validation fails, the NegotiateClientCertificateAsync would throw. I bump to this when writing tests and client did not return certificate. Then it would always throw without validation callback. The behavior for unknown/untrusted certificate would be similar.
I can make the certificate appear as SslStream.RemoteCertificate in either case. (validated or not)

[Tratcher]

If validation fails, the NegotiateClientCertificateAsync would throw.

But would the stream still be usable or would it be aborted?

If NegotiateClientCertificateAsync is canceled then the stream is aborted, correct?

Being inconsistent seems concerning. What if validation failures were only reported to RemoteCertificateValidationCallback? Or made available through a parallel property?

[Tratcher]

No reads or writes would be allowed in parallel with this operation in order to avoid corrupting internal state.

We realized there's a problem with this approach. Kestrel always has a loop running that reads from the SslStream and copies into a Pipe. While we wouldn't want the client to send any data during NegotiateClientCertificateAsync, there's no way to interrupt a pending read in order to call NegotiateClientCertificateAsync.

Would it be possible to allow the pending read to sit paused so long as no data was received during the NegotiateClientCertificateAsync call? Or would we need some way to cancel the read without aborting the SslStream?

[wfurt]

The underlying SslStream does not need to be aborted if renegotiation finished (or at least did not fail) even if the NegotiateClientCertificateAsync throws. I think it should be up to the call how they want to treat no or untrusted certificate.
For the cancellation, I would need to experiment more.
The NegotiateClientCertificateAsync will initiate the renegotiation, but that can be completed independently as long as the sides try to read and write to the stream. This can already happened today transparently if the renegotiation is requested by the remote peer.

And yes, I think we can accommodate the pending read. On Windows the renegotiate code call ReadAsyncInternal away to get the processing going. Perhaps I can adjust that and do it only if there is no pending read and guard agains changes .e.g. if there was no read, no new one would be allowed.

I have Windows implementation now and I'm working on tests to cover some of the scenarios we discussed. (hoping to get PR up today or early next week)
My goal was to release that early so we can experiment with Kestrel integration and make adjustments. Linux implementation would follow eventually.

[davidfowl]

We realized there's a problem with this approach. Kestrel always has a loop running that reads from the SslStream and copies into a Pipe. While we wouldn't want the client to send any data during NegotiateClientCertificateAsync, there's no way to interrupt a pending read in order to call NegotiateClientCertificateAsync.

We only do this with HTTP/2 which this feature is blocked for anyways right?

[halter73]

You're right @davidfowl . This would have been a problem back when we had a read loop in Kestrel's HttpsConnectionMiddleware, but I forgot that you changed it to use stream-to-pipe adapter a long time ago when @Tratcher and I were discussing it earlier.

[wfurt]

I'm wondering if there is harm in letting the read working. Seems finicky to build feature on latest Kestrel behavior.
As minimum, that could help draining the pipe if there is any lingering traffic from the peer.

[davidfowl]

I'm wondering if there is harm in letting the read working. Seems finicky to build feature on latest Kestrel behavior.
As minimum, that could help draining the pipe if there is any lingering traffic from the peer.

Nah, just make it fail.

[wfurt]

This is reminder of few things missed in initial #51905

Can we add more negative test cases? Specifically to test:

(1) If there's already-decrypted data in the buffer when Negotiate... is called, it should fail immediately
(2) If there's not, but there is a regular data frame already buffered, Negotiate... should fail immediately
(3) If there's nothing buffered, but the client sends a data frame instead of responding to the negotiate, then we should fail Negotiate... when that read completes

missing tests for 2 & 3. This will be impacted by refactoring unrelated to delayed certificate

We should add tests for early certificate with TLS 1.3 - at least on windows. There seems to be strange interaction as TLS 1.3 handshake completion use renegotiation path so as delayed certificate.

[Tratcher]

What happens if AllowRenegotiation is set to false and NegotiateClientCertificateAsync is called? Does it no-op, throw, or renegotiate?

AllowRenegotiation was added for HTTP/2 to prevent the remote party from initiating a renegotiation. Since APLN could result in the connection using HTTP/1.1 it might be OK to allow the server to initiate a renegotiation even if AllowRenegotiation is disabled. It would be the caller's responsibility to prevent that for HTTP/2.

[wfurt]

I'll need to test it but I think it will work on Linux and fail and throw on Windows.

We do copy the provided options so setting them after the stream is negotiated does nothing. There is currently no good way how to enforce the HTTTP/2 requirement AFAIK.

[wfurt]

Should be functional on Windows and Linux with current OpenSSL.
There is no support on macOS (and will not be as there is no API support).

2 remaining issues are tracked separately:

• SslStream delayed client certificate with TLS 1.3 on Linux #55757 (Linux with TLS 1.3 does not work)
• SslStream delayed client certificate with legacy OpenSSL #55761 (Linux with legacy OpenSSL 1.0.0 / 1.1.0 does not work)