Certificate Authentication rejected when TLS is terminated and Client certificate is forwarded

[NikMeyer]

Describe the bug

Client certificates are only validated in the CertificateAuthenticationHandler if the connection itself is using HTTPS (See Line 55). This behavior causes problems when the SSL connection is terminated at a load balancer and client certificates are forwarded via Headers. (See https://docs.microsoft.com/en-us/azure/app-service/app-service-web-configure-tls-mutual-auth)

This is especially a problem when combined with Azure Web App services running as containers https://docs.microsoft.com/en-us/azure/app-service/quickstart-custom-container?pivots=container-linux. The Azure documentation specifies:

In App Service, TLS termination of the request happens at the frontend load balancer. When forwarding the request to your app code with client certificates enabled, App Service injects an X-ARR-ClientCert request header with the client certificate. App Service does not do anything with this client certificate other than forwarding it to your app. Your app code is responsible for validating the client certificate.

The tricky part seems to be, that when the web application is deployed as container, then the connection between load-balancer and Kestrel uses HTTP. (While deploying the application natively uses HTTPS.)

To Reproduce

Create a new ASP.NET API (or other) application and configure client certificate authentication and forwarding:

            app.UseCertificateForwarding();
            app.UseAuthentication();
            app.UseAuthorization();

            // Setup forwarding as certificates in Azure Web App are terminated by the frontend.
            services.AddCertificateForwarding(options => { options.CertificateHeader = "X-ARR-ClientCert"; });

            services.AddAuthentication(CertificateAuthenticationDefaults.AuthenticationScheme).AddCertificate(
                options =>
                {
                    // Disable most validation options
                    options.ChainTrustValidationMode = X509ChainTrustMode.System;
                    options.RevocationMode = X509RevocationMode.NoCheck;
                    options.AllowedCertificateTypes = CertificateTypes.All;
                    options.ValidateCertificateUse = false;
                    options.ValidateValidityPeriod = false;

                    options.Events = new CertificateAuthenticationEvents
                    {
                        OnAuthenticationFailed = context =>
                        {
                            Console.WriteLine($"OnAuthenticationFailed {context.Exception}");
                            context.Fail("Invalid Certification!");
                            return Task.CompletedTask;
                        },
                        OnCertificateValidated = context =>
                        {
                            var cert = context.ClientCertificate;
                            Console.WriteLine($"OnCertificateValidated {cert}");
                            context.Success();
                            return Task.CompletedTask;
                        }
                    };
                });

See - https://github.com/NikMeyer/AzureClientCertIssue/tree/main/Sample

Now start the application in different configurations and access the test/auth endpoint that is configured to require authorization:

Running on	TLS on Kestrel	Client uses a Client Certificate	Client Passing Certificate in X-ARR-ClientCert	Result
Local Docker	YES	YES	NO	OK - CertificateAuthenticationHandler is used to validate the certificate based on the options.
Local Docker	YES	NO	YES	OK - Certificate is forwarded by the CertificateForwardingMiddleware and CertificateAuthenticationHandler is used to validate the certificate based on the options.
Local Docker	NO	NO	YES	NOT OK - Certificate is forwarded by the CertificateForwardingMiddleware but CertificateAuthenticationHandler aborts the validation with ....CertificateAuthenticationHandler[3] Not https, skipping certificate authentication.
Azure Web APP, Linux, Native	YES*	Only up to load balancer	YES by load balaner	OK - Certificate is put in the HTTP header by the load balancer and forwarding/authorization works.
Azure Web APP, Linux, Container	NO*	Only up to load balancer	YES by load balancer	NOT OK - Certificate is put in the HTTP header by the load balancer and but authorization is rejected without looking at the certificate as connection between container and load balancer is using HTTP only.

* I could not find any way to control that in Azure APP services and host the container with HTTPS support. It would likely solve the issue if the container could be using HTTPs. (There is also an issue reported on the Azure Web App side but that is closed with the claim that Azure Web App correctly forwards the headers and that its up to the application to use that header correctly - https://github.com/MicrosoftDocs/azure-docs/issues/66197)

The issue would likely be solved if having an HTTPS connection is not required. But that might have other security implications....

Exceptions (if any)

None

Logs when accessing the test/auth endpoint with TLS and Client Certificate in X-ARR-ClientCert:

info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
      Request starting HTTP/1.1 GET https://localhost:49163/test/auth text/plain 950
dbug: Microsoft.AspNetCore.Routing.Matching.DfaMatcher[1001]
      1 candidate(s) found for the request path '/test/auth'
dbug: Microsoft.AspNetCore.Routing.Matching.DfaMatcher[1005]
      Endpoint 'Sample.Controllers.TestController.GetWithAuth (Sample)' with route pattern 'test/auth' is valid for the request path '/test/auth'
dbug: Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware[1]
      Request matched endpoint 'Sample.Controllers.TestController.GetWithAuth (Sample)'
warn: Microsoft.AspNetCore.Authentication.Certificate.CertificateAuthenticationHandler[2]
      Certificate validation failed, subject was CN=<REMOVED>
      PartialChain unable to get local issuer certificate
info: Microsoft.AspNetCore.Authentication.Certificate.CertificateAuthenticationHandler[7]
      Certificate was not authenticated. Failure message: Client certificate failed validation.
info: Microsoft.AspNetCore.Authorization.DefaultAuthorizationService[2]
      Authorization failed. These requirements were not met:
      DenyAnonymousAuthorizationRequirement: Requires an authenticated user.
info: Microsoft.AspNetCore.Authentication.Certificate.CertificateAuthenticationHandler[12]
      AuthenticationScheme: Certificate was challenged.
dbug: Microsoft.AspNetCore.Server.Kestrel[9]
      Connection id "0HM8B2N64IJHF" completed keep alive response.
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/1.1 GET https://localhost:49163/test/auth text/plain 950 - 403 0 - 19.0672ms
dbug: Microsoft.AspNetCore.Server.Kestrel[25]
      Connection id "0HM8B2N64IJHF", Request id "0HM8B2N64IJHF:00000003": started reading request body.
dbug: Microsoft.AspNetCore.Server.Kestrel[26]
      Connection id "0HM8B2N64IJHF", Request id "0HM8B2N64IJHF:00000003": done reading request body.

(= Certificate is validated but rejected. I.e. A valid certificate would work)

Logs when accessing the test/auth endpoint without TLS and Client Certificate in X-ARR-ClientCert:

info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
      Request starting HTTP/1.1 GET http://localhost:49162/test/auth text/plain 950
dbug: Microsoft.AspNetCore.Routing.Matching.DfaMatcher[1001]
      1 candidate(s) found for the request path '/test/auth'
dbug: Microsoft.AspNetCore.Routing.Matching.DfaMatcher[1005]
      Endpoint 'Sample.Controllers.TestController.GetWithAuth (Sample)' with route pattern 'test/auth' is valid for the request path '/test/auth'
dbug: Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware[1]
      Request matched endpoint 'Sample.Controllers.TestController.GetWithAuth (Sample)'
dbug: Microsoft.AspNetCore.Authentication.Certificate.CertificateAuthenticationHandler[3]
      Not https, skipping certificate authentication.
dbug: Microsoft.AspNetCore.Authentication.Certificate.CertificateAuthenticationHandler[9]
      AuthenticationScheme: Certificate was not authenticated.
info: Microsoft.AspNetCore.Authorization.DefaultAuthorizationService[2]
      Authorization failed. These requirements were not met:
      DenyAnonymousAuthorizationRequirement: Requires an authenticated user.
info: Microsoft.AspNetCore.Authentication.Certificate.CertificateAuthenticationHandler[12]
      AuthenticationScheme: Certificate was challenged.
dbug: Microsoft.AspNetCore.Server.Kestrel[9]
      Connection id "0HM8B20RB3SOB" completed keep alive response.
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/1.1 GET http://localhost:49162/test/auth text/plain 950 - 403 0 - 62.3059ms
dbug: Microsoft.AspNetCore.Server.Kestrel[25]
      Connection id "0HM8B20RB3SOB", Request id "0HM8B20RB3SOB:00000003": started reading request body.
dbug: Microsoft.AspNetCore.Server.Kestrel[26]
      Connection id "0HM8B20RB3SOB", Request id "0HM8B20RB3SOB:00000003": done reading request body.

(= Certificate is not validated at all)

Further technical details

• ASP.NET Core v5.0
• Include the output of dotnet --info (See mcr.microsoft.com/dotnet/aspnet:5.0-buster-slim)
• Visual Studio 2019 windows

There is a similar issue here but that does not help as the client (load balancer) is not under my control - #18177

[blowdart]

You not only need to add certificate forwarding, you need to add protocol forwarding as well, which azure does support. That will override the actual protocol with the one the proxy server saw.

See https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/proxy-load-balancer?view=aspnetcore-5.0

[NikMeyer]

@blowdart - Thanks a lot! Adding forwarding configures fixes the issue and client certificate authentication now also works with Azure Web App Containers. I now closed the issue as it is clearly not a bug.

Code to add:

            services.Configure<ForwardedHeadersOptions>(options =>
            {
                options.ForwardedHeaders = ForwardedHeaders.XForwardedProto;
                options.ForwardedProtoHeaderName = "X-Forwarded-Proto";
            });

and

            app.UseForwardedHeaders();

[blowdart]

Wonderful, I filed a doc pr to get that azure doc updated