Intermediate cert (in addition to leaf cert) is not sent from http client to the server in .net (core/aspnetcore) 5?

[rudeGit]

I was not able to make http client code in .net 5 to send both intermediate and leaf certificates (in 3 certificate hierarchy) to the server. However I was able to send the leaf certificate from client to the server successfully.

Here is my setup:

I have 3 certificates on my windows box:

1. TestRoot.pem
2. TestIntermediate.pem
3. TestLeaf.pem (without private key for server - windows box)
4. TestLeaf.pfx (with private key for client - windows box)

None of the above certificates were added to windows certificate manager as I would like to be able to run the same code on non-windows machines eventually. For my testing, I am running following client and server code on the same windows box.

On my windows box, I have following simple client side code using .net 5:

	HttpClientHandler handler = new HttpClientHandler();
	handler.ClientCertificateOptions = ClientCertificateOption.Manual;
	handler.SslProtocols = System.Security.Authentication.SslProtocols.Tls12;

	X509Certificate2 leafCert = new X509Certificate2(File.ReadAllBytes(@"C:\Temp\TestLeaf.pfx"), "<password>");

	handler.ClientCertificates.Add(leafCert);

	HttpClient httpClient = new HttpClient(handler);

	StringContent content = new StringContent("{}"); //Test json string
	content.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue(MediaTypeNames.Application.Json);

	//With local.TestServer.com resolving to localhost in the host file
	HttpResponseMessage response = httpClient.PostAsync("https://local.TestServer.com/...", content).Result;

	if (response.IsSuccessStatusCode)
	{
		var responseString = response.Content.ReadAsStringAsync().Result;

		Console.WriteLine(responseString);
	}
	else
	{
		Console.WriteLine(x.StatusCode);
		Console.WriteLine(x.ReasonPhrase);
	}

On same window box, I have following example snippet of server side code using kestrel in .net 5:

	services.Configure<KestrelServerOptions>(options =>
	{
		// Keep track of what certs belong to each port
		var certsGroupedByPort = ...;
		var certsPerDistinctSslPortMap = ...;

		// Listen to each distinct ssl port a cert specifies
		foreach (var certsPerDistinctSslPort in certsPerDistinctSslPortMap)
		{
			options.Listen(IPAddress.Any, certsPerDistinctSslPort.Key, listenOptions =>
			{
				var httpsConnectionAdapterOptions = new HttpsConnectionAdapterOptions();
				httpsConnectionAdapterOptions.ClientCertificateValidation = (clientCertificate, chain, sslPolicyErrors) =>
				{
					bool trusted = false;
					if (sslPolicyErrors == System.Net.Security.SslPolicyErrors.RemoteCertificateChainErrors)
					{
						chain.ChainPolicy.TrustMode = X509ChainTrustMode.CustomRootTrust;

						X509Certificate2 certRoot = new X509Certificate2(@"C:\Temp\TestRoot.pem");
						X509Certificate2 certIntermdiate = new X509Certificate2(@"C:\Temp\TestIntermediate.pem");

						chain.ChainPolicy.CustomTrustStore.Add(certRoot);
						chain.ChainPolicy.ExtraStore.Add(certIntermdiate);
						trusted = chain.Build(clientCertificate);
					}

					return trusted;
				};
				httpsConnectionAdapterOptions.ServerCertificateSelector = (connectionContext, sniName) =>
				{
					var defaultCert = //Get default cert
					return defaultCert;
				};
				httpsConnectionAdapterOptions.ClientCertificateMode = ClientCertificateMode.RequireCertificate;
				httpsConnectionAdapterOptions.SslProtocols = SslProtocols.Tls12;

				listenOptions.UseHttps(httpsConnectionAdapterOptions);
			});
		}

		options.Listen(IPAddress.Any, listeningPort);
	});

The above code works as expected because the client code sends the leaf certificate to the server and the server code has access to both intermediate as well as root certificates. The server code can successfully rebuild the certificate hierarchy with received leaf certificate and its configured intermediate and root certs for the leaf certificate.

My following attempt to send the intermediate certificate (along with leaf certificate) to the server (so that it can only use the root certificate and incoming leaf and intermediate certificates in the request to build the certificate hierarchy) failed.

1. Tried to add the intermediate certificate by doing following in my client code:

	X509Certificate2 leafCert = new X509Certificate2(File.ReadAllBytes(@"C:\Temp\TestLeaf.pfx"), "<password>");
	X509Certificate2(Convert.FromBase64String(File.ReadAllText(@"C:\Temp\TestIntermediate.pem"));

	handler.ClientCertificates.Add(leafCert);
	handler.ClientCertificates.Add(intermediateCert);

This did not send the intermediate certificate to the server. I verified this with the code block for httpsConnectionAdapterOptions.ClientCertificateValidation on the server side.

Question:
Is it even possible in .net 5 to ensure that intermediate certificate is sent by the client (in addition to the leaf cert) to the server?

[ghost]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

I was not able to make http client code in .net 5 to send both intermediate and leaf certificates (in 3 certificate hierarchy) to the server. However I was able to send the leaf certificate from client to the server successfully.

Here is my setup:

I have 3 certificates on my windows box:

1. TestRoot.pem
2. TestIntermediate.pem
3. TestLeaf.pem (without private key for server - windows box)
4. TestLeaf.pfx (with private key for client - windows box)

None of the above certificates were added to windows certificate manager as I would like to be able to run the same code on non-windows machines eventually. For my testing, I am running following client and server code on the same windows box.

On my windows box, I have following simple client side code using .net 5:

HttpClientHandler handler = new HttpClientHandler();
handler.ClientCertificateOptions = ClientCertificateOption.Manual;
handler.SslProtocols = System.Security.Authentication.SslProtocols.Tls12;

X509Certificate2 leafCert = new X509Certificate2(File.ReadAllBytes(@"C:\Temp\TestLeaf.pfx"), "<password>");

handler.ClientCertificates.Add(leafCert);

HttpClient httpClient = new HttpClient(handler);

StringContent content = new StringContent("{}"); //Test json string
content.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue(MediaTypeNames.Application.Json);

//With local.TestServer.com resolving to localhost in the host file
HttpResponseMessage response = httpClient.PostAsync("https://local.TestServer.com/...", content).Result;

if (response.IsSuccessStatusCode)
{
	var responseString = response.Content.ReadAsStringAsync().Result;

	Console.WriteLine(responseString);
}
else
{
	Console.WriteLine(x.StatusCode);
	Console.WriteLine(x.ReasonPhrase);
}

On same window box, I have following example snippet of server side code using kestrel in .net 5:

services.Configure<KestrelServerOptions>(options =>
{
	// Keep track of what certs belong to each port
	var certsGroupedByPort = ...;
	var certsPerDistinctSslPortMap = ...;

	// Listen to each distinct ssl port a cert specifies
	foreach (var certsPerDistinctSslPort in certsPerDistinctSslPortMap)
	{
		options.Listen(IPAddress.Any, certsPerDistinctSslPort.Key, listenOptions =>
		{
			var httpsConnectionAdapterOptions = new HttpsConnectionAdapterOptions();
			httpsConnectionAdapterOptions.ClientCertificateValidation = (clientCertificate, chain, sslPolicyErrors) =>
			{
				bool trusted = false;
				if (sslPolicyErrors == System.Net.Security.SslPolicyErrors.RemoteCertificateChainErrors)
				{
					chain.ChainPolicy.TrustMode = X509ChainTrustMode.CustomRootTrust;

					X509Certificate2 certRoot = new X509Certificate2(@"C:\Temp\TestRoot.pem");
					X509Certificate2 certIntermdiate = new X509Certificate2(@"C:\Temp\TestIntermediate.pem");

					chain.ChainPolicy.CustomTrustStore.Add(certRoot);
					chain.ChainPolicy.ExtraStore.Add(certIntermdiate);
					trusted = chain.Build(clientCertificate);
				}

				return trusted;
			};
			httpsConnectionAdapterOptions.ServerCertificateSelector = (connectionContext, sniName) =>
			{
				var defaultCert = //Get default cert
				return defaultCert;
			};
			httpsConnectionAdapterOptions.ClientCertificateMode = ClientCertificateMode.RequireCertificate;
			httpsConnectionAdapterOptions.SslProtocols = SslProtocols.Tls12;

			listenOptions.UseHttps(httpsConnectionAdapterOptions);
		});
	}

	options.Listen(IPAddress.Any, listeningPort);
});

The above code works as expected because the client code sends the leaf certificate to the server and the server code has access to both intermediate as well as root certificates. The server code can successfully rebuild the certificate hierarchy with received leaf certificate and its configured intermediate and root certs for the leaf certificate.

My following attempt to send the intermediate certificate (along with leaf certificate) to the server (so that it can only use the root certificate and incoming leaf and intermediate certificates in the request to build the certificate hierarchy) failed.

1. Tried to add the intermediate certificate by doing following in my client code:

   X509Certificate2 leafCert = new X509Certificate2(File.ReadAllBytes(@"C:\Temp\TestLeaf.pfx"), "");
   X509Certificate2(Convert.FromBase64String(File.ReadAllText(@"C:\Temp\TestIntermediate.pem"));

   handler.ClientCertificates.Add(leafCert);
   handler.ClientCertificates.Add(intermediateCert);

   This did not send the intermediate certificate to the server. I verified this with the code block for httpsConnectionAdapterOptions.ClientCertificateValidation on the server side.

Question:
Is it even possible in .net 5 to ensure that intermediate certificate is sent by the client (in addition to the leaf cert) to the server?

Author:	rudeGit
Assignees:	-
Labels:	area-System.Net.Http, untriaged
Milestone:	-

[davidfowl]

cc @wfurt

[wfurt]

what platform you are on @rudeGit? I can take a look next week. May be related to #47580 as well.

Also you can put itnto system or user intermediate store just like the CertitificateContext does.

runtime/src/libraries/System.Net.Security/src/System/Net/Security/SslStreamCertificateContext.Windows.cs

Lines 42 to 46 in 0aad03b

	// OS failed to build the chain but we have at least some intermediates.
	// We will try to add them to "Intermediate Certification Authorities" store.
	if (!osCanBuildChain)
	{
	X509Store? store = new X509Store(StoreName.CertificateAuthority, StoreLocation.LocalMachine);

[rudeGit]

@wfurt Currently using windows (details below) for testing. But I would like to see a solution that works on all platforms without any code modification.

Edition Windows 10 Enterprise Insider Preview
Version 2004
Installed on ‎1/‎28/‎2021
OS build 21301.1000
Experience Windows Feature Experience Pack 221.501.0.3

[wfurt]

Windows is the most difficult platform. There is no way how .NET can pass the chain to OS and underlying Schannel can only work with certificates visible in store. Adding your intermediates to StoreName.CertificateAuthority is right approach. (that will not make them trusted - just available for OS)

This should also make it work on other platforms. On macOS & Linux, we use X509Chain.Build to compute the chain and create list of certificates to send. Certificates in StoreName.CertificateAuthority should be picked up automaticaly and you should get same behavior.

The handler.ClientCertificates is really meant for the leaf certificates. If automatic selection is in place, it will ignore all Certificates without private key - like the intermediate(s).

[rudeGit]

@wfurt I modified my client side to do following:

	var intermediateCertBytes = Convert.FromBase64String(File.ReadAllText(@"C:\Temp\TestIntermediate.pem"));
	X509Certificate2 intermediateCert = new X509Certificate2(intermediateCertBytes);

	using X509Store intermediateStore = new X509Store(StoreName.CertificateAuthority, StoreLocation.LocalMachine);
	
	//******** The following line requires admin privileges for the code because of Write flag ********
	intermediateStore.Open(OpenFlags.ReadWrite);
	//******** 
	
	// Without validOnly=false, the following code does not find the certificate if the thumbprint in the store is lowercase and
	//  the passed thumbprint is uppercase and vice versa.
	var foundIntermediateCerts = intermediateStore.Certificates.Find(X509FindType.FindByThumbprint, intermediateCert.Thumbprint, validOnly:false);
	if (foundIntermediateCerts.Count == 0)
	{
		intermediateStore.Add(intermediateCert);
	}
	
	//The following code is as before
	HttpClientHandler handler = new HttpClientHandler();
	handler.ClientCertificateOptions = ClientCertificateOption.Manual;
	handler.SslProtocols = System.Security.Authentication.SslProtocols.Tls12;

The above client code resulted in sending the intermediate cert to the server. And the server side (on the same machine) was able to see both leaf and intermediate certs in the chain.ChainPolicy.ExtraStore in the httpsConnectionAdapterOptions.ClientCertificateValidation handler. However I could not be sure if this is happening because the server side code is also running on the same machine. I will move the server side code a different windows machine and will test again to make sure that this is working as expected.

However I do have an issue with client side having admin privileges so that intermediate cert can be added to the cert store. This whole approach seems arcane to me.

Should there not be a temporary store (without admin privileges) that we can build and pass it to HttpClientHandler instance so that .net core code can use both this temporary store as well as windows store to come up with hierarchy to send?

[wfurt]

`StoreLocation.CurrentUser' should work equally without admin privilege. I would skip the search logic. If the given cert is already in the store adding it again will be no-op. (e.g. there will not be increasing duplicate entries)

As far as the verification, you can always use Wireshark and check what comes on the wire. What you should see is certificate chain without root CA. For PKI to work properly, root should be installed on peer's machine and trusted.

The client capabilities could be probably improved. It needs to start with SslStream. The difficult part is that we would want something that work consistently across all supported platforms and that is not trivial.

[rudeGit]

@wfurt I tested with CurrentUser store and it worked without requiring admin privileges. I also verified with WireShark that the intermediate certificate is being transferred over the wire to the server. On the server side, I can see both leaf and intermediate certificates in the chain.ChainPolicy.ExtraStore object. Now I am confident that both certs will shows on the remote server as well.

This gets me going for now. However as you have mentioned in the #48017, it is better not to mess with the certificate store at all and it is better to have clean API to add the certificate hierarchy on the client side.

Thank you for your help.

[karelz]

Triage: Question seems to be answered.
We still might want to file a new issue to capture the ask to use cert chain without putting it in store -- new API would be required. (@wfurt to file a mega-issue linking the issues)