Kestrel config behaves differently when set using env vars vs. in-code

[qui8t]

I have a Web API application that I want to configure it to listen on an HTTPS port. The app is containerized and orchestrated using Docker compose.

I can configure it by setting the following env variable in docker-compose, which works as expected.

services:
  webapi:
    environment:
      - ASPNETCORE_URLS=https://+:443

However, when I use the following approach as an alternative to setting the environment variable, it fails with the following error message.

builder.WebHost.ConfigureKestrel(options =>
{
    options.Listen(
    address: IPAddress.Any,
        port: 443,
        configure: listenOptions =>
        {
            listenOptions.Protocols = HttpProtocols.Http1AndHttp2;
            listenOptions.UseHttps();
        });
});

Unable to configure HTTPS endpoint. No server certificate was specified, and the default developer certificate could not be found or is out of date.

When I provide the filename and password to the SSL certificate file as the following, it works as expected.

builder.WebHost.ConfigureKestrel(options =>
{
    options.Listen(
    address: IPAddress.Any,
        port: 443,
        configure: listenOptions =>
        {
            listenOptions.Protocols = HttpProtocols.Http1AndHttp2;
            listenOptions.UseHttps("/root/.aspnet/https/cert.pfx", "Password");
        });
});

I assume setting the environment variable ASPNETCORE_URLS or calling ConfigureKestrel are equivalent, though, I am not sure what I am missing in the second approach.

Update

I was previously getting the following error, which was related to mapping the port 443 to an incorrect port of the container.

Detected a TLS handshake to an endpoint that does not have TLS enabled.

[Tratcher]

The app is containerized and orchestrated using Docker compose.

I think this explains the difference. If you try environment variables vs code outside of the container they should work the same. Inside the container there are extra tooling steps to make the dev certificate available. I don't know if that is happening when you switch to the code version.

Detected a TLS handshake to an endpoint that does not have TLS enabled.

That's a weird error, you should only get that if you left out UseHttps. Even if it couldn't load the cert, it should fail to start instead.

[qui8t]

Inside the container there are extra tooling steps to make the dev certificate available. I don't know if that is happening when you switch to the code version.

Is mounting the certs onto the container sufficient?! I guess so, because when the certificate file and its password are explicity provided, it works as expected.

volumes:
  - ${APPDATA}/Microsoft/UserSecrets:/root/.microsoft/usersecrets:ro
  - ${APPDATA}/ASP.NET/Https:/root/.aspnet/https:ro

Detected a TLS handshake to an endpoint that does not have TLS enabled.

That's a weird error, you should only get that if you left out UseHttps. Even if it couldn't load the cert, it should fail to start instead.

This was caused by an error on my part mapping the port 443 to incorrect container port. My apologies for the confusion; I updated my question.

[adityamandaleeka]

@NCarlsonMSFT Can you take a look at the original post in this issue and see if there's something missing from the workflow here (why does it not work when the certificate path isn't specified)?

[NCarlsonMSFT]

@qui8t for clarification, are you using the VS tools for running your compose app? If so, I believe we key off the value for ASPNETCORE_URLS to know whether to do our magic to enable HTTPS so that may be what is causing your strange behavior. If you don't want the value in your main compose file, you can specify it in docker-compose.override.yml located next your main docker-compose.yml file. As of now there is no other way to let the tools know that the service needs HTTPS support.

Alternatively you can add the volumes from your above comment, and export the dev certificate to %AppData%/ASP.NET/Https.pfx and store the password as a user secret named "Kestrel:Certificates:Development:Password". Having that user secret (or correspondingg environment variable etc.) tells kestrel to look for the file in the well known location.

[qui8t]

Yes, I am using VS tools.

I already have the setup in docker-compose.override.yml that mounts the directory containing the certificates onto the container. Though it seems it only works if I explicitly provide the certificate names and password.

[NCarlsonMSFT]

@adityamandaleeka I've replicated the scenario and can confirm that despite setting the same user secret and providing the same .pfx file as the compose tooling, attempting to configure HTTPs using the above method results in an error. Looking at the code it appears that the UseHttps extension method does not attempt to call FindDeveloperCertificateFile so it can't use this fall-back mechanism.

@qui8t as a work-around you can at least update what you have to also use the user secret (and only be used in development):

if (builder.Environment.IsDevelopment())
{
    listenOptions.UseHttps("/root/.aspnet/https/WebApplication1.pfx", builder.Configuration["Kestrel:Certificates:Development:Password"]);
}

I also modified my launch profile to manually use https as the scheme:

"Docker Compose": {
  "commandName": "DockerCompose",
  "commandVersion": "1.0",
  "composeLaunchAction": "LaunchBrowser",
  "composeLaunchServiceName": "webapplication1",
  "composeLaunchUrl": "https://localhost:{ServicePort}",
  "serviceActions": {
    "webapplication1": "StartDebugging"
  }
}

Bonus: The container tools won't regenerate the cert/secret for you w/o the Environment variable, but here is a script that when run in the Web project's folder will take care of doing the same steps

$password = [Guid]::NewGuid().ToString("N")
$projectName = [System.IO.Path]::GetFileNameWithoutExtension((Get-Item *.csproj)[0])

dotnet dev-certs https --trust --export-path "$Env:APPDATA\ASP.NET\Https\$projectName.pfx" -p $password
dotnet user-secrets init
dotnet user-secrets set "Kestrel:Certificates:Development:Password" "$password"

[qui8t]

@NCarlsonMSFT Thank you for looking into this.

[halter73]

It appears that we have two pieces of code that try to load the development cert. There's the following which doesn't look at config at all.

aspnetcore/src/Servers/Kestrel/Core/src/KestrelServerOptions.cs

Lines 283 to 324 in d2074d7

	private void EnsureDefaultCert()
	{
	if (DefaultCertificate == null && !IsDevCertLoaded)
	{
	IsDevCertLoaded = true; // Only try once
	var logger = ApplicationServices!.GetRequiredService<ILogger<KestrelServer>>();
	try
	{
	DefaultCertificate = CertificateManager.Instance.ListCertificates(StoreName.My, StoreLocation.CurrentUser, isValid: true, requireExportable: false)
	.FirstOrDefault();
	if (DefaultCertificate != null)
	{
	var status = CertificateManager.Instance.CheckCertificateState(DefaultCertificate, interactive: false);
	if (!status.Success)
	{
	// Display a warning indicating to the user that a prompt might appear and provide instructions on what to do in that
	// case. The underlying implementation of this check is specific to Mac OS and is handled within CheckCertificateState.
	// Kestrel must NEVER cause a UI prompt on a production system. We only attempt this here because Mac OS is not supported
	// in production.
	Debug.Assert(status.FailureMessage != null, "Status with a failure result must have a message.");
	logger.DeveloperCertificateFirstRun(status.FailureMessage);
	// Prevent binding to HTTPS if the certificate is not valid (avoid the prompt)
	DefaultCertificate = null;
	}
	else if (!CertificateManager.Instance.IsTrusted(DefaultCertificate))
	{
	logger.DeveloperCertificateNotTrusted();
	}
	}
	else
	{
	logger.UnableToLocateDevelopmentCertificate();
	}
	}
	catch
	{
	logger.UnableToLocateDevelopmentCertificate();
	}
	}
	}

The above ends up being the only thing that tries to set the DefaultCertificate if KestrelConfigurationLoader.Load() never gets called, which would mean the following logic couldn't set it first:

aspnetcore/src/Servers/Kestrel/Core/src/KestrelConfigurationLoader.cs

Lines 399 to 451 in d2074d7

	private void LoadDefaultCert()
	{
	if (ConfigurationReader.Certificates.TryGetValue("Default", out var defaultCertConfig))
	{
	var (defaultCert, _ /* cert chain */) = CertificateConfigLoader.LoadCertificate(defaultCertConfig, "Default");
	if (defaultCert != null)
	{
	DefaultCertificateConfig = defaultCertConfig;
	Options.DefaultCertificate = defaultCert;
	}
	}
	else
	{
	var (certificate, certificateConfig) = FindDeveloperCertificateFile();
	if (certificate != null)
	{
	Logger.LocatedDevelopmentCertificate(certificate);
	DefaultCertificateConfig = certificateConfig;
	Options.DefaultCertificate = certificate;
	}
	}
	}
	private (X509Certificate2?, CertificateConfig?) FindDeveloperCertificateFile()
	{
	string? certificatePath = null;
	if (ConfigurationReader.Certificates.TryGetValue("Development", out var certificateConfig) &&
	certificateConfig.Path == null &&
	certificateConfig.Password != null &&
	TryGetCertificatePath(out certificatePath) &&
	File.Exists(certificatePath))
	{
	try
	{
	var certificate = new X509Certificate2(certificatePath, certificateConfig.Password);
	if (IsDevelopmentCertificate(certificate))
	{
	return (certificate, certificateConfig);
	}
	}
	catch (CryptographicException)
	{
	Logger.FailedToLoadDevelopmentCertificate(certificatePath);
	}
	}
	else if (!string.IsNullOrEmpty(certificatePath))
	{
	Logger.FailedToLocateDevelopmentCertificateFile(certificatePath);
	}
	return (null, null);
	}

I'm surprised KestrelConfigurationLoader.Load() isn't being called here in a WebApplicationBuilder-based app:

aspnetcore/src/Servers/Kestrel/Core/src/Internal/KestrelServerImpl.cs

Line 315 in d2074d7

Options.ConfigurationLoader?.Load();

And `Options.ConfigurationLoader should always be set in WebApplicationBuilder-based apps by the following:

aspnetcore/src/DefaultBuilder/src/WebHost.cs

Lines 225 to 228 in d2074d7

	builder.UseKestrel((builderContext, options) =>
	{
	options.Configure(builderContext.Configuration.GetSection("Kestrel"), reloadOnChange: true);
	})

There are scenarios where we know KestrelServerOptions.ConfigurationLoader is not set though. I bet the logic for reading the development certificate from config was put into KestrelConfigurationLoader because that's about the only place Kestrel has access to the IConfiguration, and if it's not set, that means that the user didn't configure a "Kestrel" config section and therefore might be surprised to see something trying to read "Kestrel:Certificates:Development:Password".

It's unfortunate that our tooling tooling requires Kestrel to bind to the "Kestrel" section of config for the dev cert to work on docker. It might make sense to fix Kestrel to look there for the dev cert regardless of whether KestrelServerOptions.Configure(IConfiguration) was ever called if otherwise it would fail.

[halter73]

It looks like UseHttps() is probably calling KestrelServerOptions.ApplyDefaultCert() too early before KestrelConfigurationLoader.Load().

[Tratcher]

There are scenarios where we know KestrelServerOptions.ConfigurationLoader is not set though.

Such as?

[halter73]

There are scenarios where we know KestrelServerOptions.ConfigurationLoader is not set though.

Such as?

When you use a non-default builder. So ConfigureWebHost instead of ConfigureWebHostDefaults. It can also be overridden even if you are using the defaults.

This seems related to #28120 and #26258.