Kestrel does not listen for https when run as a service on Ubuntu

[Praveen-Rai]

I am trying to host an ASP.Net Core MVC application (https redirection is enabled) on Ubuntu server, using Nginx as a reverse proxy. I have created and installed a local SSL certificate using OpenSSL. When i run my application using dotnet CLI it listens on both http://localhost:5000 & https://localhost:5001, and i am able to access it on web using https (http requests are being redirect to https by Nginx).

The problem is when i try to run the as a service, it only listens on http://localhost:5000.

Here's the *.service file :

[Unit]
Description=Test ASP.Net core web application service.

[Service]
WorkingDirectory=/home/ubuntu/MyAppFolder
ExecStart=/usr/bin/dotnet/home/ubuntu/MyAppFolder/MyApplication.dll
Restart=always
# Restart service after 10 seconds if the dotnet service crashes:
RestartSec=10
SyslogIdentifier=MyApplication
User=www-data
Environment=ASPNETCORE_ENVIRONMENT=Development
Environment=DOTNET_PRINT_TELEMETRY_MESSAGE=false
Environment=ASPNETCORE_HTTPS_PORT=5001
Environment=ASPNETCORE_URLS=http://localhost:5000;https://localhost:5001

[Install]
WantedBy=multi-user.target

Environment details : ASP.Net Core 2.1.1, ASP.Net Core SDK 2.1.3, Nginx 1.14, Ubuntu 16.04

[Tratcher]

@javiercn

[javiercn]

@Praveen-Rai Thanks for contacting us.

When you run the tool on the CLI you are using your current user. When you run it as a service the user is www-data. For it to work you will need to run the dotnet-devcerts tool as the www-data user.

Alternatively you can generate an ssl cert and configure kestrel to use it. I think that is a better option if you are running the app as a service. Hope this helps.

[Praveen-Rai]

@Tratcher @javiercn Thanks a lot for your prompt reply !!

you will need to run the dotnet-devcerts tool as the www-data user

Any hints on how to do that ?

Alternatively you can generate an ssl cert and configure kestrel to use it. I think that is a better option if you are running the app as a service.

Thanks, i will try this.

[javiercn]

@Praveen-Rai I'm not an expert in Linux, but I believe sudo or su allows you to do that. Even if you do this I'm not confident that approach will work, feel free to try it out, but otherwise just generate a certificate and configure Kestrel to use it.

[Praveen-Rai]

@javiercn Though i am not sure about the dotnet-devcerts, but i will surely try adding the SSL certificate to Kestrel configuration. For your information i am also looking forward to get the docs Hosting Docs updated for the same.
Again, thanks !! your inputs are valuable and it would be preferable if we get more information on dotnet-devcerts. It would be great if you could bring someone knowledgeable in dotnet-devcerts into the conversation.

[javiercn]

@Praveen-Rai I'm the main developer on it. dotnet dev-certs will setup the developer certificate in the current user store and that's what kestrel loads. dotnet dev-certs https will make sure that there is a certificate on the current user store, but I don't recommend you use it with the approach you are taking and instead strongly encourage you to create your own certs. While technically things should work with what i've mentioned, its not one of the main scenarios for this tool.

Hope this helps.

[Praveen-Rai]

@javiercn , @Tratcher

When you run the tool on the CLI you are using your current user. When you run it as a service the user is www-data.

Yes, i ran the service as the root user and it listens on both the ports. Though it would be a bad idea to run a service using root privileges, Right ?

Alternatively you can generate an ssl cert and configure kestrel to use it. I think that is a better option if you are running the app as a service. Hope this helps.

I tried configuring Kestrel to use the certificate using appsettings.json. Though the app started listening on ssl port ( 5001 ), but it didn't work. wget says

12.0.0.1:5001 ... connected.
Unable to establish ssl connection"

appsettings.json
"Kestrel": { "Endpoints": { "HTTPS": { "Url": "https://localhost:5001", "Certificate": { "Path": "/etc/ssl/certs/nginx-selfsigned.crt" } } } }

I suspect that the above configuration is expecting *.pfx and not *.crt. But then i am wondering how does the application works with root user access for the same certificate. Any hints ?

References :
Net Core 2.1 HTTPS Improvements Blog Post
Configuring HTTSP Blog Post
Github issue

Notes :

• Configuring Kestrel using config file lacks documentation.
• Seems like we dropped for configuration of Kestrel using the config file in .Net Core 2.0 and brought it back in Core 2.1 and hence is confusion and missing articles.
• dev-certs documentation isn't complete too. Kestrel EndPoint Configuration and ASP Net Core 2.1 HTTPS reference each other, and suggest to run dotnet dev-certs https --trust command. Though when i run it i get

Specify --help for a list of available options and commands.

[Praveen-Rai]

@javiercn , @Tratcher
I am still struggling with the issue. One more observation i would like to share here. When i configure the service to run as an Admin user, i get an info stating :

info : Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[0] User profile is available. Using /home/<username>/.aspnet/DataProtection-keys as key repository ; keys will not be encrypted at rest.

And when the application is run as service as 'www-data', i get a warning :
warn : Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[35] No XML encryptor configured, key <GUID> may be persisted to storage in unencrypted form.

Could this be the cause of issue ??

[davidfowl]

Nope. That's normal.

[Praveen-Rai]

To update, i've added www-data to ssl-cert group. Granted read and execute permissions on the private key directory to the ssl-cert group. Still i am facing the issue.

[javiercn]

@Praveen-Rai Trusting the certificate is only available on Windows and Mac OS. There is no standard way of trusting the certificate on Linux.

You need to have the certificate along with the private key in a pxf format. @Tratcher we should throw an error if the certificate doesn’t have an associated private key.

You should run the tool as the www-data user, not as root. I believe you can do that with sido or su on Ubuntu, but check your rostro documentation.

Finally, running any kind of service with the development certificate in production is not supported and completely discouraged.

@danroth27 regarding docs for HTTPS can you loop in the right folks?

Hope this helps

[Praveen-Rai]

@javiercn

Trusting the certificate is only available on Windows and Mac OS. There is no standard way of trusting the certificate on Linux.

Do you mean to say that this issue is due to trusting of certificate ? I have a doubt then, how does it work with Root user ? Doesn't the certificate trust apply machine wide ?

You need to have the certificate along with the private key in a pxf format.

Now i don't feel that's the matter, coz the application is working with root privileges without the *.pxf format.

You should run the tool as the www-data user, not as root

The service ( AspNetCore App ) is ran as 'www-data', though i couldn't directly run the dotnet command as 'www-data' ( doing a bit of search, i feel that's as designed ). And that's where the problem is.

Finally, running any kind of service with the development certificate in production is not supported and completely discouraged.

I agree, i am using this certificate for development only.

we should throw an error if the certificate doesn’t have an associated private key.

My upvote for that. We've been always restricted to MS OS concepts with .Net, but with .Net core we are introduced to MacOs & Linux and it's distributions. So, may be it would be safer to log warnings & errors more sensitively than we used to do before with .Net.

[Praveen-Rai]

@javiercn

Finally i figured out the issue. The issue is that a developer ssl certificate is installed with dotnet SDK with the name localhost. In case of Ubuntu the certificate is located at /home/{user name} /.dotnet/corefx/cryptography/x509stores/my

I believe Kestrel just searches in the home directory of executing user, which does not exists for 'www-data', hence it couldn't locate the development certificate. Due to which it doesn't bind to default https port.

Here are the key points ( please provide your comments ) :

1. The dotnet sdk installer must install the local development certificate to /etc/ssl/certs as it is the designated store for SSL certificates in Ubuntu ( varies as per distribution ).

2. Kestrel must search the development certificate in the common SSL certificate store, rather in the current user home. Because not all users have home directory for e.g. 'www-data'.

3. Whenever kestrel is loading a local development certificate installed by dotnet sdk, it must log with level "Warning". Presently it logs at level "Dbug" (even in the production environment). In production environment the log level might further be escalated.

4. Kestrel must also produce warnings for incorrect or non-existing endpoint configuration. Kestrel does have defaults to fallback to, but it always expects the endpoints to be configured.

5. We need documentation on configuration of kestrel ( Infact logging and there might be more ). For kestrel endpoint specification using configuration file, i could locate some content authored by @danroth27 & @Tratcher . We must formalize this content into the ASP.Net Core documentation along with other configuration options such as Logging. Also, Kestrel supports specifying the certificates in both Store-Key and File-Password format. This needs to be clearly mentioned with examples, as there might be cases where one is developing on Windows machine and Hosting on Linux.

6. Running an application using
   dotnet <DotNetCoreApp>
   does not print detailed log, even with the use of command options and with configuration file specifying the logging options. Until the command is run from the application directory.

7. Kestrel currently seems to support PKCS12 ( *.pfx ) only, whereas PEM seems to be de-facto for Apache, Nginx or Linux on the whole ( as per the limited knowledge i've captured recently via. web ). If that's the case, we need to highlight it in our docs. Secondly, can we start supporting PEM in Kestrel ?

[javiercn]

1. The dotnet sdk installer must install the local development certificate to /etc/ssl/certs as it is the designated store for SSL certificates in Ubuntu ( varies as per distribution ).

No. ASP.NET uses the current user personal certificate store which was made to work cross-platform. We don't have any plans to do anything else as there is no standard way of achieving this on linux systems and we just simply rely on .NET functionality.

2. Kestrel must search the development certificate in the common SSL certificate store, rather in the current user home. Because not all users have home directory for e.g. 'www-data'.

No. The main purpose of the dev-certs tool is to support the development of APIs and Websites in the most common development environments and scenarios. Doing development of your app by running it as a service is not something that we consider a core development scenario (compared to for example, doing dotnet run). You can still develop using HTTPS by manually loading a certificate in pfx format using configuration.

3. Whenever kestrel is loading a local development certificate installed by dotnet sdk, it must log with level "Warning". Presently it logs at level "Dbug" (even in the production environment). In production environment the log level might further be escalated.

I don't think so. That will produce a jarring experience during regular development.

4. Kestrel must also produce warnings for incorrect or non-existing endpoint configuration. Kestrel does have defaults to fallback to, but it always expects the endpoints to be configured.

I'll leave this up to @Tratcher to answer.

5. We need documentation on configuration of kestrel ( Infact logging and there might be more ). For kestrel endpoint specification using configuration file, i could locate some content authored by @danroth27 & @Tratcher . We must formalize this content into the ASP.Net Core documentation along with other configuration options such as Logging. Also, Kestrel supports specifying the certificates in both Store-Key and File-Password format. This needs to be clearly mentioned with examples, as there might be cases where one is developing on Windows machine and Hosting on Linux.

If this is the case @danroth27 can you get the doc writers involved here?

6. Running an application using
   dotnet <DotNetCoreApp>
   does not print detailed log, even with the use of command options and with configuration file specifying the logging options. Until the command is run from the application directory.

I have no idea about this. @Tratcher or @danroth27 can comment.

7. Kestrel currently seems to support PKCS12 ( *.pfx ) only, whereas PEM seems to be de-facto for Apache, Nginx or Linux on the whole ( as per the limited knowledge i've captured recently via. web ). If that's the case, we need to highlight it in our docs. Secondly, can we start supporting PEM in Kestrel ?

We support what .NET Core supports. You can probably file an issue on https://github.com/dotnet/corefx and they can probably tell you whether its supported or what the plans are if any.

Hope this helps.

[danroth27]

The kestrel endpoint configuration is documented here: https://docs.microsoft.com/en-us/aspnet/core/fundamentals/servers/kestrel?view=aspnetcore-2.1#how-to-use-kestrel-in-aspnet-core-apps. It's a bit buried in the section, which we plan to fix: dotnet/AspNetCore.Docs#8496.

[Praveen-Rai]

@javiercn I think you're missing a couple of key points here :

1. Linux doesn't have a concept of Key Store, hence we cannot just rely on .Net functionality. What all distributions does have is a standard ( or atleast widely accepted ) locations for storing the certificates and we'll have to respect it to be able to say the .Net Core is Cross-Platform. All the certificates must be created and read from these locations only.

2. Most of Linux servers are accessed using SSH and any process triggered using a SSH terminal is terminated upon the end of session. Even in a Windows Environment you would like to run a WebApp as a service, and not just as a program initiated by a User. Hence, no matter the platform, the ultimate destiny of any Asp.Net core app would be running as a service. And it's a good idea to develop in an environment which closely resembles your production environment.

3. When something is important to be present in Production, my expectation is to log warning in Development and Error in Production. Presently, Kestrel does none of it and just doesn't bind to the HTTPS port, which has led to this discussion. Also, please note that even now when you run a application a lot of stuff is getting printed on the command prompt including 'info'. Adding a single line won't hurt, though i accept that we must work towards logging only the important ones.

4. Last but not the least, .Net Framework has been here for more than a decade and it has a huge fan following ( including me ). But we all understand we missed a lot of Open-Source community stuffed, being confined to MS eco-system. Net Core is a effort to bring that all together by making .Net Framework cross-platform ( and i cannot express how happy i was with the announcement of .Net Core ). And hence we simply cannot ignore Linux ( atleast not for web applications and apis ).

Hope that addresses your concerns.

[danroth27]

Configuring logging is documented here: https://docs.microsoft.com/en-us/aspnet/core/fundamentals/logging/?view=aspnetcore-2.1#configuration.

@Praveen-Rai Could you please take a look at these docs and then file issues in the https://github.com/aspnet/Docs repo for any content that you think is missing?

[Praveen-Rai]

@danroth27 Sorry, i meant configuration using files. Because, you wouldn't want to hard-code the endpoint settings, SSL certificate path .. etc into your code.

I just realized today, that the configuration settings in the code can be directly mapped into the configuration file. Reference Configuration and Options

Still what i suggest is to have a document of the kind you've written in your blog

[Tratcher]

The kestrel config file support is doc'd here: https://docs.microsoft.com/en-us/aspnet/core/fundamentals/servers/kestrel?view=aspnetcore-2.1#endpoint-configuration

[Praveen-Rai]

Configuring logging is documented here: https://docs.microsoft.com/en-us/aspnet/core/fundamentals/logging/?view=aspnetcore-2.1#configuration.

Yes, i've visited this page before, but i didn't find a Kestrel configuration sections on the docs. So, i thought why not to have a module of Configuration Using File, which will include individual topics for Kestrel, Logging and anything else that we allow configuring using config file.

@Praveen-Rai Could you please take a look at these docs and then file issues in the https://github.com/aspnet/Docs repo for any content that you think is missing?

Sure, i will. I just wanted to get all this clarified before i can approach docs team to act on it. I already tried tagging @guardrex, @scottaddie & @shirhatti on this issue, but i couldn't.

[guardrex]

Let's + @tdykstra ... Tom just said he was starting work on the Logging topic.

[EDIT] I've also placed the Kestrel topic update issue on Sprint 142 (next) for a quick resolution.

[Praveen-Rai]

@guardrex We must include the Kestrel endpoint configuration to :

Host ASP.NET Core on Linux with Nginx

Host ASP.NET Core on Linux with Apache

[guardrex]

Thanks @Praveen-Rai ... I'll be sure to review this and do that (+ anything else actionable here) on the update. I'll be out for a week starting Sept. 26. I should be able to get to this as soon as I get back.

[Praveen-Rai]

@guardrex For sure .. Meanwhile i'll try to consolidate everything and get it ready for you. Also i request you to please follow this discussion and also any discussion on Docs repository related to Linux and SSL. Thanks.

[Praveen-Rai]

@danroth27 @Tratcher So, i understand that there's already a some documentation regarding the configuration and we already have plans to upgrade it. Hence, for now we can skip the discussion on documentation and focus on the other issues, which are :

• SSL Certificate locations for Linux distributions.

• Kestrel logs and warnings related to Endpoint configurations & SSL certificates.

@javiercn Your opinion please.

Original Comment

[danroth27]

@Praveen-Rai As I believe @javiercn mentioned, we don't directly manage or control the location of the SSL certificates. We rely on the .NET certificate store APIs to manage all of that. If you think the certificate store APIs are using the wrong locations for the local machine or current user certificate stores on specific Linux distributions then you would need to raise that on the https://github.com/dotnet/corefx repo.

[Praveen-Rai]

@danroth27 , @javiercn
What about producing and logging warnings from Kestrel ?
In my opinion the warnings should be logged in following cases :

1. When Kestrel Endpoints are not configured.
2. When Kestrel Endpoints configuration has errors.
3. When Kestrel could not load a certificate.
4. In any case when Kestrel falls back to default, it must echo what made it to fall-back to default. And where from the defaults are being loaded.

Regarding the dotnet command, with the configuration file defining the log level to 'Debug' and using [-d|--diagnostics] as well as [-v|--verbosity], it still doesn't generate a Debug level log, Unless it is run from the application directory. Can you please check on this ?

[Praveen-Rai]

@javiercn , @danroth27 Please consider this a gentle reminder.

[javiercn]

I'm not familiar with the area. I'll let @danroth27 and @Tratcher chip in.

[Praveen-Rai]

@danroth27 , @Tratcher Any updates on the Kestrel error logging for End-Point configuration issues ?

[danroth27]

@shirhatti

[babli514]

Please I'm new to linux. I deployed a web api core/angular app to apache 2.4 ubuntu 18.04. I’m using apache as a proxy for kestrel. I installed a let's encrypt SSL certificate for my domain. When I try to run the following service:

[Unit]
Description=My App
[Service]
WorkingDirectory=/var/www/html/myapp
ExecStart=/usr/bin/dotnet /var/www/htmlmyapp/MyApp.API.dll
Restart=always
RestartSec=10
SyslogIdentifier=dotnet-demo
User=www-data
Environment=ASPNETCORE_ENVIRONMENT=Production
Environment=ASPNETCORE_HTTP_PORT=5000
Environment=ASPNETCORE_HTTPS_PORT=5001
Environment=ASPNETCORE_URLS=http://localhost:5000;https://localhost:5001
[Install]
WantedBy=multi-user.target

I get the following error message:

System.InvalidOperationException: Unable to configure HTTPS endpoint. No server certificate was specified, and the default developer certificate could not be found.
Nov 14 03:10:05 ssdnodes-11354 dotnet-demo[28065]: To generate a developer certificate run ‘dotnet dev-certs https’. To trust the certificate (Windows and macOS only) run ‘dotnet dev-certs https --trust’.
Nov 14 03:10:05 ssdnodes-11354 dotnet-demo[28065]: For more information on configuring HTTPS see https://go.microsoft.com/fwlink/?linkid=848054.

I tried to follow the advice in the error message and run dotnet dev-certs https without luck. I’m new to linux. Can anybody give me a clue about what’s wrong with my setup ?
Thanks

[Praveen-Rai]

@babli514 I've posted an answer on Stack-Overflow, that might help you. Follow this link

If you've not already done it, you need to configure Apache to use SSL. Refer this article.

[jimmi79]

@Praveen-Rai, I've been struggling with this issue for two days. Thank you so much for your Stack-Overflow solution. I'll be trying it this afternoon.

@javiercn - I'm a little lost in the haze of others coming in and out of the discussion, so I'm not sure if you are the best contact for this, but I think it's worth pointing out that using a service to launch the app is outlined in the MS Doc: Host ASP.NET Core on Linux with Nginx. This seems like a great place to either add the points @Praveen-Rai has raised or just to remove entirely.

[zeinali0]

Is there any limitation for .pfx file we use in Kestrel?
Because i'm using a pfx file which is work fine on IIS but when use it with kestrel it doesn't work:

[host]# openssl s_client -connect localhost:5000
CONNECTED(00000004)
write:errno=0
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 317 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)

curl -v -l https://localhost:5000
* Rebuilt URL to: https://localhost:5000/
*   Trying ::1...
* TCP_NODELAY set
* connect to ::1 port 5000 failed: Connection refused
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 5000 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to localhost:5000
* Closing connection 0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to localhost:5000

option.Listen(IPAddress.Any, 5000, builder =>
                        {
                            builder.UseHttps("my-cert.pfx", "mypassword");
                        });

I'm running asp.net core 3.1 on Centos8

[ghost]

Thank you for contacting us. Due to a lack of activity on this discussion issue we're closing it in an effort to keep our backlog clean. If you believe there is a concern related to the ASP.NET Core framework, which hasn't been addressed yet, please file a new issue.

This issue will be locked after 30 more days of inactivity. If you still wish to discuss this subject after then, please create a new issue!