Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kestrel does not listen for https when run as a service on Ubuntu #4724

Open
Praveen-Rai opened this issue Sep 7, 2018 · 34 comments

Comments

Projects
None yet
9 participants
@Praveen-Rai
Copy link

commented Sep 7, 2018

I am trying to host an ASP.Net Core MVC application (https redirection is enabled) on Ubuntu server, using Nginx as a reverse proxy. I have created and installed a local SSL certificate using OpenSSL. When i run my application using dotnet CLI it listens on both http://localhost:5000 & https://localhost:5001, and i am able to access it on web using https (http requests are being redirect to https by Nginx).

The problem is when i try to run the as a service, it only listens on http://localhost:5000.

Here's the *.service file :

[Unit]
Description=Test ASP.Net core web application service.

[Service]
WorkingDirectory=/home/ubuntu/MyAppFolder
ExecStart=/usr/bin/dotnet/home/ubuntu/MyAppFolder/MyApplication.dll
Restart=always
# Restart service after 10 seconds if the dotnet service crashes:
RestartSec=10
SyslogIdentifier=MyApplication
User=www-data
Environment=ASPNETCORE_ENVIRONMENT=Development
Environment=DOTNET_PRINT_TELEMETRY_MESSAGE=false
Environment=ASPNETCORE_HTTPS_PORT=5001
Environment=ASPNETCORE_URLS=http://localhost:5000;https://localhost:5001

[Install]
WantedBy=multi-user.target

Environment details : ASP.Net Core 2.1.1, ASP.Net Core SDK 2.1.3, Nginx 1.14, Ubuntu 16.04

@Tratcher

This comment has been minimized.

Copy link
Member

commented Sep 7, 2018

@javiercn

This comment has been minimized.

Copy link
Member

commented Sep 7, 2018

@Praveen-Rai Thanks for contacting us.

When you run the tool on the CLI you are using your current user. When you run it as a service the user is www-data. For it to work you will need to run the dotnet-devcerts tool as the www-data user.

Alternatively you can generate an ssl cert and configure kestrel to use it. I think that is a better option if you are running the app as a service. Hope this helps.

@Praveen-Rai

This comment has been minimized.

Copy link
Author

commented Sep 7, 2018

@Tratcher @javiercn Thanks a lot for your prompt reply !!

you will need to run the dotnet-devcerts tool as the www-data user

Any hints on how to do that ?

Alternatively you can generate an ssl cert and configure kestrel to use it. I think that is a better option if you are running the app as a service.

Thanks, i will try this.

@javiercn

This comment has been minimized.

Copy link
Member

commented Sep 7, 2018

@Praveen-Rai I'm not an expert in Linux, but I believe sudo or su allows you to do that. Even if you do this I'm not confident that approach will work, feel free to try it out, but otherwise just generate a certificate and configure Kestrel to use it.

@Praveen-Rai

This comment has been minimized.

Copy link
Author

commented Sep 7, 2018

@javiercn Though i am not sure about the dotnet-devcerts, but i will surely try adding the SSL certificate to Kestrel configuration. For your information i am also looking forward to get the docs Hosting Docs updated for the same.
Again, thanks !! your inputs are valuable and it would be preferable if we get more information on dotnet-devcerts. It would be great if you could bring someone knowledgeable in dotnet-devcerts into the conversation.

@javiercn

This comment has been minimized.

Copy link
Member

commented Sep 7, 2018

@Praveen-Rai I'm the main developer on it. dotnet dev-certs will setup the developer certificate in the current user store and that's what kestrel loads. dotnet dev-certs https will make sure that there is a certificate on the current user store, but I don't recommend you use it with the approach you are taking and instead strongly encourage you to create your own certs. While technically things should work with what i've mentioned, its not one of the main scenarios for this tool.

Hope this helps.

@Praveen-Rai

This comment has been minimized.

Copy link
Author

commented Sep 9, 2018

@javiercn , @Tratcher

When you run the tool on the CLI you are using your current user. When you run it as a service the user is www-data.

Yes, i ran the service as the root user and it listens on both the ports. Though it would be a bad idea to run a service using root privileges, Right ?

Alternatively you can generate an ssl cert and configure kestrel to use it. I think that is a better option if you are running the app as a service. Hope this helps.

I tried configuring Kestrel to use the certificate using appsettings.json. Though the app started listening on ssl port ( 5001 ), but it didn't work. wget says

12.0.0.1:5001 ... connected.
Unable to establish ssl connection"

appsettings.json
"Kestrel": { "Endpoints": { "HTTPS": { "Url": "https://localhost:5001", "Certificate": { "Path": "/etc/ssl/certs/nginx-selfsigned.crt" } } } }

I suspect that the above configuration is expecting *.pfx and not *.crt. But then i am wondering how does the application works with root user access for the same certificate. Any hints ?

References :
Net Core 2.1 HTTPS Improvements Blog Post
Configuring HTTSP Blog Post
Github issue

Notes :

  • Configuring Kestrel using config file lacks documentation.
  • Seems like we dropped for configuration of Kestrel using the config file in .Net Core 2.0 and brought it back in Core 2.1 and hence is confusion and missing articles.
  • dev-certs documentation isn't complete too. Kestrel EndPoint Configuration and ASP Net Core 2.1 HTTPS reference each other, and suggest to run dotnet dev-certs https --trust command. Though when i run it i get

Specify --help for a list of available options and commands.

@Praveen-Rai

This comment has been minimized.

Copy link
Author

commented Sep 15, 2018

@javiercn , @Tratcher
I am still struggling with the issue. One more observation i would like to share here. When i configure the service to run as an Admin user, i get an info stating :

info : Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[0] User profile is available. Using /home/<username>/.aspnet/DataProtection-keys as key repository ; keys will not be encrypted at rest.

And when the application is run as service as 'www-data', i get a warning :
warn : Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[35] No XML encryptor configured, key <GUID> may be persisted to storage in unencrypted form.

Could this be the cause of issue ??

@davidfowl

This comment has been minimized.

Copy link
Member

commented Sep 15, 2018

Nope. That's normal.

@Praveen-Rai

This comment has been minimized.

Copy link
Author

commented Sep 15, 2018

To update, i've added www-data to ssl-cert group. Granted read and execute permissions on the private key directory to the ssl-cert group. Still i am facing the issue.

@javiercn

This comment has been minimized.

Copy link
Member

commented Sep 15, 2018

@Praveen-Rai Trusting the certificate is only available on Windows and Mac OS. There is no standard way of trusting the certificate on Linux.

You need to have the certificate along with the private key in a pxf format. @Tratcher we should throw an error if the certificate doesn’t have an associated private key.

You should run the tool as the www-data user, not as root. I believe you can do that with sido or su on Ubuntu, but check your rostro documentation.

Finally, running any kind of service with the development certificate in production is not supported and completely discouraged.

@danroth27 regarding docs for HTTPS can you loop in the right folks?

Hope this helps

@Praveen-Rai

This comment has been minimized.

Copy link
Author

commented Sep 16, 2018

@javiercn

Trusting the certificate is only available on Windows and Mac OS. There is no standard way of trusting the certificate on Linux.

Do you mean to say that this issue is due to trusting of certificate ? I have a doubt then, how does it work with Root user ? Doesn't the certificate trust apply machine wide ?

You need to have the certificate along with the private key in a pxf format.

Now i don't feel that's the matter, coz the application is working with root privileges without the *.pxf format.

You should run the tool as the www-data user, not as root

The service ( AspNetCore App ) is ran as 'www-data', though i couldn't directly run the dotnet command as 'www-data' ( doing a bit of search, i feel that's as designed ). And that's where the problem is.

Finally, running any kind of service with the development certificate in production is not supported and completely discouraged.

I agree, i am using this certificate for development only.

we should throw an error if the certificate doesn’t have an associated private key.

My upvote for that. We've been always restricted to MS OS concepts with .Net, but with .Net core we are introduced to MacOs & Linux and it's distributions. So, may be it would be safer to log warnings & errors more sensitively than we used to do before with .Net.

@Praveen-Rai

This comment has been minimized.

Copy link
Author

commented Sep 19, 2018

@javiercn

Finally i figured out the issue. The issue is that a developer ssl certificate is installed with dotnet SDK with the name localhost. In case of Ubuntu the certificate is located at /home/{user name} /.dotnet/corefx/cryptography/x509stores/my

I believe Kestrel just searches in the home directory of executing user, which does not exists for 'www-data', hence it couldn't locate the development certificate. Due to which it doesn't bind to default https port.

Here are the key points ( please provide your comments ) :

  1. The dotnet sdk installer must install the local development certificate to /etc/ssl/certs as it is the designated store for SSL certificates in Ubuntu ( varies as per distribution ).

  2. Kestrel must search the development certificate in the common SSL certificate store, rather in the current user home. Because not all users have home directory for e.g. 'www-data'.

  3. Whenever kestrel is loading a local development certificate installed by dotnet sdk, it must log with level "Warning". Presently it logs at level "Dbug" (even in the production environment). In production environment the log level might further be escalated.

  4. Kestrel must also produce warnings for incorrect or non-existing endpoint configuration. Kestrel does have defaults to fallback to, but it always expects the endpoints to be configured.

  5. We need documentation on configuration of kestrel ( Infact logging and there might be more ). For kestrel endpoint specification using configuration file, i could locate some content authored by @danroth27 & @Tratcher . We must formalize this content into the ASP.Net Core documentation along with other configuration options such as Logging. Also, Kestrel supports specifying the certificates in both Store-Key and File-Password format. This needs to be clearly mentioned with examples, as there might be cases where one is developing on Windows machine and Hosting on Linux.

  6. Running an application using
    dotnet <DotNetCoreApp>
    does not print detailed log, even with the use of command options and with configuration file specifying the logging options. Until the command is run from the application directory.

  7. Kestrel currently seems to support PKCS12 ( *.pfx ) only, whereas PEM seems to be de-facto for Apache, Nginx or Linux on the whole ( as per the limited knowledge i've captured recently via. web ). If that's the case, we need to highlight it in our docs. Secondly, can we start supporting PEM in Kestrel ?

@javiercn

This comment has been minimized.

Copy link
Member

commented Sep 19, 2018

  1. The dotnet sdk installer must install the local development certificate to /etc/ssl/certs as it is the designated store for SSL certificates in Ubuntu ( varies as per distribution ).

No. ASP.NET uses the current user personal certificate store which was made to work cross-platform. We don't have any plans to do anything else as there is no standard way of achieving this on linux systems and we just simply rely on .NET functionality.

  1. Kestrel must search the development certificate in the common SSL certificate store, rather in the current user home. Because not all users have home directory for e.g. 'www-data'.

No. The main purpose of the dev-certs tool is to support the development of APIs and Websites in the most common development environments and scenarios. Doing development of your app by running it as a service is not something that we consider a core development scenario (compared to for example, doing dotnet run). You can still develop using HTTPS by manually loading a certificate in pfx format using configuration.

  1. Whenever kestrel is loading a local development certificate installed by dotnet sdk, it must log with level "Warning". Presently it logs at level "Dbug" (even in the production environment). In production environment the log level might further be escalated.

I don't think so. That will produce a jarring experience during regular development.

  1. Kestrel must also produce warnings for incorrect or non-existing endpoint configuration. Kestrel does have defaults to fallback to, but it always expects the endpoints to be configured.

I'll leave this up to @Tratcher to answer.

  1. We need documentation on configuration of kestrel ( Infact logging and there might be more ). For kestrel endpoint specification using configuration file, i could locate some content authored by @danroth27 & @Tratcher . We must formalize this content into the ASP.Net Core documentation along with other configuration options such as Logging. Also, Kestrel supports specifying the certificates in both Store-Key and File-Password format. This needs to be clearly mentioned with examples, as there might be cases where one is developing on Windows machine and Hosting on Linux.

If this is the case @danroth27 can you get the doc writers involved here?

  1. Running an application using
    dotnet <DotNetCoreApp>
    does not print detailed log, even with the use of command options and with configuration file specifying the logging options. Until the command is run from the application directory.

I have no idea about this. @Tratcher or @danroth27 can comment.

  1. Kestrel currently seems to support PKCS12 ( *.pfx ) only, whereas PEM seems to be de-facto for Apache, Nginx or Linux on the whole ( as per the limited knowledge i've captured recently via. web ). If that's the case, we need to highlight it in our docs. Secondly, can we start supporting PEM in Kestrel ?

We support what .NET Core supports. You can probably file an issue on https://github.com/dotnet/corefx and they can probably tell you whether its supported or what the plans are if any.

Hope this helps.

@danroth27

This comment has been minimized.

Copy link
Member

commented Sep 19, 2018

The kestrel endpoint configuration is documented here: https://docs.microsoft.com/en-us/aspnet/core/fundamentals/servers/kestrel?view=aspnetcore-2.1#how-to-use-kestrel-in-aspnet-core-apps. It's a bit buried in the section, which we plan to fix: aspnet/AspNetCore.Docs#8496.

@Praveen-Rai

This comment has been minimized.

Copy link
Author

commented Sep 19, 2018

@javiercn I think you're missing a couple of key points here :

  1. Linux doesn't have a concept of Key Store, hence we cannot just rely on .Net functionality. What all distributions does have is a standard ( or atleast widely accepted ) locations for storing the certificates and we'll have to respect it to be able to say the .Net Core is Cross-Platform. All the certificates must be created and read from these locations only.

  2. Most of Linux servers are accessed using SSH and any process triggered using a SSH terminal is terminated upon the end of session. Even in a Windows Environment you would like to run a WebApp as a service, and not just as a program initiated by a User. Hence, no matter the platform, the ultimate destiny of any Asp.Net core app would be running as a service. And it's a good idea to develop in an environment which closely resembles your production environment.

  3. When something is important to be present in Production, my expectation is to log warning in Development and Error in Production. Presently, Kestrel does none of it and just doesn't bind to the HTTPS port, which has led to this discussion. Also, please note that even now when you run a application a lot of stuff is getting printed on the command prompt including 'info'. Adding a single line won't hurt, though i accept that we must work towards logging only the important ones.

  4. Last but not the least, .Net Framework has been here for more than a decade and it has a huge fan following ( including me ). But we all understand we missed a lot of Open-Source community stuffed, being confined to MS eco-system. Net Core is a effort to bring that all together by making .Net Framework cross-platform ( and i cannot express how happy i was with the announcement of .Net Core ). And hence we simply cannot ignore Linux ( atleast not for web applications and apis ).

Hope that addresses your concerns.

@danroth27

This comment has been minimized.

Copy link
Member

commented Sep 19, 2018

Configuring logging is documented here: https://docs.microsoft.com/en-us/aspnet/core/fundamentals/logging/?view=aspnetcore-2.1#configuration.

@Praveen-Rai Could you please take a look at these docs and then file issues in the https://github.com/aspnet/Docs repo for any content that you think is missing?

@Praveen-Rai

This comment has been minimized.

Copy link
Author

commented Sep 19, 2018

@danroth27 Sorry, i meant configuration using files. Because, you wouldn't want to hard-code the endpoint settings, SSL certificate path .. etc into your code.

I just realized today, that the configuration settings in the code can be directly mapped into the configuration file. Reference Configuration and Options

Still what i suggest is to have a document of the kind you've written in your blog

@Tratcher

This comment has been minimized.

@Praveen-Rai

This comment has been minimized.

Copy link
Author

commented Sep 19, 2018

Configuring logging is documented here: https://docs.microsoft.com/en-us/aspnet/core/fundamentals/logging/?view=aspnetcore-2.1#configuration.

Yes, i've visited this page before, but i didn't find a Kestrel configuration sections on the docs. So, i thought why not to have a module of Configuration Using File, which will include individual topics for Kestrel, Logging and anything else that we allow configuring using config file.

@Praveen-Rai Could you please take a look at these docs and then file issues in the https://github.com/aspnet/Docs repo for any content that you think is missing?

Sure, i will. I just wanted to get all this clarified before i can approach docs team to act on it. I already tried tagging @guardrex, @scottaddie & @shirhatti on this issue, but i couldn't.

@guardrex

This comment has been minimized.

Copy link
Contributor

commented Sep 19, 2018

Let's + @tdykstra ... Tom just said he was starting work on the Logging topic.

[EDIT] I've also placed the Kestrel topic update issue on Sprint 142 (next) for a quick resolution.

@Praveen-Rai

This comment has been minimized.

@guardrex

This comment has been minimized.

Copy link
Contributor

commented Sep 19, 2018

Thanks @Praveen-Rai ... I'll be sure to review this and do that (+ anything else actionable here) on the update. I'll be out for a week starting Sept. 26. I should be able to get to this as soon as I get back.

@Praveen-Rai

This comment has been minimized.

Copy link
Author

commented Sep 19, 2018

@guardrex For sure .. Meanwhile i'll try to consolidate everything and get it ready for you. Also i request you to please follow this discussion and also any discussion on Docs repository related to Linux and SSL. Thanks.

@Praveen-Rai

This comment has been minimized.

Copy link
Author

commented Sep 20, 2018

@danroth27 @Tratcher So, i understand that there's already a some documentation regarding the configuration and we already have plans to upgrade it. Hence, for now we can skip the discussion on documentation and focus on the other issues, which are :

  • SSL Certificate locations for Linux distributions.

  • Kestrel logs and warnings related to Endpoint configurations & SSL certificates.

@javiercn Your opinion please.

Original Comment

@danroth27

This comment has been minimized.

Copy link
Member

commented Sep 20, 2018

@Praveen-Rai As I believe @javiercn mentioned, we don't directly manage or control the location of the SSL certificates. We rely on the .NET certificate store APIs to manage all of that. If you think the certificate store APIs are using the wrong locations for the local machine or current user certificate stores on specific Linux distributions then you would need to raise that on the https://github.com/dotnet/corefx repo.

@Praveen-Rai

This comment has been minimized.

Copy link
Author

commented Sep 20, 2018

@danroth27 , @javiercn
What about producing and logging warnings from Kestrel ?
In my opinion the warnings should be logged in following cases :

  1. When Kestrel Endpoints are not configured.
  2. When Kestrel Endpoints configuration has errors.
  3. When Kestrel could not load a certificate.
  4. In any case when Kestrel falls back to default, it must echo what made it to fall-back to default. And where from the defaults are being loaded.

Regarding the dotnet command, with the configuration file defining the log level to 'Debug' and using [-d|--diagnostics] as well as [-v|--verbosity], it still doesn't generate a Debug level log, Unless it is run from the application directory. Can you please check on this ?

@Praveen-Rai

This comment has been minimized.

Copy link
Author

commented Sep 26, 2018

@javiercn , @danroth27 Please consider this a gentle reminder.

@javiercn

This comment has been minimized.

Copy link
Member

commented Sep 26, 2018

I'm not familiar with the area. I'll let @danroth27 and @Tratcher chip in.

@Praveen-Rai

This comment has been minimized.

Copy link
Author

commented Oct 24, 2018

@danroth27 , @Tratcher Any updates on the Kestrel error logging for End-Point configuration issues ?

@danroth27

This comment has been minimized.

Copy link
Member

commented Oct 24, 2018

@babli514

This comment has been minimized.

Copy link

commented Nov 14, 2018

Please I'm new to linux. I deployed a web api core/angular app to apache 2.4 ubuntu 18.04. I’m using apache as a proxy for kestrel. I installed a let's encrypt SSL certificate for my domain. When I try to run the following service:

[Unit]
Description=My App
[Service]
WorkingDirectory=/var/www/html/myapp
ExecStart=/usr/bin/dotnet /var/www/htmlmyapp/MyApp.API.dll
Restart=always
RestartSec=10
SyslogIdentifier=dotnet-demo
User=www-data
Environment=ASPNETCORE_ENVIRONMENT=Production
Environment=ASPNETCORE_HTTP_PORT=5000
Environment=ASPNETCORE_HTTPS_PORT=5001
Environment=ASPNETCORE_URLS=http://localhost:5000;https://localhost:5001
[Install]
WantedBy=multi-user.target

I get the following error message:

System.InvalidOperationException: Unable to configure HTTPS endpoint. No server certificate was specified, and the default developer certificate could not be found.
Nov 14 03:10:05 ssdnodes-11354 dotnet-demo[28065]: To generate a developer certificate run ‘dotnet dev-certs https’. To trust the certificate (Windows and macOS only) run ‘dotnet dev-certs https --trust’.
Nov 14 03:10:05 ssdnodes-11354 dotnet-demo[28065]: For more information on configuring HTTPS see https://go.microsoft.com/fwlink/?linkid=848054.

I tried to follow the advice in the error message and run dotnet dev-certs https without luck. I’m new to linux. Can anybody give me a clue about what’s wrong with my setup ?
Thanks

@Praveen-Rai

This comment has been minimized.

Copy link
Author

commented Nov 14, 2018

@babli514 I've posted an answer on Stack-Overflow, that might help you. Follow this link

If you've not already done it, you need to configure Apache to use SSL. Refer this article.

@aspnet-hello aspnet-hello transferred this issue from aspnet/KestrelHttpServer Dec 13, 2018

@aspnet-hello aspnet-hello added this to the Discussions milestone Dec 13, 2018

@jimmi79

This comment has been minimized.

Copy link

commented Feb 27, 2019

@Praveen-Rai, I've been struggling with this issue for two days. Thank you so much for your Stack-Overflow solution. I'll be trying it this afternoon.

@javiercn - I'm a little lost in the haze of others coming in and out of the discussion, so I'm not sure if you are the best contact for this, but I think it's worth pointing out that using a service to launch the app is outlined in the MS Doc: Host ASP.NET Core on Linux with Nginx. This seems like a great place to either add the points @Praveen-Rai has raised or just to remove entirely.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.