Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
RC1 update1 works very slow on Linux/Mono #1181
I've ran into the issue with my application, so to test if it is the app or something else I've created a template ASP.NET 5 app (web app template, not empty, no authorization). It targeted solely to dnx451 framework and is built under Linux/Mono without any warnings.
To host it I've created a clean Ubuntu-1404-trusty-64-minimal installation. Server is a VM with 1 vCore and 1GB RAM.
I've installed DNVM with Mono runtime, rc1-update1, and libuv according to this: https://docs.asp.net/en/latest/getting-started/installing-on-linux.html
Application is started by Supervisor tool, with config:
command=su -c "/root/.dnx/runtimes/dnx-mono.1.0.0-rc1-update1/bin/dnx web" root directory=/var/www/test
Then requests are passed to application by nginx:
In app_std log I see the app is started up normally and reporting it's listening on its port:
Hosting environment: Production
Once I made a request I usually see:
And this is it. Sometimes the page is finally served (so I see log records about calling a view, then serving linked resouces and files, etc), but response times in browser are huge (like 60 seconds for the page and 0.3-0.5 for any linked css or jpg). At the same time app log says in such cases that the request is processed in about 0.6 sec for a page and a few milliseconds for files.
(strange - page is rendered in about 0.6, and loaded in about 60, then file served in 0.003, but loaded in 0.3, i.e. always about 100x longer that it recorded in app log)
It looks like there is some huge delay between actions calling. For example, executing an action method, that loooong delay before rendering a view, then looooong delay to do something else and so on.
However, it often ends in nginx just returning 504. Sometimes page is loading only partially.
When 504, it is just 504 in nginx log. Like it wasn't nginx fault, but the application not responded and the request was timed out. Oh, and the default page nginx is showing once it installed is loading instantly.
What else, apart of reading logs, can I do to diagnose it?
I've already did the same setup recently, but with beta8 (and, well, maybe actual versions of mono/libuv/etc were a bit different), and it worked well. But not this time. :/
Done that. Installed Stable 22.214.171.124.
It didn't changed anything much. I was able to see some full served pages in log, but it happened with 4.2.1 also. For example, execution log for Home/Contact (of the template ASP.NET app) is:
But it was never loaded in my browser. Page is either loaded or not in 60 second, so I thought it's something about nginx's timeout. So I've done some checks here.
First I've set 600 seconds timeout for the app (in case it needs more time). And while the same request (/Home/Contact) was "finished" according to app log in 0.1559ms this time (and the whole log is similar to previous example), the page is still "waiting" in the browser now while I'm writing this. It never loaded once request was "done" after a few minutes, but also it wasn't a 504 error.
Then I've set timeouts in nginx to 3 seconds only. Now I have the same result (white screen of nothing), but in 3 seconds.
Then I tried to open /Home/Index. The page is loaded after those 3 second, once nginx hit its timeout, obviously. It also served all the linked resources quite normally (in 250-300 ms, what I assume is okay when ping to the server is 151ms).
What strange is page is not completely loaded. A few last strings are missing, so there is no closing body and html tags.
So in this case it appears like the page is actually rendered, then application starts to pass it to nginx, but suddenly stops transmitting data without warning. And transmission is always cut only when close to the end, so if there is more data, the part of this data can make its way to nginx. And if there is very small amount of data to be passed, then almost nothing is passed at all. Nginx is unaware of the transmission stop and just waits it to continue, and once it hits the timeout, it just transmits all the data it received to the moment. For /Home/Contact it is just response headers. For the larger page /Home/Index it is also most of the page's html.
Weird, isn't it?
It doesn't explain why in some cases log ends on "Executing action method Test.Controllers.HomeController.Whatever" and never gets to the view, though.
Nope, I've rebuilt the template under Core 5.0 and tested it.
Had to get rid of:
There were some issues related to one of them when executing a page (maybe coreclr doesn't fully support it on linux?), so I decided to get rid of all debugging stuff just in case. Let's pretend we are in production after all! Installed coreclr-linux-x64-rc1-update1 and all the prerequisites according to https://docs.asp.net/en/latest/getting-started/installing-on-linux.html
The behavior didn't changed of what I've already described, though.
Also, while testing I've ran into 504 once, and this is what I've seen in app log:
Usually, however, it just continues to serve incomplete responses without any fails or warnings.
Sure. Or I could PM you with root access to the server so that you could gather any information you need on the actual installation (in case any specific versions are important). It is a closed sandbox VM with no important data on it, so there are no security concerns. Would you prefer server or repo?
Then, here is the template source archive: https://drive.google.com/file/d/0B9avZq1tZylgdElmektaWFlOWkU/view?usp=sharing
The OS on my VM is Ubuntu 14.04.3 LTS.
CoreCLR and the prerequisites are installed according to the doc: https://docs.asp.net/en/latest/getting-started/installing-on-linux.html
Also you may need nginx (version 1.4.6-1ubuntu3.3 in my case) Because VM is not local for me, I'm accessing it via internet and thus can't access directly to the application port. So nginx could be a part of the issue...
Its proxy configuration can be like the following (added at the end of the "default" config):
Basically, this is the minimal configuration. Then it must be enough to upload the project, then
Then just open the page. If you have to wait 10 seconds (according to nginx proxy timeout settings) before you see any (lack of) result in the browser, then the issue is likely reproduced.
Regrettably I was having this issue some time ago (days, a week) on
So likely the issue is some rare conditional problem, or combination of
On Thu, Dec 17, 2015 at 4:51 AM, yaapelsinko email@example.com
I'm seeing these kinds of "hangs" on my embedded Linux platform (Mono 4.0.4) with Kestrel serving port 80 directly and also on Windows with IIS Express + Kestrel.
Moments ago serving a static css file took three seconds instead of the usual milliseconds on a Surface Pro 4.
I've reproduced the issue on another VM. What I described earlier was on a hoster's VM, so, who knows what exactly they've made with their system images.
I have an Ubuntu VM (14.04 again) in my Hyper-V, installed from Ubuntu Server ISO image, so I configured it the same way I did with the hoster's VM. Mono 4.2.1, DNX rc1-update1 and so on. Mono because I don't want to deal with possible unsupported stuff on linux coreclr. The app is still an unedited ASP.NET 5 Web App template without authentication. Then again I'm creating a proxy with nginx, calling dnx web, and the app hangs or give an incomplete response.
More of that, at some moment it seemed to suspend completely on
So I can't even exit with Ctrl+C.
Um, if the issue is so repetitive, I assume your board must already be full of reports. But it isn't. o_O
Tragetaschen, how did you exposed the app on the 80 port directly from Kestrel? As far as I know it allows only local ports (so one can make it work on 127.0.0.1:80, but not on 192.168.0.1:80), so you need a way to reroute requests some way, isn't it?
Kestrel binds to
I've done yet another test. This time I made a VM with Ubuntu Desktop so I can access the app locally with the browser.
Also installed only CoreCLR runtime to exclude mono.
It appeared that while I'm accessing the app directly (127.0.0.1:5002), it works just right. But once I'm introducing nginx to the request chain, the issue is appearing again. No matter if I'm accessing 192.168.0.1:8080 (the nginx proxy port) from the outside of VM, or just 127.0.0.1:8080 from the browser directly in VM.
So may be it is the nginx to blame, but it is pretty mature tool and it was successfully used that way earlier. So maybe it something about Nginx-Kestrel interactions?
Also, updating nginx 1.4.6 to 1.8.0 didn't help.
Alrighty, here is the conclusion.
As it seems to be nginx-kestrel reverse-proxy interaction issue, I've narrowed the search and found the issue: aspnet/KestrelHttpServer#468
For me, the solution was also to add "proxy_set_header Connection keep-alive" to the proxy configuration.
Also I don't understand then, why there were definite hangs/suspensions/freezes after app logged:
Also I've found something that could be relevant: http://serverfault.com/questions/524878/nginx-as-reverse-proxy-causes-failure-to-respond-to-certain-post-requests
The problem: ASP.NET development server sends a 100 Continue response even when there was no Expect: 100-continue in the request. Nginx will issue a request with no Expect header; ASP.NET will respond with one, then receive the rest of the request, and then send the actual response. Nginx, however, will only forward the 100 Continue part. It will not forward the actual response; that gets swallowed, at least on nginx 1.4.2.
Don't know then if there is a point to continue this issue here. I'll add to the issue in kestrel's branch. But again, I'll remind that back then with beta8 version there was no such issue.
I think aspnet/KestrelHttpServer#406 is the relevant issue that is affecting non keep-alive requests (i.e. connection: close). This should be fixed in rc2.
@yaapelsinko The serverfault link is referencing an old development server. I don't think there is an issue with Kestrel sending unexpected 100-continue responses.