New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RC1 update1 works very slow on Linux/Mono #1181

Closed
yaapelsinko opened this Issue Dec 17, 2015 · 24 comments

Comments

Projects
None yet
8 participants
@yaapelsinko

I've ran into the issue with my application, so to test if it is the app or something else I've created a template ASP.NET 5 app (web app template, not empty, no authorization). It targeted solely to dnx451 framework and is built under Linux/Mono without any warnings.

To host it I've created a clean Ubuntu-1404-trusty-64-minimal installation. Server is a VM with 1 vCore and 1GB RAM.

I've installed DNVM with Mono runtime, rc1-update1, and libuv according to this: https://docs.asp.net/en/latest/getting-started/installing-on-linux.html
Installed Mono like described here: http://www.mono-project.com/docs/getting-started/install/linux/ (4.2.1 version is shown on mono -V)

Application is started by Supervisor tool, with config:

command=su -c "/root/.dnx/runtimes/dnx-mono.1.0.0-rc1-update1/bin/dnx web" root directory=/var/www/test
autorestart=true
autostart=true
stdout_logfile=/var/www/test/logs/app_std_out.log
stderr_logfile=/var/www/test/logs/app_err.log

Then requests are passed to application by nginx:

server {
listen 80;
server_name test.ru; // resolved by "hosts" file
location / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://127.0.0.1:5002;
}
}

In app_std log I see the app is started up normally and reporting it's listening on its port:

Hosting environment: Production
Now listening on: http://localhost:5002
Application started. Press Ctrl+C to shut down.

Once I made a request I usually see:

^[[32minfo^[[39m^[[37m: Microsoft.AspNet.Hosting.Internal.HostingEngine[1]^[[39m
^[[97m Request starting HTTP/1.0 GET http://test.ru/ ^[[39m
^[[32minfo^[[39m^[[37m: Microsoft.AspNet.Mvc.Controllers.ControllerActionInvoker[1]^[[39m
^[[97m Executing action method Test.Controllers.HomeController.Index with arguments () - ModelState is Valid'^[[39m

And this is it. Sometimes the page is finally served (so I see log records about calling a view, then serving linked resouces and files, etc), but response times in browser are huge (like 60 seconds for the page and 0.3-0.5 for any linked css or jpg). At the same time app log says in such cases that the request is processed in about 0.6 sec for a page and a few milliseconds for files.

(strange - page is rendered in about 0.6, and loaded in about 60, then file served in 0.003, but loaded in 0.3, i.e. always about 100x longer that it recorded in app log)

It looks like there is some huge delay between actions calling. For example, executing an action method, that loooong delay before rendering a view, then looooong delay to do something else and so on.

However, it often ends in nginx just returning 504. Sometimes page is loading only partially.

When 504, it is just 504 in nginx log. Like it wasn't nginx fault, but the application not responded and the request was timed out. Oh, and the default page nginx is showing once it installed is loading instantly.

What else, apart of reading logs, can I do to diagnose it?

I've already did the same setup recently, but with beta8 (and, well, maybe actual versions of mono/libuv/etc were a bit different), and it worked well. But not this time. :/

@davidfowl

This comment has been minimized.

Show comment
Hide comment
@davidfowl

davidfowl Dec 17, 2015

Member

Did you update mono between beta8 and rc1?

Member

davidfowl commented Dec 17, 2015

Did you update mono between beta8 and rc1?

@yaapelsinko

This comment has been minimized.

Show comment
Hide comment
@yaapelsinko

yaapelsinko Dec 17, 2015

No, this is a new installation, I mean completely, starting from a new clean VM, not update from older version.

No, this is a new installation, I mean completely, starting from a new clean VM, not update from older version.

@davidfowl

This comment has been minimized.

Show comment
Hide comment
@davidfowl

davidfowl Dec 17, 2015

Member

What version of mono were you using before and after?

Member

davidfowl commented Dec 17, 2015

What version of mono were you using before and after?

@yaapelsinko

This comment has been minimized.

Show comment
Hide comment
@yaapelsinko

yaapelsinko Dec 17, 2015

I don't remember what version exactly it was back then, but sure it was some 4.x one. Now, with current installation, it is Mono 4.2.1

Stable 4.2.1.102/6dd2d0d Thu Nov 12 09:52:44 UTC 2015, to be specific.

I don't remember what version exactly it was back then, but sure it was some 4.x one. Now, with current installation, it is Mono 4.2.1

Stable 4.2.1.102/6dd2d0d Thu Nov 12 09:52:44 UTC 2015, to be specific.

@davidfowl

This comment has been minimized.

Show comment
Hide comment
@davidfowl

davidfowl Dec 17, 2015

Member

So what you're seeing could be due to a mono change. Try using mono 4.0.5 or 4.0.5.1

Member

davidfowl commented Dec 17, 2015

So what you're seeing could be due to a mono change. Try using mono 4.0.5 or 4.0.5.1

@yaapelsinko

This comment has been minimized.

Show comment
Hide comment
@yaapelsinko

yaapelsinko Dec 17, 2015

Done that. Installed Stable 4.0.5.1.

It didn't changed anything much. I was able to see some full served pages in log, but it happened with 4.2.1 also. For example, execution log for Home/Contact (of the template ASP.NET app) is:

info: Microsoft.AspNet.Hosting.Internal.HostingEngine[1]
Request starting HTTP/1.0 GET http://test.ru/Home/Contact
info: Microsoft.AspNet.Mvc.Controllers.ControllerActionInvoker[1]
Executing action method Test.Controllers.HomeController.Contact with arguments () - ModelState is Valid'
info: Microsoft.AspNet.Mvc.ViewFeatures.ViewResultExecutor[1]
Executing ViewResult, running view at path /Views/Home/Contact.cshtml.
info: Microsoft.AspNet.Mvc.Infrastructure.MvcRouteHandler[2]
Executed action Test.Controllers.HomeController.Contact in 0.0257ms
info: Microsoft.AspNet.Hosting.Internal.HostingEngine[2]
Request finished in 0.0263ms 200 text/html; charset=utf-8

But it was never loaded in my browser. Page is either loaded or not in 60 second, so I thought it's something about nginx's timeout. So I've done some checks here.

First I've set 600 seconds timeout for the app (in case it needs more time). And while the same request (/Home/Contact) was "finished" according to app log in 0.1559ms this time (and the whole log is similar to previous example), the page is still "waiting" in the browser now while I'm writing this. It never loaded once request was "done" after a few minutes, but also it wasn't a 504 error.

Then I've set timeouts in nginx to 3 seconds only. Now I have the same result (white screen of nothing), but in 3 seconds.

Then I tried to open /Home/Index. The page is loaded after those 3 second, once nginx hit its timeout, obviously. It also served all the linked resources quite normally (in 250-300 ms, what I assume is okay when ping to the server is 151ms).

What strange is page is not completely loaded. A few last strings are missing, so there is no closing body and html tags.

So in this case it appears like the page is actually rendered, then application starts to pass it to nginx, but suddenly stops transmitting data without warning. And transmission is always cut only when close to the end, so if there is more data, the part of this data can make its way to nginx. And if there is very small amount of data to be passed, then almost nothing is passed at all. Nginx is unaware of the transmission stop and just waits it to continue, and once it hits the timeout, it just transmits all the data it received to the moment. For /Home/Contact it is just response headers. For the larger page /Home/Index it is also most of the page's html.

Weird, isn't it?

It doesn't explain why in some cases log ends on "Executing action method Test.Controllers.HomeController.Whatever" and never gets to the view, though.

Done that. Installed Stable 4.0.5.1.

It didn't changed anything much. I was able to see some full served pages in log, but it happened with 4.2.1 also. For example, execution log for Home/Contact (of the template ASP.NET app) is:

info: Microsoft.AspNet.Hosting.Internal.HostingEngine[1]
Request starting HTTP/1.0 GET http://test.ru/Home/Contact
info: Microsoft.AspNet.Mvc.Controllers.ControllerActionInvoker[1]
Executing action method Test.Controllers.HomeController.Contact with arguments () - ModelState is Valid'
info: Microsoft.AspNet.Mvc.ViewFeatures.ViewResultExecutor[1]
Executing ViewResult, running view at path /Views/Home/Contact.cshtml.
info: Microsoft.AspNet.Mvc.Infrastructure.MvcRouteHandler[2]
Executed action Test.Controllers.HomeController.Contact in 0.0257ms
info: Microsoft.AspNet.Hosting.Internal.HostingEngine[2]
Request finished in 0.0263ms 200 text/html; charset=utf-8

But it was never loaded in my browser. Page is either loaded or not in 60 second, so I thought it's something about nginx's timeout. So I've done some checks here.

First I've set 600 seconds timeout for the app (in case it needs more time). And while the same request (/Home/Contact) was "finished" according to app log in 0.1559ms this time (and the whole log is similar to previous example), the page is still "waiting" in the browser now while I'm writing this. It never loaded once request was "done" after a few minutes, but also it wasn't a 504 error.

Then I've set timeouts in nginx to 3 seconds only. Now I have the same result (white screen of nothing), but in 3 seconds.

Then I tried to open /Home/Index. The page is loaded after those 3 second, once nginx hit its timeout, obviously. It also served all the linked resources quite normally (in 250-300 ms, what I assume is okay when ping to the server is 151ms).

What strange is page is not completely loaded. A few last strings are missing, so there is no closing body and html tags.

So in this case it appears like the page is actually rendered, then application starts to pass it to nginx, but suddenly stops transmitting data without warning. And transmission is always cut only when close to the end, so if there is more data, the part of this data can make its way to nginx. And if there is very small amount of data to be passed, then almost nothing is passed at all. Nginx is unaware of the transmission stop and just waits it to continue, and once it hits the timeout, it just transmits all the data it received to the moment. For /Home/Contact it is just response headers. For the larger page /Home/Index it is also most of the page's html.

Weird, isn't it?

It doesn't explain why in some cases log ends on "Executing action method Test.Controllers.HomeController.Whatever" and never gets to the view, though.

@davidfowl

This comment has been minimized.

Show comment
Hide comment
@davidfowl

davidfowl Dec 17, 2015

Member

Sounds like a mono bug to me. Try with coreclr and see if that works. If it does I'd recommend filing an issue on mono with repro steps so that they can investigate the issue.

Member

davidfowl commented Dec 17, 2015

Sounds like a mono bug to me. Try with coreclr and see if that works. If it does I'd recommend filing an issue on mono with repro steps so that they can investigate the issue.

@yaapelsinko

This comment has been minimized.

Show comment
Hide comment
@yaapelsinko

yaapelsinko Dec 17, 2015

Nope, I've rebuilt the template under Core 5.0 and tested it.

Had to get rid of:
"Microsoft.AspNet.Diagnostics.Entity": "7.0.0-rc1-final",
"Microsoft.ApplicationInsights.AspNet": "1.0.0-rc1",
"Microsoft.VisualStudio.Web.BrowserLink.Loader": "14.0.0-rc1-final"

There were some issues related to one of them when executing a page (maybe coreclr doesn't fully support it on linux?), so I decided to get rid of all debugging stuff just in case. Let's pretend we are in production after all! Installed coreclr-linux-x64-rc1-update1 and all the prerequisites according to https://docs.asp.net/en/latest/getting-started/installing-on-linux.html

The behavior didn't changed of what I've already described, though.

Nope, I've rebuilt the template under Core 5.0 and tested it.

Had to get rid of:
"Microsoft.AspNet.Diagnostics.Entity": "7.0.0-rc1-final",
"Microsoft.ApplicationInsights.AspNet": "1.0.0-rc1",
"Microsoft.VisualStudio.Web.BrowserLink.Loader": "14.0.0-rc1-final"

There were some issues related to one of them when executing a page (maybe coreclr doesn't fully support it on linux?), so I decided to get rid of all debugging stuff just in case. Let's pretend we are in production after all! Installed coreclr-linux-x64-rc1-update1 and all the prerequisites according to https://docs.asp.net/en/latest/getting-started/installing-on-linux.html

The behavior didn't changed of what I've already described, though.

@yaapelsinko

This comment has been minimized.

Show comment
Hide comment
@yaapelsinko

yaapelsinko Dec 17, 2015

Also, while testing I've ran into 504 once, and this is what I've seen in app log:

fail: Microsoft.AspNet.Server.Kestrel[13]
An unhandled exception was thrown by the application.
Microsoft.AspNet.Server.Kestrel.Networking.UvException: Error -32 EPIPE broken pipe
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.AspNet.Server.Kestrel.Http.Frame.d__85.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.AspNet.Mvc.HttpResponseStreamWriter.d__25.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.AspNet.Mvc.ViewFeatures.ViewExecutor.d__14.MoveNext()

Usually, however, it just continues to serve incomplete responses without any fails or warnings.

Also, while testing I've ran into 504 once, and this is what I've seen in app log:

fail: Microsoft.AspNet.Server.Kestrel[13]
An unhandled exception was thrown by the application.
Microsoft.AspNet.Server.Kestrel.Networking.UvException: Error -32 EPIPE broken pipe
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.AspNet.Server.Kestrel.Http.Frame.d__85.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.AspNet.Mvc.HttpResponseStreamWriter.d__25.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.AspNet.Mvc.ViewFeatures.ViewExecutor.d__14.MoveNext()

Usually, however, it just continues to serve incomplete responses without any fails or warnings.

@davidfowl

This comment has been minimized.

Show comment
Hide comment
@davidfowl

davidfowl Dec 17, 2015

Member

If you can provide a repository with an application and instructions on how to reproduce the problem on CoreCLR, we can take a look.

/cc @halter73 @cesarbs

Member

davidfowl commented Dec 17, 2015

If you can provide a repository with an application and instructions on how to reproduce the problem on CoreCLR, we can take a look.

/cc @halter73 @cesarbs

@yaapelsinko

This comment has been minimized.

Show comment
Hide comment
@yaapelsinko

yaapelsinko Dec 17, 2015

Sure. Or I could PM you with root access to the server so that you could gather any information you need on the actual installation (in case any specific versions are important). It is a closed sandbox VM with no important data on it, so there are no security concerns. Would you prefer server or repo?

Sure. Or I could PM you with root access to the server so that you could gather any information you need on the actual installation (in case any specific versions are important). It is a closed sandbox VM with no important data on it, so there are no security concerns. Would you prefer server or repo?

@davidfowl

This comment has been minimized.

Show comment
Hide comment
@davidfowl

davidfowl Dec 17, 2015

Member

Repo, that way we can know if it's environmental. If we get to that stage and we only see the issue on that one machine then we can go deeper.

Member

davidfowl commented Dec 17, 2015

Repo, that way we can know if it's environmental. If we get to that stage and we only see the issue on that one machine then we can go deeper.

@yaapelsinko

This comment has been minimized.

Show comment
Hide comment
@yaapelsinko

yaapelsinko Dec 17, 2015

Then, here is the template source archive: https://drive.google.com/file/d/0B9avZq1tZylgdElmektaWFlOWkU/view?usp=sharing

The OS on my VM is Ubuntu 14.04.3 LTS.

CoreCLR and the prerequisites are installed according to the doc: https://docs.asp.net/en/latest/getting-started/installing-on-linux.html

Also you may need nginx (version 1.4.6-1ubuntu3.3 in my case) Because VM is not local for me, I'm accessing it via internet and thus can't access directly to the application port. So nginx could be a part of the issue...

Its proxy configuration can be like the following (added at the end of the "default" config):

server {
    listen 8080; // not to bother with domain names

    proxy_connect_timeout       10;
    proxy_send_timeout          10;
    proxy_read_timeout          10;
    send_timeout                10;

    location / {
        proxy_set_header    Host    $host;
        proxy_set_header    X-Real-IP    $remote_addr;
        proxy_set_header    X-Forwarded-For    $proxy_add_x_forwarded_for;
        proxy_pass    http://127.0.0.1:5002; // or your actual port, in repo it is set to 5002
    }
}

Basically, this is the minimal configuration. Then it must be enough to upload the project, then
dnu restore
dnu build
dnx web

Then just open the page. If you have to wait 10 seconds (according to nginx proxy timeout settings) before you see any (lack of) result in the browser, then the issue is likely reproduced.

Then, here is the template source archive: https://drive.google.com/file/d/0B9avZq1tZylgdElmektaWFlOWkU/view?usp=sharing

The OS on my VM is Ubuntu 14.04.3 LTS.

CoreCLR and the prerequisites are installed according to the doc: https://docs.asp.net/en/latest/getting-started/installing-on-linux.html

Also you may need nginx (version 1.4.6-1ubuntu3.3 in my case) Because VM is not local for me, I'm accessing it via internet and thus can't access directly to the application port. So nginx could be a part of the issue...

Its proxy configuration can be like the following (added at the end of the "default" config):

server {
    listen 8080; // not to bother with domain names

    proxy_connect_timeout       10;
    proxy_send_timeout          10;
    proxy_read_timeout          10;
    send_timeout                10;

    location / {
        proxy_set_header    Host    $host;
        proxy_set_header    X-Real-IP    $remote_addr;
        proxy_set_header    X-Forwarded-For    $proxy_add_x_forwarded_for;
        proxy_pass    http://127.0.0.1:5002; // or your actual port, in repo it is set to 5002
    }
}

Basically, this is the minimal configuration. Then it must be enough to upload the project, then
dnu restore
dnu build
dnx web

Then just open the page. If you have to wait 10 seconds (according to nginx proxy timeout settings) before you see any (lack of) result in the browser, then the issue is likely reproduced.

@tenowg

This comment has been minimized.

Show comment
Hide comment
@tenowg

tenowg Dec 17, 2015

Regrettably I was having this issue some time ago (days, a week) on
Windows, the page load would just stop for several seconds, at the same
point mentioned above, the line mentioning Model state is valid...
sometimes for a few seconds, sometimes for a minute. Unfortunately I don't
have any logs anymore, but I had figured it was my setup with VS as it was
hanging on javascript parsing as well, so I ported over to VSC... no
problems since.

So likely the issue is some rare conditional problem, or combination of
uncommon issues and it not likely only related to Linux

On Thu, Dec 17, 2015 at 4:51 AM, yaapelsinko notifications@github.com
wrote:

Then, here is the template source archive:
https://drive.google.com/file/d/0B9avZq1tZylgdElmektaWFlOWkU/view?usp=sharing

The OS on my VM is Ubuntu 14.04.3 LTS.

CoreCLR and the prerequisites are installed according to the doc:
https://docs.asp.net/en/latest/getting-started/installing-on-linux.html

Also you may need nginx (version 1.4.6-1ubuntu3.3 in my case) Because VM
is not local for me, I'm accessing it via internet and thus can't access
directly to the application port. So nginx could be a part of the issue...

Its proxy configuration can be like the following (added at the end of the
"default" config):

server {
listen 8080; // not to bother with domain names

proxy_connect_timeout 10;
proxy_send_timeout 10;
proxy_read_timeout 10;
send_timeout 10;

location / {

proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://127.0.0.1:5002; // or your actual port, in repo it is
set to 5002
}
}

Basically, this is the minimal configuration. Then it must be enough to
upload the project, then
dnu restore
dnu build
dnx web

Then just open the page. If you have to wait 10 seconds (according to
nginx proxy timeout settings) before you see any (lack of) result in the
browser, then the issue is likely reproduced.


Reply to this email directly or view it on GitHub
#1181 (comment).

tenowg commented Dec 17, 2015

Regrettably I was having this issue some time ago (days, a week) on
Windows, the page load would just stop for several seconds, at the same
point mentioned above, the line mentioning Model state is valid...
sometimes for a few seconds, sometimes for a minute. Unfortunately I don't
have any logs anymore, but I had figured it was my setup with VS as it was
hanging on javascript parsing as well, so I ported over to VSC... no
problems since.

So likely the issue is some rare conditional problem, or combination of
uncommon issues and it not likely only related to Linux

On Thu, Dec 17, 2015 at 4:51 AM, yaapelsinko notifications@github.com
wrote:

Then, here is the template source archive:
https://drive.google.com/file/d/0B9avZq1tZylgdElmektaWFlOWkU/view?usp=sharing

The OS on my VM is Ubuntu 14.04.3 LTS.

CoreCLR and the prerequisites are installed according to the doc:
https://docs.asp.net/en/latest/getting-started/installing-on-linux.html

Also you may need nginx (version 1.4.6-1ubuntu3.3 in my case) Because VM
is not local for me, I'm accessing it via internet and thus can't access
directly to the application port. So nginx could be a part of the issue...

Its proxy configuration can be like the following (added at the end of the
"default" config):

server {
listen 8080; // not to bother with domain names

proxy_connect_timeout 10;
proxy_send_timeout 10;
proxy_read_timeout 10;
send_timeout 10;

location / {

proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://127.0.0.1:5002; // or your actual port, in repo it is
set to 5002
}
}

Basically, this is the minimal configuration. Then it must be enough to
upload the project, then
dnu restore
dnu build
dnx web

Then just open the page. If you have to wait 10 seconds (according to
nginx proxy timeout settings) before you see any (lack of) result in the
browser, then the issue is likely reproduced.


Reply to this email directly or view it on GitHub
#1181 (comment).

@yaapelsinko

This comment has been minimized.

Show comment
Hide comment
@yaapelsinko

yaapelsinko Dec 17, 2015

Hm, the same code is working well on my Windows dev machine.

Hm, the same code is working well on my Windows dev machine.

@Tragetaschen

This comment has been minimized.

Show comment
Hide comment
@Tragetaschen

Tragetaschen Dec 18, 2015

I'm seeing these kinds of "hangs" on my embedded Linux platform (Mono 4.0.4) with Kestrel serving port 80 directly and also on Windows with IIS Express + Kestrel.

Moments ago serving a static css file took three seconds instead of the usual milliseconds on a Surface Pro 4.

I'm seeing these kinds of "hangs" on my embedded Linux platform (Mono 4.0.4) with Kestrel serving port 80 directly and also on Windows with IIS Express + Kestrel.

Moments ago serving a static css file took three seconds instead of the usual milliseconds on a Surface Pro 4.

@yaapelsinko

This comment has been minimized.

Show comment
Hide comment
@yaapelsinko

yaapelsinko Dec 18, 2015

I've reproduced the issue on another VM. What I described earlier was on a hoster's VM, so, who knows what exactly they've made with their system images.

I have an Ubuntu VM (14.04 again) in my Hyper-V, installed from Ubuntu Server ISO image, so I configured it the same way I did with the hoster's VM. Mono 4.2.1, DNX rc1-update1 and so on. Mono because I don't want to deal with possible unsupported stuff on linux coreclr. The app is still an unedited ASP.NET 5 Web App template without authentication. Then again I'm creating a proxy with nginx, calling dnx web, and the app hangs or give an incomplete response.

More of that, at some moment it seemed to suspend completely on
Executing action method TestApp.Controllers.HomeController.Index with arguments () - ModelState is Valid'

So I can't even exit with Ctrl+C.

Um, if the issue is so repetitive, I assume your board must already be full of reports. But it isn't. o_O

Tragetaschen, how did you exposed the app on the 80 port directly from Kestrel? As far as I know it allows only local ports (so one can make it work on 127.0.0.1:80, but not on 192.168.0.1:80), so you need a way to reroute requests some way, isn't it?

I've reproduced the issue on another VM. What I described earlier was on a hoster's VM, so, who knows what exactly they've made with their system images.

I have an Ubuntu VM (14.04 again) in my Hyper-V, installed from Ubuntu Server ISO image, so I configured it the same way I did with the hoster's VM. Mono 4.2.1, DNX rc1-update1 and so on. Mono because I don't want to deal with possible unsupported stuff on linux coreclr. The app is still an unedited ASP.NET 5 Web App template without authentication. Then again I'm creating a proxy with nginx, calling dnx web, and the app hangs or give an incomplete response.

More of that, at some moment it seemed to suspend completely on
Executing action method TestApp.Controllers.HomeController.Index with arguments () - ModelState is Valid'

So I can't even exit with Ctrl+C.

Um, if the issue is so repetitive, I assume your board must already be full of reports. But it isn't. o_O

Tragetaschen, how did you exposed the app on the 80 port directly from Kestrel? As far as I know it allows only local ports (so one can make it work on 127.0.0.1:80, but not on 192.168.0.1:80), so you need a way to reroute requests some way, isn't it?

@Tragetaschen

This comment has been minimized.

Show comment
Hide comment
@Tragetaschen

Tragetaschen Dec 18, 2015

Kestrel binds to IPAddress.IPv6Any when anything but localhost is specified: "web": "Microsoft.AspNet.Server.Kestrel --server.urls http://*:80/". Keep in mind though, that this requires some sort of super user permissions to bind to that port. You have to decide if this is a good thing in your environment.

Kestrel binds to IPAddress.IPv6Any when anything but localhost is specified: "web": "Microsoft.AspNet.Server.Kestrel --server.urls http://*:80/". Keep in mind though, that this requires some sort of super user permissions to bind to that port. You have to decide if this is a good thing in your environment.

@yaapelsinko

This comment has been minimized.

Show comment
Hide comment
@yaapelsinko

yaapelsinko Dec 19, 2015

I've done yet another test. This time I made a VM with Ubuntu Desktop so I can access the app locally with the browser.

Also installed only CoreCLR runtime to exclude mono.

It appeared that while I'm accessing the app directly (127.0.0.1:5002), it works just right. But once I'm introducing nginx to the request chain, the issue is appearing again. No matter if I'm accessing 192.168.0.1:8080 (the nginx proxy port) from the outside of VM, or just 127.0.0.1:8080 from the browser directly in VM.

So may be it is the nginx to blame, but it is pretty mature tool and it was successfully used that way earlier. So maybe it something about Nginx-Kestrel interactions?

Also, updating nginx 1.4.6 to 1.8.0 didn't help.

I've done yet another test. This time I made a VM with Ubuntu Desktop so I can access the app locally with the browser.

Also installed only CoreCLR runtime to exclude mono.

It appeared that while I'm accessing the app directly (127.0.0.1:5002), it works just right. But once I'm introducing nginx to the request chain, the issue is appearing again. No matter if I'm accessing 192.168.0.1:8080 (the nginx proxy port) from the outside of VM, or just 127.0.0.1:8080 from the browser directly in VM.

So may be it is the nginx to blame, but it is pretty mature tool and it was successfully used that way earlier. So maybe it something about Nginx-Kestrel interactions?

Also, updating nginx 1.4.6 to 1.8.0 didn't help.

@yaapelsinko

This comment has been minimized.

Show comment
Hide comment
@yaapelsinko

yaapelsinko Dec 19, 2015

Alrighty, here is the conclusion.

As it seems to be nginx-kestrel reverse-proxy interaction issue, I've narrowed the search and found the issue: aspnet/KestrelHttpServer#468

For me, the solution was also to add "proxy_set_header Connection keep-alive" to the proxy configuration.

Also I don't understand then, why there were definite hangs/suspensions/freezes after app logged:
Executing action method TestApp.Controllers.HomeController.Index with arguments () - ModelState is Valid'

Also I've found something that could be relevant: http://serverfault.com/questions/524878/nginx-as-reverse-proxy-causes-failure-to-respond-to-certain-post-requests

The problem: ASP.NET development server sends a 100 Continue response even when there was no Expect: 100-continue in the request. Nginx will issue a request with no Expect header; ASP.NET will respond with one, then receive the rest of the request, and then send the actual response. Nginx, however, will only forward the 100 Continue part. It will not forward the actual response; that gets swallowed, at least on nginx 1.4.2.

Don't know then if there is a point to continue this issue here. I'll add to the issue in kestrel's branch. But again, I'll remind that back then with beta8 version there was no such issue.

Alrighty, here is the conclusion.

As it seems to be nginx-kestrel reverse-proxy interaction issue, I've narrowed the search and found the issue: aspnet/KestrelHttpServer#468

For me, the solution was also to add "proxy_set_header Connection keep-alive" to the proxy configuration.

Also I don't understand then, why there were definite hangs/suspensions/freezes after app logged:
Executing action method TestApp.Controllers.HomeController.Index with arguments () - ModelState is Valid'

Also I've found something that could be relevant: http://serverfault.com/questions/524878/nginx-as-reverse-proxy-causes-failure-to-respond-to-certain-post-requests

The problem: ASP.NET development server sends a 100 Continue response even when there was no Expect: 100-continue in the request. Nginx will issue a request with no Expect header; ASP.NET will respond with one, then receive the rest of the request, and then send the actual response. Nginx, however, will only forward the 100 Continue part. It will not forward the actual response; that gets swallowed, at least on nginx 1.4.2.

Don't know then if there is a point to continue this issue here. I'll add to the issue in kestrel's branch. But again, I'll remind that back then with beta8 version there was no such issue.

@halter73

This comment has been minimized.

Show comment
Hide comment
@halter73

halter73 Dec 28, 2015

Member

I think aspnet/KestrelHttpServer#406 is the relevant issue that is affecting non keep-alive requests (i.e. connection: close). This should be fixed in rc2.

@yaapelsinko The serverfault link is referencing an old development server. I don't think there is an issue with Kestrel sending unexpected 100-continue responses.

Member

halter73 commented Dec 28, 2015

I think aspnet/KestrelHttpServer#406 is the relevant issue that is affecting non keep-alive requests (i.e. connection: close). This should be fixed in rc2.

@yaapelsinko The serverfault link is referencing an old development server. I don't think there is an issue with Kestrel sending unexpected 100-continue responses.

@halter73 halter73 closed this Dec 28, 2015

@in10se

This comment has been minimized.

Show comment
Hide comment
@in10se

in10se Jan 6, 2016

I ran into the same issue but adding "proxy_set_header Connection keep-alive;" didn't work for me. What did work was replacing NGINX with Apache2. Say what you want, Apache2 is still the best overall.

in10se commented Jan 6, 2016

I ran into the same issue but adding "proxy_set_header Connection keep-alive;" didn't work for me. What did work was replacing NGINX with Apache2. Say what you want, Apache2 is still the best overall.

@cesarbs

This comment has been minimized.

Show comment
Hide comment
@cesarbs

cesarbs Jan 7, 2016

@in10se Try setting proxy_http_version 1.1; as well.

cesarbs commented Jan 7, 2016

@in10se Try setting proxy_http_version 1.1; as well.

@tom-clark-zonal

This comment has been minimized.

Show comment
Hide comment
@tom-clark-zonal

tom-clark-zonal Jan 26, 2016

@yaapelsinko Thanks! Adding the keep alive header in Nginx worked for me too. Nice and fast now.

@yaapelsinko Thanks! Adding the keep alive header in Nginx worked for me too. Nice and fast now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment