Skip to content
This repository has been archived by the owner. It is now read-only.

rc1 INCOMPLETE_CHUNKED_ENCODING when using nginx #341

Closed
staff0rd opened this issue Nov 8, 2015 · 39 comments
Closed

rc1 INCOMPLETE_CHUNKED_ENCODING when using nginx #341

staff0rd opened this issue Nov 8, 2015 · 39 comments
Assignees
Labels
Milestone

Comments

@staff0rd
Copy link

@staff0rd staff0rd commented Nov 8, 2015

Opened this here also, but I'm getting the feeling this more specific to kestrel rc1.

I have a docker container that has an aspnet5-rc1 application inside, listening on port 5000. If I make a request directly to that port, the response is 200 and quick.

If I make a request via nginx, then the response data comes back immediately (200), but does not terminate and rather about a minute later the response fails with; (in chrome) ERR_INCOMPLETE_CHUNKED_ENCODING.

Error does not occur when using nginx with aspnet-beta7. Headers for both aspnet-beta7 and aspnet-rc1 are the same;

HTTP/1.1 200 OK
Server: nginx/1.9.6
Date: Sun, 08 Nov 2015 04:38:28 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
X-Frame-Options: SAMEORIGIN
@Tratcher
Copy link
Member

@Tratcher Tratcher commented Nov 10, 2015

I have a local repro that looks similar. If I throw an exception after the first response write then there is no chunked terminator and the connection is closed gracefully rather than reset.

            app.Run(async (context) =>
            {
                await context.Response.WriteAsync("hello world");
                throw new Exception("Test exception");
            });

The behavior is even stranger behind HttpPlatformHandler. It appears to suppress the chunking error and re-chunk the response including a terminator. Verify if it behaves correctly after kestrel changes to a reset.

@Tratcher Tratcher added the bug label Nov 10, 2015
@Tratcher Tratcher added this to the 1.0.0-rc2 milestone Nov 10, 2015
@staff0rd
Copy link
Author

@staff0rd staff0rd commented Nov 11, 2015

@Tratcher any hints as to why I'm seeing this behind nginx reverse proxy but not when accessing it directly?

@Tratcher
Copy link
Member

@Tratcher Tratcher commented Nov 11, 2015

@staff0rd That's odd. In my case the reverse proxy (HttpPlatformHandler) hid the behavior by re-chunking the response.

You just posted the headers, did any of the body make it through? Or did you do an explicit Flush to send the headers before an exception occurred?

@staff0rd
Copy link
Author

@staff0rd staff0rd commented Nov 12, 2015

@Tratcher if you browse to http://test.atqu.in you can see the issue occur. If you browse to http://test.atqu.in:5000 you'll reach the same site directly (without reverse proxy) and you'll see the issue does not occur. As you'll see, the whole site is is delivered both times, but the connection stays open on the reverse-proxy for about a minute until it finally dies with INCOMPLETE_CHUNKED_ENCODING.

The code is here and is just a fresh MVC 6 beta8 project updated to rc1, so no extra flushes occur, nor are any exceptions thrown.

@Tratcher
Copy link
Member

@Tratcher Tratcher commented Nov 16, 2015

@staff0rd Let's re-test your scenario after we get the identified bug fixed.

@pakrym pakrym added 1 - Ready and removed 2 - Working labels Nov 16, 2015
@halter73 halter73 assigned cesarblum and unassigned pakrym Nov 18, 2015
@cesarblum
Copy link
Contributor

@cesarblum cesarblum commented Nov 18, 2015

The problem here is the same as what we're facing in #368: we're not sending an RST when we expect to (ConnectionControl.End(ProduceEndType.SocketDisconnect)).

Libuv currently does not expose the underlying socket enough for us to force an RST. Someone tried to submit a PR in the past but it seems like that never got in: joyent/libuv#498

Basically what needs to be done is to set SO_LINGER on the socket with a timeout of 0. That should force an RST.

@cesarblum
Copy link
Contributor

@cesarblum cesarblum commented Nov 19, 2015

Forget my previous comment 😄

nginx makes HTTP/1.0 requests when acting as a proxy. There is a known issue for Connection: close requests (which HTTP/1.0 requests implicitly are): #406

There's an easy fix for putting Kestrel behind an nginx reverse proxy: set proxy_http_version 1.1; to have nginx send HTTP/1.1 requests.

@staff0rd
Copy link
Author

@staff0rd staff0rd commented Nov 19, 2015

This is already configuring HTTP/1.1. Looking at the response headers of test.atqu.in also confirms that HTTP/1.1 is being used - same headers as in first post above.

@cesarblum
Copy link
Contributor

@cesarblum cesarblum commented Nov 19, 2015

Can you try a different version of nginx? I repro'd your issue on 1.8 and found that using that setting fixed the problem. Then I switched to 1.9.6 and on that specific version the requests seem to take a long time to be served (but I didn't see the original issue), even when running on localhost. I don't see that happening on 1.9.7 though.

@staff0rd
Copy link
Author

@staff0rd staff0rd commented Nov 19, 2015

Updated, same symptoms - 1.3min response time.

Response headers;

HTTP/1.1 200 OK
Server: nginx/1.9.7
Date: Thu, 19 Nov 2015 23:49:28 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
@cesarblum
Copy link
Contributor

@cesarblum cesarblum commented Nov 19, 2015

I can hit that URL just fine. Wasn't that the URL where you were seeing the error?

@staff0rd
Copy link
Author

@staff0rd staff0rd commented Nov 19, 2015

Seriously? I just tried chrome + firefox + edge. Same symptoms on all - page immediately pops up, but loading doesn't stop for 1 min. However, the page imediately pops up and stops loading if I go around nginx. (http://test.atqu.in:5000)

@cesarblum
Copy link
Contributor

@cesarblum cesarblum commented Nov 20, 2015

Ohh, sorry, I didn't notice it was still loading 😞

What addresses are you binding to? I figured out why 1.9.6 was taking so long to forward the request to Kestrel. I was binding Kestrel to my IPv6 loopback interface only, but nginx was first trying to forward the request to the IPv4 interface. It took a minute to time out and try the IPv6 interface. Maybe something similar is happening in your setup?

@staff0rd
Copy link
Author

@staff0rd staff0rd commented Nov 20, 2015

Do you mean this or something on the nginx side?

@cesarblum
Copy link
Contributor

@cesarblum cesarblum commented Nov 20, 2015

Yep, that. Although you're already binding to an IPv4 interface... Can you try binding to localhost or * and see if that makes a difference?

@staff0rd
Copy link
Author

@staff0rd staff0rd commented Nov 20, 2015

Its hosted in a docker container so I don't think that binding to localhost will work. What's the syntax for *, http://*:5000?

@cesarblum
Copy link
Contributor

@cesarblum cesarblum commented Nov 20, 2015

That's correct.

@staff0rd
Copy link
Author

@staff0rd staff0rd commented Nov 20, 2015

Nah, same result;

Failed to load resource: net::ERR_INCOMPLETE_CHUNKED_ENCODING
tobz added a commit to nuclearfurnace/blockcert that referenced this issue Mar 10, 2016
As mentioned on aspnet/KestrelHttpServer#341, Kestrel seems to have a bug when behind a proxy (like when we use webpack-dev-server..) and so sending the keep-alive headers to it seems to temporarily fix the glitch until it's fixed upstream. *shrug*
@paralin
Copy link

@paralin paralin commented Apr 8, 2016

Can this be fixed for real?

Nginx sends Connection: close because when proxying requests it's best to close the connection immediately and not keep it alive.

@benaadams
Copy link
Contributor

@benaadams benaadams commented Apr 8, 2016

@paralin this is fixed in RC2

@paralin
Copy link

@paralin paralin commented Apr 8, 2016

@benaadams Any way a fix for this can be made for rc1? I don't think switching the entire stack to the AspCore dev versions is wise but I can apply a single patch to rc1.

@davidfowl
Copy link
Member

@davidfowl davidfowl commented Apr 8, 2016

See #341 (comment) for a workaround:

proxy_set_header Connection keep-alive;
@amcdnl
Copy link

@amcdnl amcdnl commented May 23, 2016

@benaadams I'm experiencing similar to this in RC2. See: #636 (comment)

@muratg muratg reopened this May 23, 2016
@muratg muratg assigned mikeharder and unassigned cesarblum May 23, 2016
@muratg muratg modified the milestones: 1.0.0, 1.0.0-rc2 May 23, 2016
@cesarblum
Copy link
Contributor

@cesarblum cesarblum commented May 23, 2016

#636 #734 seem to be the same as this one (actually #636 is about BrowserSync). Let's center the discussion here, since this was the original issue.

@amcdnl Which version of nginx are you using, and on which OS?

@halter73
Copy link
Member

@halter73 halter73 commented May 23, 2016

This issue is related to Kestrel not sending the chunked response suffix. This was fixed by e4fd91b which closed this issue and #406.

Unless people are still seeing INCOMPLETE_CHUNKED_ENCODING in chrome after the nginx timeout, I would close this.

@amcdnl
Copy link

@amcdnl amcdnl commented May 23, 2016

@CesarBS I updated the details in #636 to reflect your questions.

@mikeharder
Copy link
Contributor

@mikeharder mikeharder commented May 24, 2016

@amcdnl: Can we close this issue, or are you still seeing INCOMPLETE_CHUNKED_ENCODING with RC2?

@amcdnl
Copy link

@amcdnl amcdnl commented May 24, 2016

@mikeharder Good to go.

@cesarblum cesarblum closed this May 24, 2016
@halter73 halter73 modified the milestones: 1.0.0-rc2, 1.0.0 May 24, 2016
@halter73 halter73 assigned cesarblum and unassigned mikeharder May 24, 2016
@jmbalanag
Copy link

@jmbalanag jmbalanag commented Jul 27, 2016

Im currently encoutering this issue:

"Microsoft.AspNetCore.Server.Kestrel": "1.0.0",

@davidfowl
Copy link
Member

@davidfowl davidfowl commented Jul 27, 2016

@jmbalanag can you provide more details about your setup and provide specific repro steps?

@jmbalanag
Copy link

@jmbalanag jmbalanag commented Jul 27, 2016

This is the problem code:
[Route("api/lib_fund_source")]
public ActionResult GetFundSource()
{
return Json(db.lib_fund_source.Where(x => x.is_active == true).Select(x => new { Id = x.fund_source_id, Name = x.name }));

    }

if i remove, Where(x => x.is_active == true) it works just fine.

I think Its no longer related with kestrel.

@muratg
Copy link
Contributor

@muratg muratg commented Jul 27, 2016

@jmbalang Yeah, doesn't look like a Kestrel issue

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet