New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rc1 INCOMPLETE_CHUNKED_ENCODING when using nginx #341

Closed
staff0rd opened this Issue Nov 8, 2015 · 39 comments

Comments

Projects
None yet
@staff0rd

staff0rd commented Nov 8, 2015

Opened this here also, but I'm getting the feeling this more specific to kestrel rc1.

I have a docker container that has an aspnet5-rc1 application inside, listening on port 5000. If I make a request directly to that port, the response is 200 and quick.

If I make a request via nginx, then the response data comes back immediately (200), but does not terminate and rather about a minute later the response fails with; (in chrome) ERR_INCOMPLETE_CHUNKED_ENCODING.

Error does not occur when using nginx with aspnet-beta7. Headers for both aspnet-beta7 and aspnet-rc1 are the same;

HTTP/1.1 200 OK
Server: nginx/1.9.6
Date: Sun, 08 Nov 2015 04:38:28 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
X-Frame-Options: SAMEORIGIN
@Tratcher

This comment has been minimized.

Member

Tratcher commented Nov 10, 2015

I have a local repro that looks similar. If I throw an exception after the first response write then there is no chunked terminator and the connection is closed gracefully rather than reset.

            app.Run(async (context) =>
            {
                await context.Response.WriteAsync("hello world");
                throw new Exception("Test exception");
            });

The behavior is even stranger behind HttpPlatformHandler. It appears to suppress the chunking error and re-chunk the response including a terminator. Verify if it behaves correctly after kestrel changes to a reset.

@Tratcher Tratcher added the bug label Nov 10, 2015

@Tratcher Tratcher added this to the 1.0.0-rc2 milestone Nov 10, 2015

@staff0rd

This comment has been minimized.

staff0rd commented Nov 11, 2015

@Tratcher any hints as to why I'm seeing this behind nginx reverse proxy but not when accessing it directly?

@Tratcher

This comment has been minimized.

Member

Tratcher commented Nov 11, 2015

@staff0rd That's odd. In my case the reverse proxy (HttpPlatformHandler) hid the behavior by re-chunking the response.

You just posted the headers, did any of the body make it through? Or did you do an explicit Flush to send the headers before an exception occurred?

@staff0rd

This comment has been minimized.

staff0rd commented Nov 12, 2015

@Tratcher if you browse to http://test.atqu.in you can see the issue occur. If you browse to http://test.atqu.in:5000 you'll reach the same site directly (without reverse proxy) and you'll see the issue does not occur. As you'll see, the whole site is is delivered both times, but the connection stays open on the reverse-proxy for about a minute until it finally dies with INCOMPLETE_CHUNKED_ENCODING.

The code is here and is just a fresh MVC 6 beta8 project updated to rc1, so no extra flushes occur, nor are any exceptions thrown.

@Tratcher

This comment has been minimized.

Member

Tratcher commented Nov 16, 2015

@staff0rd Let's re-test your scenario after we get the identified bug fixed.

@pakrym pakrym added 1 - Ready and removed 2 - Working labels Nov 16, 2015

@halter73 halter73 assigned cesarbs and unassigned pakrym Nov 18, 2015

@cesarbs

This comment has been minimized.

Contributor

cesarbs commented Nov 18, 2015

The problem here is the same as what we're facing in #368: we're not sending an RST when we expect to (ConnectionControl.End(ProduceEndType.SocketDisconnect)).

Libuv currently does not expose the underlying socket enough for us to force an RST. Someone tried to submit a PR in the past but it seems like that never got in: joyent/libuv#498

Basically what needs to be done is to set SO_LINGER on the socket with a timeout of 0. That should force an RST.

@cesarbs

This comment has been minimized.

Contributor

cesarbs commented Nov 19, 2015

Forget my previous comment 😄

nginx makes HTTP/1.0 requests when acting as a proxy. There is a known issue for Connection: close requests (which HTTP/1.0 requests implicitly are): #406

There's an easy fix for putting Kestrel behind an nginx reverse proxy: set proxy_http_version 1.1; to have nginx send HTTP/1.1 requests.

@staff0rd

This comment has been minimized.

staff0rd commented Nov 19, 2015

This is already configuring HTTP/1.1. Looking at the response headers of test.atqu.in also confirms that HTTP/1.1 is being used - same headers as in first post above.

@cesarbs

This comment has been minimized.

Contributor

cesarbs commented Nov 19, 2015

Can you try a different version of nginx? I repro'd your issue on 1.8 and found that using that setting fixed the problem. Then I switched to 1.9.6 and on that specific version the requests seem to take a long time to be served (but I didn't see the original issue), even when running on localhost. I don't see that happening on 1.9.7 though.

@staff0rd

This comment has been minimized.

staff0rd commented Nov 19, 2015

Updated, same symptoms - 1.3min response time.

Response headers;

HTTP/1.1 200 OK
Server: nginx/1.9.7
Date: Thu, 19 Nov 2015 23:49:28 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
@cesarbs

This comment has been minimized.

Contributor

cesarbs commented Nov 19, 2015

I can hit that URL just fine. Wasn't that the URL where you were seeing the error?

@staff0rd

This comment has been minimized.

staff0rd commented Nov 19, 2015

Seriously? I just tried chrome + firefox + edge. Same symptoms on all - page immediately pops up, but loading doesn't stop for 1 min. However, the page imediately pops up and stops loading if I go around nginx. (http://test.atqu.in:5000)

@cesarbs

This comment has been minimized.

Contributor

cesarbs commented Nov 20, 2015

Ohh, sorry, I didn't notice it was still loading 😞

What addresses are you binding to? I figured out why 1.9.6 was taking so long to forward the request to Kestrel. I was binding Kestrel to my IPv6 loopback interface only, but nginx was first trying to forward the request to the IPv4 interface. It took a minute to time out and try the IPv6 interface. Maybe something similar is happening in your setup?

@staff0rd

This comment has been minimized.

staff0rd commented Nov 20, 2015

Do you mean this or something on the nginx side?

@cesarbs

This comment has been minimized.

Contributor

cesarbs commented Nov 20, 2015

Yep, that. Although you're already binding to an IPv4 interface... Can you try binding to localhost or * and see if that makes a difference?

@staff0rd

This comment has been minimized.

staff0rd commented Nov 20, 2015

Its hosted in a docker container so I don't think that binding to localhost will work. What's the syntax for *, http://*:5000?

@cesarbs

This comment has been minimized.

Contributor

cesarbs commented Nov 20, 2015

That's correct.

@staff0rd

This comment has been minimized.

staff0rd commented Nov 20, 2015

Nah, same result;

Failed to load resource: net::ERR_INCOMPLETE_CHUNKED_ENCODING
@paralin

This comment has been minimized.

paralin commented Apr 8, 2016

@benaadams Any way a fix for this can be made for rc1? I don't think switching the entire stack to the AspCore dev versions is wise but I can apply a single patch to rc1.

@davidfowl

This comment has been minimized.

Member

davidfowl commented Apr 8, 2016

See #341 (comment) for a workaround:

proxy_set_header Connection keep-alive;
@amcdnl

This comment has been minimized.

amcdnl commented May 23, 2016

@benaadams I'm experiencing similar to this in RC2. See: #636 (comment)

@muratg muratg reopened this May 23, 2016

@muratg muratg added 1 - Ready and removed 3 - Done labels May 23, 2016

@muratg muratg assigned mikeharder and unassigned cesarbs May 23, 2016

@muratg muratg modified the milestones: 1.0.0, 1.0.0-rc2 May 23, 2016

@cesarbs

This comment has been minimized.

Contributor

cesarbs commented May 23, 2016

#636 #734 seem to be the same as this one (actually #636 is about BrowserSync). Let's center the discussion here, since this was the original issue.

@amcdnl Which version of nginx are you using, and on which OS?

@halter73

This comment has been minimized.

Member

halter73 commented May 23, 2016

This issue is related to Kestrel not sending the chunked response suffix. This was fixed by e4fd91b which closed this issue and #406.

Unless people are still seeing INCOMPLETE_CHUNKED_ENCODING in chrome after the nginx timeout, I would close this.

@amcdnl

This comment has been minimized.

amcdnl commented May 23, 2016

@cesarbs I updated the details in #636 to reflect your questions.

@mikeharder

This comment has been minimized.

Contributor

mikeharder commented May 24, 2016

@amcdnl: Can we close this issue, or are you still seeing INCOMPLETE_CHUNKED_ENCODING with RC2?

@amcdnl

This comment has been minimized.

amcdnl commented May 24, 2016

@mikeharder Good to go.

@cesarbs cesarbs closed this May 24, 2016

@halter73 halter73 removed the 1 - Ready label May 24, 2016

@halter73 halter73 modified the milestones: 1.0.0-rc2, 1.0.0 May 24, 2016

@halter73 halter73 added the 3 - Done label May 24, 2016

@halter73 halter73 assigned cesarbs and unassigned mikeharder May 24, 2016

@jmbalanag

This comment has been minimized.

jmbalanag commented Jul 27, 2016

Im currently encoutering this issue:

"Microsoft.AspNetCore.Server.Kestrel": "1.0.0",

@davidfowl

This comment has been minimized.

Member

davidfowl commented Jul 27, 2016

@jmbalanag can you provide more details about your setup and provide specific repro steps?

@jmbalanag

This comment has been minimized.

jmbalanag commented Jul 27, 2016

This is the problem code:
[Route("api/lib_fund_source")]
public ActionResult GetFundSource()
{
return Json(db.lib_fund_source.Where(x => x.is_active == true).Select(x => new { Id = x.fund_source_id, Name = x.name }));

    }

if i remove, Where(x => x.is_active == true) it works just fine.

I think Its no longer related with kestrel.

@muratg

This comment has been minimized.

Member

muratg commented Jul 27, 2016

@jmbalang Yeah, doesn't look like a Kestrel issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment