New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rc1 INCOMPLETE_CHUNKED_ENCODING when using nginx #341

Closed
staff0rd opened this Issue Nov 8, 2015 · 39 comments

Comments

Projects
None yet
@staff0rd

staff0rd commented Nov 8, 2015

Opened this here also, but I'm getting the feeling this more specific to kestrel rc1.

I have a docker container that has an aspnet5-rc1 application inside, listening on port 5000. If I make a request directly to that port, the response is 200 and quick.

If I make a request via nginx, then the response data comes back immediately (200), but does not terminate and rather about a minute later the response fails with; (in chrome) ERR_INCOMPLETE_CHUNKED_ENCODING.

Error does not occur when using nginx with aspnet-beta7. Headers for both aspnet-beta7 and aspnet-rc1 are the same;

HTTP/1.1 200 OK
Server: nginx/1.9.6
Date: Sun, 08 Nov 2015 04:38:28 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
X-Frame-Options: SAMEORIGIN
@Tratcher

This comment has been minimized.

Show comment
Hide comment
@Tratcher

Tratcher Nov 10, 2015

Member

I have a local repro that looks similar. If I throw an exception after the first response write then there is no chunked terminator and the connection is closed gracefully rather than reset.

            app.Run(async (context) =>
            {
                await context.Response.WriteAsync("hello world");
                throw new Exception("Test exception");
            });

The behavior is even stranger behind HttpPlatformHandler. It appears to suppress the chunking error and re-chunk the response including a terminator. Verify if it behaves correctly after kestrel changes to a reset.

Member

Tratcher commented Nov 10, 2015

I have a local repro that looks similar. If I throw an exception after the first response write then there is no chunked terminator and the connection is closed gracefully rather than reset.

            app.Run(async (context) =>
            {
                await context.Response.WriteAsync("hello world");
                throw new Exception("Test exception");
            });

The behavior is even stranger behind HttpPlatformHandler. It appears to suppress the chunking error and re-chunk the response including a terminator. Verify if it behaves correctly after kestrel changes to a reset.

@Tratcher Tratcher added the bug label Nov 10, 2015

@Tratcher Tratcher added this to the 1.0.0-rc2 milestone Nov 10, 2015

@staff0rd

This comment has been minimized.

Show comment
Hide comment
@staff0rd

staff0rd Nov 11, 2015

@Tratcher any hints as to why I'm seeing this behind nginx reverse proxy but not when accessing it directly?

staff0rd commented Nov 11, 2015

@Tratcher any hints as to why I'm seeing this behind nginx reverse proxy but not when accessing it directly?

@Tratcher

This comment has been minimized.

Show comment
Hide comment
@Tratcher

Tratcher Nov 11, 2015

Member

@staff0rd That's odd. In my case the reverse proxy (HttpPlatformHandler) hid the behavior by re-chunking the response.

You just posted the headers, did any of the body make it through? Or did you do an explicit Flush to send the headers before an exception occurred?

Member

Tratcher commented Nov 11, 2015

@staff0rd That's odd. In my case the reverse proxy (HttpPlatformHandler) hid the behavior by re-chunking the response.

You just posted the headers, did any of the body make it through? Or did you do an explicit Flush to send the headers before an exception occurred?

@staff0rd

This comment has been minimized.

Show comment
Hide comment
@staff0rd

staff0rd Nov 12, 2015

@Tratcher if you browse to http://test.atqu.in you can see the issue occur. If you browse to http://test.atqu.in:5000 you'll reach the same site directly (without reverse proxy) and you'll see the issue does not occur. As you'll see, the whole site is is delivered both times, but the connection stays open on the reverse-proxy for about a minute until it finally dies with INCOMPLETE_CHUNKED_ENCODING.

The code is here and is just a fresh MVC 6 beta8 project updated to rc1, so no extra flushes occur, nor are any exceptions thrown.

staff0rd commented Nov 12, 2015

@Tratcher if you browse to http://test.atqu.in you can see the issue occur. If you browse to http://test.atqu.in:5000 you'll reach the same site directly (without reverse proxy) and you'll see the issue does not occur. As you'll see, the whole site is is delivered both times, but the connection stays open on the reverse-proxy for about a minute until it finally dies with INCOMPLETE_CHUNKED_ENCODING.

The code is here and is just a fresh MVC 6 beta8 project updated to rc1, so no extra flushes occur, nor are any exceptions thrown.

@Tratcher

This comment has been minimized.

Show comment
Hide comment
@Tratcher

Tratcher Nov 16, 2015

Member

@staff0rd Let's re-test your scenario after we get the identified bug fixed.

Member

Tratcher commented Nov 16, 2015

@staff0rd Let's re-test your scenario after we get the identified bug fixed.

@pakrym pakrym added 1 - Ready and removed 2 - Working labels Nov 16, 2015

@halter73 halter73 assigned cesarbs and unassigned pakrym Nov 18, 2015

@cesarbs

This comment has been minimized.

Show comment
Hide comment
@cesarbs

cesarbs Nov 18, 2015

Contributor

The problem here is the same as what we're facing in #368: we're not sending an RST when we expect to (ConnectionControl.End(ProduceEndType.SocketDisconnect)).

Libuv currently does not expose the underlying socket enough for us to force an RST. Someone tried to submit a PR in the past but it seems like that never got in: joyent/libuv#498

Basically what needs to be done is to set SO_LINGER on the socket with a timeout of 0. That should force an RST.

Contributor

cesarbs commented Nov 18, 2015

The problem here is the same as what we're facing in #368: we're not sending an RST when we expect to (ConnectionControl.End(ProduceEndType.SocketDisconnect)).

Libuv currently does not expose the underlying socket enough for us to force an RST. Someone tried to submit a PR in the past but it seems like that never got in: joyent/libuv#498

Basically what needs to be done is to set SO_LINGER on the socket with a timeout of 0. That should force an RST.

@cesarbs

This comment has been minimized.

Show comment
Hide comment
@cesarbs

cesarbs Nov 19, 2015

Contributor

Forget my previous comment 😄

nginx makes HTTP/1.0 requests when acting as a proxy. There is a known issue for Connection: close requests (which HTTP/1.0 requests implicitly are): #406

There's an easy fix for putting Kestrel behind an nginx reverse proxy: set proxy_http_version 1.1; to have nginx send HTTP/1.1 requests.

Contributor

cesarbs commented Nov 19, 2015

Forget my previous comment 😄

nginx makes HTTP/1.0 requests when acting as a proxy. There is a known issue for Connection: close requests (which HTTP/1.0 requests implicitly are): #406

There's an easy fix for putting Kestrel behind an nginx reverse proxy: set proxy_http_version 1.1; to have nginx send HTTP/1.1 requests.

@staff0rd

This comment has been minimized.

Show comment
Hide comment
@staff0rd

staff0rd Nov 19, 2015

This is already configuring HTTP/1.1. Looking at the response headers of test.atqu.in also confirms that HTTP/1.1 is being used - same headers as in first post above.

staff0rd commented Nov 19, 2015

This is already configuring HTTP/1.1. Looking at the response headers of test.atqu.in also confirms that HTTP/1.1 is being used - same headers as in first post above.

@cesarbs

This comment has been minimized.

Show comment
Hide comment
@cesarbs

cesarbs Nov 19, 2015

Contributor

Can you try a different version of nginx? I repro'd your issue on 1.8 and found that using that setting fixed the problem. Then I switched to 1.9.6 and on that specific version the requests seem to take a long time to be served (but I didn't see the original issue), even when running on localhost. I don't see that happening on 1.9.7 though.

Contributor

cesarbs commented Nov 19, 2015

Can you try a different version of nginx? I repro'd your issue on 1.8 and found that using that setting fixed the problem. Then I switched to 1.9.6 and on that specific version the requests seem to take a long time to be served (but I didn't see the original issue), even when running on localhost. I don't see that happening on 1.9.7 though.

@staff0rd

This comment has been minimized.

Show comment
Hide comment
@staff0rd

staff0rd Nov 19, 2015

Updated, same symptoms - 1.3min response time.

Response headers;

HTTP/1.1 200 OK
Server: nginx/1.9.7
Date: Thu, 19 Nov 2015 23:49:28 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive

staff0rd commented Nov 19, 2015

Updated, same symptoms - 1.3min response time.

Response headers;

HTTP/1.1 200 OK
Server: nginx/1.9.7
Date: Thu, 19 Nov 2015 23:49:28 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
@cesarbs

This comment has been minimized.

Show comment
Hide comment
@cesarbs

cesarbs Nov 19, 2015

Contributor

I can hit that URL just fine. Wasn't that the URL where you were seeing the error?

Contributor

cesarbs commented Nov 19, 2015

I can hit that URL just fine. Wasn't that the URL where you were seeing the error?

@staff0rd

This comment has been minimized.

Show comment
Hide comment
@staff0rd

staff0rd Nov 19, 2015

Seriously? I just tried chrome + firefox + edge. Same symptoms on all - page immediately pops up, but loading doesn't stop for 1 min. However, the page imediately pops up and stops loading if I go around nginx. (http://test.atqu.in:5000)

staff0rd commented Nov 19, 2015

Seriously? I just tried chrome + firefox + edge. Same symptoms on all - page immediately pops up, but loading doesn't stop for 1 min. However, the page imediately pops up and stops loading if I go around nginx. (http://test.atqu.in:5000)

@cesarbs

This comment has been minimized.

Show comment
Hide comment
@cesarbs

cesarbs Nov 20, 2015

Contributor

Ohh, sorry, I didn't notice it was still loading 😞

What addresses are you binding to? I figured out why 1.9.6 was taking so long to forward the request to Kestrel. I was binding Kestrel to my IPv6 loopback interface only, but nginx was first trying to forward the request to the IPv4 interface. It took a minute to time out and try the IPv6 interface. Maybe something similar is happening in your setup?

Contributor

cesarbs commented Nov 20, 2015

Ohh, sorry, I didn't notice it was still loading 😞

What addresses are you binding to? I figured out why 1.9.6 was taking so long to forward the request to Kestrel. I was binding Kestrel to my IPv6 loopback interface only, but nginx was first trying to forward the request to the IPv4 interface. It took a minute to time out and try the IPv6 interface. Maybe something similar is happening in your setup?

@staff0rd

This comment has been minimized.

Show comment
Hide comment
@staff0rd

staff0rd Nov 20, 2015

Do you mean this or something on the nginx side?

staff0rd commented Nov 20, 2015

Do you mean this or something on the nginx side?

@cesarbs

This comment has been minimized.

Show comment
Hide comment
@cesarbs

cesarbs Nov 20, 2015

Contributor

Yep, that. Although you're already binding to an IPv4 interface... Can you try binding to localhost or * and see if that makes a difference?

Contributor

cesarbs commented Nov 20, 2015

Yep, that. Although you're already binding to an IPv4 interface... Can you try binding to localhost or * and see if that makes a difference?

@staff0rd

This comment has been minimized.

Show comment
Hide comment
@staff0rd

staff0rd Nov 20, 2015

Its hosted in a docker container so I don't think that binding to localhost will work. What's the syntax for *, http://*:5000?

staff0rd commented Nov 20, 2015

Its hosted in a docker container so I don't think that binding to localhost will work. What's the syntax for *, http://*:5000?

@cesarbs

This comment has been minimized.

Show comment
Hide comment
@cesarbs

cesarbs Nov 20, 2015

Contributor

That's correct.

Contributor

cesarbs commented Nov 20, 2015

That's correct.

@staff0rd

This comment has been minimized.

Show comment
Hide comment
@staff0rd

staff0rd Nov 20, 2015

Nah, same result;

Failed to load resource: net::ERR_INCOMPLETE_CHUNKED_ENCODING

staff0rd commented Nov 20, 2015

Nah, same result;

Failed to load resource: net::ERR_INCOMPLETE_CHUNKED_ENCODING
@cesarbs

This comment has been minimized.

Show comment
Hide comment
@cesarbs

cesarbs Nov 20, 2015

Contributor

Ok, I've managed to repro your issue and it's in fact the same issue linked above - #406.

For some reason nginx sends a Connection: close header to Kestrel. The nginx-proxy template actually erases any Connection headers in the request, but that's not really the issue because for HTTP/1.1 no Connection == Connection: keep-alive.

When I added

proxy_set_header Connection keep-alive;

to my nginx.conf the issue was gone. Can you try doing that?

Contributor

cesarbs commented Nov 20, 2015

Ok, I've managed to repro your issue and it's in fact the same issue linked above - #406.

For some reason nginx sends a Connection: close header to Kestrel. The nginx-proxy template actually erases any Connection headers in the request, but that's not really the issue because for HTTP/1.1 no Connection == Connection: keep-alive.

When I added

proxy_set_header Connection keep-alive;

to my nginx.conf the issue was gone. Can you try doing that?

@staff0rd

This comment has been minimized.

Show comment
Hide comment
@staff0rd

staff0rd Nov 20, 2015

nice one @cesarbs, that indeed fixes it. I don't understand how/why that fixes though*, nor why nginx-proxy removes it, I guess I should take it up with them.

* because I'm an http server noob :)

staff0rd commented Nov 20, 2015

nice one @cesarbs, that indeed fixes it. I don't understand how/why that fixes though*, nor why nginx-proxy removes it, I guess I should take it up with them.

* because I'm an http server noob :)

@cesarbs

This comment has been minimized.

Show comment
Hide comment
@cesarbs

cesarbs Nov 20, 2015

Contributor

Closing this since we've found a workaround and we know that #406 will eventually fix the problem.

Contributor

cesarbs commented Nov 20, 2015

Closing this since we've found a workaround and we know that #406 will eventually fix the problem.

@cesarbs cesarbs closed this Nov 20, 2015

@cesarbs

This comment has been minimized.

Show comment
Hide comment
@cesarbs

cesarbs Nov 20, 2015

Contributor

@staff0rd The real issue is #406, the Connection: close sent by nginx hits that issue. Setting Connection: keep-alive avoids the buggy code path.

Contributor

cesarbs commented Nov 20, 2015

@staff0rd The real issue is #406, the Connection: close sent by nginx hits that issue. Setting Connection: keep-alive avoids the buggy code path.

@tugberkugurlu

This comment has been minimized.

Show comment
Hide comment
@tugberkugurlu

tugberkugurlu Jan 16, 2016

Member

had the same issue, luckily has been pointed to hear before losing all my hairs 😄 http://stackoverflow.com/q/34821881/463785

Member

tugberkugurlu commented Jan 16, 2016

had the same issue, luckily has been pointed to hear before losing all my hairs 😄 http://stackoverflow.com/q/34821881/463785

@halter73 halter73 added 3 - Done and removed 1 - Ready labels Mar 9, 2016

@tobz

This comment has been minimized.

Show comment
Hide comment
@tobz

tobz Mar 10, 2016

Just for anyone that stumbles on this issue: I'm also seeing this behavior when using the webpack-dev-server/hot reload setup proxying to Kestrel. Talking straight to Kestrel works as intended. I haven't yet investigated if a temporary workaround exists or not.

tobz commented Mar 10, 2016

Just for anyone that stumbles on this issue: I'm also seeing this behavior when using the webpack-dev-server/hot reload setup proxying to Kestrel. Talking straight to Kestrel works as intended. I haven't yet investigated if a temporary workaround exists or not.

@tobz

This comment has been minimized.

Show comment
Hide comment
@tobz

tobz Mar 10, 2016

Spent a little time looking at it, and forcing node-http-proxy to send Connection: keep-alive to Kestrel seems to do the trick. Here's the proxy section of my webpack-dev-server configuration:

devServer: {
    proxy: {
        '*': {
            target: 'http://localhost:5000/',
            secure: false,
            changeOrigin: true,
            headers: {
                'Connection': 'keep-alive'
            }
      },
},

tobz commented Mar 10, 2016

Spent a little time looking at it, and forcing node-http-proxy to send Connection: keep-alive to Kestrel seems to do the trick. Here's the proxy section of my webpack-dev-server configuration:

devServer: {
    proxy: {
        '*': {
            target: 'http://localhost:5000/',
            secure: false,
            changeOrigin: true,
            headers: {
                'Connection': 'keep-alive'
            }
      },
},

tobz added a commit to nuclearfurnace/blockcert that referenced this issue Mar 10, 2016

Send keep-alive headers to Kestrel to avoid the chunked encoded bug.
As mentioned on aspnet/KestrelHttpServer#341, Kestrel seems to have a bug when behind a proxy (like when we use webpack-dev-server..) and so sending the keep-alive headers to it seems to temporarily fix the glitch until it's fixed upstream. *shrug*
@paralin

This comment has been minimized.

Show comment
Hide comment
@paralin

paralin Apr 8, 2016

Can this be fixed for real?

Nginx sends Connection: close because when proxying requests it's best to close the connection immediately and not keep it alive.

paralin commented Apr 8, 2016

Can this be fixed for real?

Nginx sends Connection: close because when proxying requests it's best to close the connection immediately and not keep it alive.

@benaadams

This comment has been minimized.

Show comment
Hide comment
@benaadams

benaadams Apr 8, 2016

Contributor

@paralin this is fixed in RC2

Contributor

benaadams commented Apr 8, 2016

@paralin this is fixed in RC2

@paralin

This comment has been minimized.

Show comment
Hide comment
@paralin

paralin Apr 8, 2016

@benaadams Any way a fix for this can be made for rc1? I don't think switching the entire stack to the AspCore dev versions is wise but I can apply a single patch to rc1.

paralin commented Apr 8, 2016

@benaadams Any way a fix for this can be made for rc1? I don't think switching the entire stack to the AspCore dev versions is wise but I can apply a single patch to rc1.

@davidfowl

This comment has been minimized.

Show comment
Hide comment
@davidfowl

davidfowl Apr 8, 2016

Member

See #341 (comment) for a workaround:

proxy_set_header Connection keep-alive;
Member

davidfowl commented Apr 8, 2016

See #341 (comment) for a workaround:

proxy_set_header Connection keep-alive;
@amcdnl

This comment has been minimized.

Show comment
Hide comment
@amcdnl

amcdnl May 23, 2016

@benaadams I'm experiencing similar to this in RC2. See: #636 (comment)

amcdnl commented May 23, 2016

@benaadams I'm experiencing similar to this in RC2. See: #636 (comment)

@muratg muratg reopened this May 23, 2016

@muratg muratg added 1 - Ready and removed 3 - Done labels May 23, 2016

@muratg muratg assigned mikeharder and unassigned cesarbs May 23, 2016

@muratg muratg modified the milestones: 1.0.0, 1.0.0-rc2 May 23, 2016

@cesarbs

This comment has been minimized.

Show comment
Hide comment
@cesarbs

cesarbs May 23, 2016

Contributor

#636 #734 seem to be the same as this one (actually #636 is about BrowserSync). Let's center the discussion here, since this was the original issue.

@amcdnl Which version of nginx are you using, and on which OS?

Contributor

cesarbs commented May 23, 2016

#636 #734 seem to be the same as this one (actually #636 is about BrowserSync). Let's center the discussion here, since this was the original issue.

@amcdnl Which version of nginx are you using, and on which OS?

@halter73

This comment has been minimized.

Show comment
Hide comment
@halter73

halter73 May 23, 2016

Member

This issue is related to Kestrel not sending the chunked response suffix. This was fixed by e4fd91b which closed this issue and #406.

Unless people are still seeing INCOMPLETE_CHUNKED_ENCODING in chrome after the nginx timeout, I would close this.

Member

halter73 commented May 23, 2016

This issue is related to Kestrel not sending the chunked response suffix. This was fixed by e4fd91b which closed this issue and #406.

Unless people are still seeing INCOMPLETE_CHUNKED_ENCODING in chrome after the nginx timeout, I would close this.

@amcdnl

This comment has been minimized.

Show comment
Hide comment
@amcdnl

amcdnl May 23, 2016

@cesarbs I updated the details in #636 to reflect your questions.

amcdnl commented May 23, 2016

@cesarbs I updated the details in #636 to reflect your questions.

@mikeharder

This comment has been minimized.

Show comment
Hide comment
@mikeharder

mikeharder May 24, 2016

Contributor

@amcdnl: Can we close this issue, or are you still seeing INCOMPLETE_CHUNKED_ENCODING with RC2?

Contributor

mikeharder commented May 24, 2016

@amcdnl: Can we close this issue, or are you still seeing INCOMPLETE_CHUNKED_ENCODING with RC2?

@amcdnl

This comment has been minimized.

Show comment
Hide comment
@amcdnl

amcdnl commented May 24, 2016

@mikeharder Good to go.

@cesarbs cesarbs closed this May 24, 2016

@halter73 halter73 removed the 1 - Ready label May 24, 2016

@halter73 halter73 modified the milestones: 1.0.0-rc2, 1.0.0 May 24, 2016

@halter73 halter73 added the 3 - Done label May 24, 2016

@halter73 halter73 assigned cesarbs and unassigned mikeharder May 24, 2016

@jmbalanag

This comment has been minimized.

Show comment
Hide comment
@jmbalanag

jmbalanag Jul 27, 2016

Im currently encoutering this issue:

"Microsoft.AspNetCore.Server.Kestrel": "1.0.0",

jmbalanag commented Jul 27, 2016

Im currently encoutering this issue:

"Microsoft.AspNetCore.Server.Kestrel": "1.0.0",

@davidfowl

This comment has been minimized.

Show comment
Hide comment
@davidfowl

davidfowl Jul 27, 2016

Member

@jmbalanag can you provide more details about your setup and provide specific repro steps?

Member

davidfowl commented Jul 27, 2016

@jmbalanag can you provide more details about your setup and provide specific repro steps?

@jmbalanag

This comment has been minimized.

Show comment
Hide comment
@jmbalanag

jmbalanag Jul 27, 2016

This is the problem code:
[Route("api/lib_fund_source")]
public ActionResult GetFundSource()
{
return Json(db.lib_fund_source.Where(x => x.is_active == true).Select(x => new { Id = x.fund_source_id, Name = x.name }));

    }

if i remove, Where(x => x.is_active == true) it works just fine.

I think Its no longer related with kestrel.

jmbalanag commented Jul 27, 2016

This is the problem code:
[Route("api/lib_fund_source")]
public ActionResult GetFundSource()
{
return Json(db.lib_fund_source.Where(x => x.is_active == true).Select(x => new { Id = x.fund_source_id, Name = x.name }));

    }

if i remove, Where(x => x.is_active == true) it works just fine.

I think Its no longer related with kestrel.

@muratg

This comment has been minimized.

Show comment
Hide comment
@muratg

muratg Jul 27, 2016

Member

@jmbalang Yeah, doesn't look like a Kestrel issue

Member

muratg commented Jul 27, 2016

@jmbalang Yeah, doesn't look like a Kestrel issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment