proxy closing connection before sending all the data #406

Closed
NachoSoto opened this Issue Apr 18, 2013 · 6 comments

Comments

Projects
None yet
4 participants
@NachoSoto

I recently upgraded my backend to node 0.10.4 and node-http-proxy 0.10.1 and some of my services stopped serving static files correctly. After a while poking around, I curled the file locally using http and it was serving it correctly, but then I noticed this when curling behind the proxy:

curl -kv https://a.xxx.com
* About to connect() to a.xxx.com port 443 (#0)
*   Trying IP...
* connected
* Connected to a.xxx.com (IP) port 443 (#0)
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using AES256-SHA
* Server certificate:
*    subject: OU=Domain Control Validated; OU=PositiveSSL; CN=a.xxx.com
*    start date: 2012-10-30 00:00:00 GMT
*    expire date: 2013-10-30 23:59:59 GMT
*    subjectAltName does not match a.xxx.com
> GET file HTTP/1.1
> User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8r zlib/1.2.5
> Host: a.xxx.com
> Accept: */*
>
< HTTP/1.1 200 OK
< server: gunicorn/0.17.2
< date: Thu, 18 Apr 2013 01:59:57 GMT
< connection: close
< last-modified: Wed, 17 Apr 2013 04:24:04 GMT
< content-length: 340416
< content-type: application/javascript
< content-language: en-us
< vary: Accept-Language, Cookie
<
.
trimmed data here (missing the last part of the file)
.
* transfer closed with 176576 bytes remaining to read
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):
curl: (18) transfer closed with 176576 bytes remaining to read

This is how I'm using http-proxy:

var options = {
    hostnameOnly: true,
    router: {
        'a.xxx.com': '127.0.0.1:8000',
        'b.xxx.com': '127.0.0.1:9000',
    },
    https: {
        ca: ca,
        key: fs.readFileSync('../ssl/xxx.com.key', 'utf8'),
        cert: fs.readFileSync('../ssl/xxx.com.crt', 'utf8')
    }
};

var proxyServer = httpProxy.createServer(options);
proxyServer.listen(443, function() {
    console.log('Proxy running');
});

Any ideas why this stopped working all of a sudden? Let me know if you need any more details.
Thanks in advance.

@RGBboy

This comment has been minimized.

Show comment Hide comment
@RGBboy

RGBboy Jul 5, 2013

I'm having the same issue. I investigated further and found that any content over 10901 bytes in length has it's connection closed.

Any ideas?

Node 10.6
http-proxy 0.10.3

RGBboy commented Jul 5, 2013

I'm having the same issue. I investigated further and found that any content over 10901 bytes in length has it's connection closed.

Any ideas?

Node 10.6
http-proxy 0.10.3

@RGBboy

This comment has been minimized.

Show comment Hide comment
@RGBboy

RGBboy Jul 5, 2013

Ok so I figured out a solution to my problem. I was using a HTTPS to HTTPS proxy with an Apache server. The Apache server had Keep Alive turned off. Turning it on seems to have solved the issue.

RGBboy commented Jul 5, 2013

Ok so I figured out a solution to my problem. I was using a HTTPS to HTTPS proxy with an Apache server. The Apache server had Keep Alive turned off. Turning it on seems to have solved the issue.

@tuomassalo

This comment has been minimized.

Show comment Hide comment
@tuomassalo

tuomassalo Aug 21, 2013

I have the same issue. I have a backend Apache server (http) and a frontend https node-http-proxy. The problem happens on about 20 % of the requests. I'm trying to fetch a document of 243543 bytes, but sometimes the (frontend) connection is closed when 14431 bytes are remaining.

This doesn't happen if I change the frontend to http.

Turning on keep-alive in the backend seems to fix it, but only for certain clients. If you run curl in HTTP 1.0 mode (curl -0 ...), it will fail again.

Maybe there's a timing problem somewhere in the http-proxy code, a one that' sonly triggered in a certain keepalive config, and for some reason only when serving https.

How to fix? Any workarounds?

I have the same issue. I have a backend Apache server (http) and a frontend https node-http-proxy. The problem happens on about 20 % of the requests. I'm trying to fetch a document of 243543 bytes, but sometimes the (frontend) connection is closed when 14431 bytes are remaining.

This doesn't happen if I change the frontend to http.

Turning on keep-alive in the backend seems to fix it, but only for certain clients. If you run curl in HTTP 1.0 mode (curl -0 ...), it will fail again.

Maybe there's a timing problem somewhere in the http-proxy code, a one that' sonly triggered in a certain keepalive config, and for some reason only when serving https.

How to fix? Any workarounds?

tuomassalo pushed a commit to tuomassalo/node-http-proxy that referenced this issue Aug 21, 2013

@tuomassalo

This comment has been minimized.

Show comment Hide comment
@tuomassalo

tuomassalo Aug 21, 2013

I tried to write a test case on this. It was difficult, since the problem seems to only appear when there's some latency between the client and the proxy server. I didn't have time to find a tool to simulate the latency, so the test doesn't show the problem when executed on one machine.

However, the problem should appear when server.js is started on one host and curl-test.sh on another, maybe in another network.

How to test:

  • Clone my fork: https://github.com/tuomassalo/node-http-proxy.git
  • Go to test/keepalive/
  • run node server.js
  • On another machine, modify curl-test.sh and run it. It should print a bunch of OK lines, and then an occasional error with something like * transfer closed with 6568 bytes remaining to read. That's the bug.

UPDATE: I first thought that the problem only appeared with some specific keepalive settings. Now I'm able to reproduce it without curl's -0 and without sending Connection: close in the backend server. Odd, but I don't have more time to look at this now.

UPDATE 2: It seems that the bug only happens on specific versions of eg. curl. This seems like a very hard one. I was able to reproduce on Ubuntu 10.04 and 12.04, but not OSX 10.8.4.

But there seems to be more to it. I created a simple HTTPS server that only sends data (see https://gist.github.com/tuomassalo/6296041), and most versions of curl seem to fail with even that. No http-proxy in that example. I'm running that server at https://212.246.29.149:11443, so you can try eg. this:

while true; do curl -s -k -0 https://212.246.29.149:11443 2>&1 | wc -c; sleep .5; done

My curl prints mostly 243543, but sometimes 229251.

I'm quite confused about this whole thing. Is there a node.js problem? A http-proxy problem?

Oh, btw, I'm running node v0.10.15 on all servers.

I tried to write a test case on this. It was difficult, since the problem seems to only appear when there's some latency between the client and the proxy server. I didn't have time to find a tool to simulate the latency, so the test doesn't show the problem when executed on one machine.

However, the problem should appear when server.js is started on one host and curl-test.sh on another, maybe in another network.

How to test:

  • Clone my fork: https://github.com/tuomassalo/node-http-proxy.git
  • Go to test/keepalive/
  • run node server.js
  • On another machine, modify curl-test.sh and run it. It should print a bunch of OK lines, and then an occasional error with something like * transfer closed with 6568 bytes remaining to read. That's the bug.

UPDATE: I first thought that the problem only appeared with some specific keepalive settings. Now I'm able to reproduce it without curl's -0 and without sending Connection: close in the backend server. Odd, but I don't have more time to look at this now.

UPDATE 2: It seems that the bug only happens on specific versions of eg. curl. This seems like a very hard one. I was able to reproduce on Ubuntu 10.04 and 12.04, but not OSX 10.8.4.

But there seems to be more to it. I created a simple HTTPS server that only sends data (see https://gist.github.com/tuomassalo/6296041), and most versions of curl seem to fail with even that. No http-proxy in that example. I'm running that server at https://212.246.29.149:11443, so you can try eg. this:

while true; do curl -s -k -0 https://212.246.29.149:11443 2>&1 | wc -c; sleep .5; done

My curl prints mostly 243543, but sometimes 229251.

I'm quite confused about this whole thing. Is there a node.js problem? A http-proxy problem?

Oh, btw, I'm running node v0.10.15 on all servers.

@tuomassalo

This comment has been minimized.

Show comment Hide comment
@tuomassalo

tuomassalo Aug 22, 2013

@NachoSoto this bug is probably fixed in v0.11.x. See joyent/node#6107. Can you test by running http-proxy on v0.11.x?

@NachoSoto this bug is probably fixed in v0.11.x. See joyent/node#6107. Can you test by running http-proxy on v0.11.x?

@tuomassalo

This comment has been minimized.

Show comment Hide comment
@tuomassalo

tuomassalo Oct 14, 2013

This turned out to be a node.js problem, and will be fixed in node.js v0.10.21. See joyent/node#6107.

This turned out to be a node.js problem, and will be fixed in node.js v0.10.21. See joyent/node#6107.

toolness added a commit to toolness/hive-django that referenced this issue Apr 18, 2014

Revert "attempt to use tornado w/ gunicorn to fix truncated files pro…
…blem."

This reverts commit a7518b7.

It seems the problem is related to the proxy I'm using to serve the
directory from over a custom HTTPS domain:

nodejitsu/node-http-proxy#406

@jcrugzz jcrugzz closed this Jan 20, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment