http: fixed socket destruction, timeout and leaking #2534

tshemsedinov · 2015-08-25T01:59:16Z

Prevent 2 minutes sockets and memory hanging/leaking after server finished writing last response and connection hang inactive waiting for nothing (it is actual only in keep-alive mode, single HTTP requests does not hanging for 2 minutes, browsers do not reuse them and somehow do not close them too)
Fixed res._last, because it was not calculated correctly in this place before this PR and as a result socket.destroySoon() was never executed
Implemented server.keepAliveTimeout (default to 5 seconds) in addition to server.timeout
Prevent firing timeout event on server for finished and closed sockets
Removed socket.destroySoon() because it worked well on prefinish and have no sense on finish

Memory usage comparison before and after:

Socket handlers usage comparison before and after:

Tests changed and provided new.
Related PR: #1411 by @indutny
Related Issue: nodejs/node-v0.x-archive#3460
Article about this problem (RU): http://habrahabr.ru/post/264851/

indutny · 2015-08-25T02:24:15Z

lib/_http_server.js

@@ -447,20 +448,25 @@ function connectionListener(socket) {
      // if the user never called req.read(), and didn't pipe() or
      // .resume() or .on('data'), then we call req._dump() so that the
      // bytes will be pulled off the wire.
-      if (!req._consuming && !req._readableState.resumeScheduled)


Unnecessary style fixes.

ChALkeR · 2015-08-25T06:57:38Z

This PR changes the expectations of at least one test without changing anything else in that test. Does it make this PR a semver-major, or would it still be semver-minor?

tshemsedinov · 2015-08-25T07:42:12Z

This expectation https://github.com/tshemsedinov/io.js/blob/master/test/sequential/test-http-pipeline-flood.js#L51 based on wrong behavior, when timeout was emitted on sockets hanging open and on sockets already closed (it is actual just for keep-alive sockets).

Fishrock123 · 2015-08-25T15:42:50Z

test/parallel/test-http-server-keep-alive-timeout.js

+// NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
+// DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
+// OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE
+// USE OR OTHER DEALINGS IN THE SOFTWARE.


Please remove this license boilerplate. The necessary licensing can already be found in the LICENSE file.

Fishrock123 · 2015-08-25T19:24:57Z

CI: https://jenkins-iojs.nodesource.com/job/node-test-pull-request/163/

issue: nodejs/node#2534 crutch from: http://habrahabr.ru/post/264851/ Signed-off-by: Oleg Verych <ole4at@gmail.com>

ChALkeR · 2015-09-04T23:20:54Z

CI: https://ci.nodejs.org/job/node-test-pull-request/253/

indutny · 2015-09-09T23:00:21Z

lib/_http_server.js


      res.detachSocket(socket);
+      res._last = outgoing.length === 0;


Why is this required?

Because in this place _last is not calculated, so when length is 0, _last is not true and res will not processed as last.

Maybe it is not calculated because the res might be not the last here?

In debugging I found that when res was last in outgoing, and _last flag is not correct, but we need it calculated correctly for next lenes, to destroy socket if it is last.

Again, it should be a separate clause for this thing. _last indicates that there are not going to be any requests on the TCP socket. While there definitely could be in case of outgoing.length === 0.

We checking that on res:finish event, why it is not a right place to detect "is outgoing queue is empty?" In this place res._last inicates that we have no nore message to send.

Because we should destroy the socket immediately when the connection wasn't keep-alive, or when TCP connection has already ended.

indutny · 2015-09-09T23:17:31Z

@tshemsedinov : decided to do a proper write up, instead of in-line comments.

I like the idea of PR, but I think the way it interacts with existing logic could be potentially buggy. res._last is used to destroy the underlying socket when either:

incoming TCP connection has ended
request didn't have Connection: keep-alive

Instead of using it for different purpose, I suggest to do it without property it all. If there are no pending requests - let's call setTimeout with that keep alive timeout value. When the new request comes - setTimeout with regular timeout value should be called, to restore it.

Thanks for submitting this!

tshemsedinov · 2015-09-10T00:06:26Z

Of course we can check outgoing queue and not changing _last immediately call setTimeout for socket destruction but I thought it was the purpose of this flag _last. I'll fix that, @indutny
What should I do with ditto marks? It is unwonted or indifferent?

indutny · 2015-09-10T00:08:21Z

@tshemsedinov there are lots of irrelevant style changes through the PR, ditto means "here too". Please change only the parts of code that are relevant to this behavior change. Also, there are some style issues (mostly 80 column limit).

indutny · 2015-09-10T00:08:23Z

Thanks!

tshemsedinov · 2015-09-10T21:46:57Z

lib/_http_server.js

@@ -482,8 +483,15 @@ function connectionListener(socket) {

      res.detachSocket(socket);

-      if (res._last) {
-        socket.destroySoon();
+      if (res._last) socket.destroySoon();


I found problem with non keep-alive sockets and here is fix for this case. @indutny, finally _last in it's untouched state is useful to detect non keep-alive sockets and destroy them as it was before this changes. Now all tests completed successfully.

ChALkeR · 2015-09-10T23:23:12Z

CI: https://ci.nodejs.org/job/node-test-pull-request/276/

indutny · 2015-09-10T23:24:13Z

@ChALkeR : a bit premature? Maybe cancel?

ChALkeR · 2015-09-10T23:27:22Z

@indutny He asked for it, why not? Most machines are free.

aqrln · 2017-05-24T00:02:59Z

Wow, I think this wasn't a successful force-push :/
Why did GitHub close this?

aqrln · 2017-05-24T00:08:50Z

That was strange, but I've reopened the PR and the commit is here, great.

aqrln · 2017-05-24T00:42:22Z

I wanted to trigger a fresh CI run on this, but something got really borked with this PR and git fetch upstream +refs/pull/2534/head:pr-2534 fetches master instead of this commit, both locally on my machine and on CI, so I had to kill the job.

node-test-commit CI job for tshemsedinov/io.js instead: https://ci.nodejs.org/job/node-test-commit/10108/
(I rebased the commit onto the latest master before force-pushing to @tshemsedinov's repo, so that should be fine.)

aqrln · 2017-05-24T15:37:24Z

Yay, everything works fine after another force-push.

CI: https://ci.nodejs.org/job/node-test-pull-request/8282/

I've made some changes to the docs, namely: YAML header with the "added:" metadata, cross-links between the server.timeout and server.keepAliveTimeout sections, and a number of stylistic fixes. Can anyone review these, please? :)

/cc @jasnell @mcollina @fhinkel

refack

Documentation generally LGTM.
Just a few nits and lite rephrasing

refack · 2017-05-25T11:16:30Z

doc/api/http.md

@@ -841,8 +841,25 @@ Note that the socket timeout logic is set up on connection, so
 changing this value only affects *new* connections to the server, not


Aren't these two bulleted list notes?

I'd rather not change it in this PR, it would be an unrelated stylistic change. But yeah, a nice thing to do in a follow-up PR.

Ah, never mind. I changed the server.timeout docs for consistency anyway.

refack · 2017-05-25T11:20:04Z

doc/api/http.md

@@ -841,8 +841,25 @@ Note that the socket timeout logic is set up on connection, so
 changing this value only affects *new* connections to the server, not
 any existing connections.

-Set to 0 to disable any kind of automatic timeout behavior on incoming
-connections.
+Set to 0 to disable timeout behavior on incoming connections.


[Suggestion] /s/Set to 0 to disable/A value of 0 will disable

P.S. what does it mean? Sockets will be destroyed ASAP, or timeout is delegated to OS?

The latter option, roughly speaking. It will just disable any Node-specific timeout logic.

refack · 2017-05-25T11:20:06Z

doc/api/http.md

-Set to 0 to disable any kind of automatic timeout behavior on incoming
-connections.
+Set to 0 to disable timeout behavior on incoming connections.
+If `server.timeout` is not 0 and [`server.keepAliveTimeout`][] is 0,


IMHO you can simplify to just If [`server.keepAliveTimeout`][] is 0, the value of `server.timeout` will be used instead.
And now it's a Note: for server.keepAliveTimeout

Hmm... or, maybe, even better to drop this sentence completely? It's not exactly accurate.

refack · 2017-05-25T11:22:33Z

doc/api/http.md

+added: REPLACEME
+-->
+
+* {Number} Defaults to 5000 (5 seconds).


* {Number} Timeout in milliseconds. Defaults to 5000.

refack · 2017-05-25T11:26:01Z

doc/api/http.md

+
+* {Number} Defaults to 5000 (5 seconds).
+
+The number of milliseconds of inactivity after a server has finished writing


The number of milliseconds of inactivity the server needs to wait for additional incoming data, after it has finished writing the last response, before a socket will be destroyed.

refack · 2017-05-25T11:27:02Z

doc/api/http.md

+socket will be destroyed.
+
+If the server receives new data before the keep-alive timeout has fired, it
+will renew the regular inactivity timeout, i.e., [`server.timeout`][].


/s/renew/reset/

aqrln · 2017-05-25T12:03:55Z

@refack thanks! I've made some amendments, can you please take another look?

refack

Docs LGTM.
Rubberstamp the code.

refack · 2017-05-25T14:40:32Z

doc/api/http.md

+
+The number of milliseconds of inactivity a server needs to wait for additional
+incoming data, after it has finished writing the last response, before a socket
+will be destroyed.


Tiny nit.
IMHO These two paragraphs could now be merged (the sentences have the same subject, so they could follow each other naturally)
_{very low importance on this}

aqrln · 2017-05-25T15:31:26Z

@jasnell as far as I see from the notifications, you're on GitHub now 😄 Can you please take a look at the latest changes while you are here? Thanks!

Implement server.keepAliveTimeout in addition to server.timeout to prevent temporary socket/memory leaking in keep-alive mode.

aqrln · 2017-05-25T20:45:25Z

Squashed and rebased onto master again.
Fresh CI: https://ci.nodejs.org/job/node-test-pull-request/8313/

I'd like to land it today or tomorrow.

/cc @jasnell @mcollina @fhinkel @refack

refack · 2017-05-25T22:37:32Z

CITGM: https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/832/

refack

LGTM.
CITGM clean.

Implement server.keepAliveTimeout in addition to server.timeout to prevent temporary socket/memory leaking in keep-alive mode. PR-URL: #2534 Author: Timur Shemsedinov <timur.shemsedinov@gmail.com> Author: Alexey Orlenko <eaglexrlnk@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Franziska Hinkelmann <franziska.hinkelmann@gmail.com> Reviewed-By: Refael Ackermann <refack@gmail.com>

aqrln · 2017-05-26T16:08:03Z

Landed in 0aa7ef5 🎉

Implement server.keepAliveTimeout in addition to server.timeout to prevent temporary socket/memory leaking in keep-alive mode. PR-URL: #2534 Author: Timur Shemsedinov <timur.shemsedinov@gmail.com> Author: Alexey Orlenko <eaglexrlnk@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Franziska Hinkelmann <franziska.hinkelmann@gmail.com> Reviewed-By: Refael Ackermann <refack@gmail.com>

Fix the logic of resetting the socket timeout of keep-alive HTTP connections and add two tests: * `test-http-server-keep-alive-timeout-slow-server` is a regression test for nodejsGH-13391. It ensures that the server-side keep-alive timeout will not fire during processing of a request. * `test-http-server-keep-alive-timeout-slow-headers` ensures that the regular socket timeout is restored as soon as a client starts sending a new request, not as soon as the whole message is received, so that the keep-alive timeout will not fire while, e.g., the client is sending large cookies. Refs: nodejs#2534 Fixes: nodejs#13391

Fix the logic of resetting the socket timeout of keep-alive HTTP connections and add two tests: * `test-http-server-keep-alive-timeout-slow-server` is a regression test for GH-13391. It ensures that the server-side keep-alive timeout will not fire during processing of a request. * `test-http-server-keep-alive-timeout-slow-client-headers` ensures that the regular socket timeout is restored as soon as a client starts sending a new request, not as soon as the whole message is received, so that the keep-alive timeout will not fire while, e.g., the client is sending large cookies. Refs: #2534 Fixes: #13391 PR-URL: #13549 Reviewed-By: Refael Ackermann <refack@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Brian White <mscdex@mscdex.net>

gibfahn · 2018-01-15T21:37:37Z

Release team decided not to backport to v6.x. Comment if you would like it to be backported.

indutny reviewed Aug 25, 2015
View reviewed changes

mscdex added the http Issues or PRs related to the http subsystem. label Aug 25, 2015

ChALkeR added the memory Issues and PRs related to the memory management or memory footprint. label Aug 25, 2015

Fishrock123 reviewed Aug 25, 2015
View reviewed changes

olecom added a commit to olecom/supro that referenced this pull request Aug 27, 2015

http: fixed socket destruction, timeout and leaking #2534 (in connect)

6069153

issue: nodejs/node#2534 crutch from: http://habrahabr.ru/post/264851/ Signed-off-by: Oleg Verych <ole4at@gmail.com>

brendanashworth mentioned this pull request Sep 2, 2015

Keep alive connection do not get closed with server.close() #2642

Closed

rvagg force-pushed the master branch from 11c25c2 to ba02bd0 Compare September 6, 2015 11:55

ChALkeR mentioned this pull request Sep 6, 2015

A weird commit from 1970 #2713

Closed

indutny reviewed Sep 9, 2015
View reviewed changes

tshemsedinov reviewed Sep 10, 2015
View reviewed changes

aqrln reopened this May 24, 2017

refack reviewed May 25, 2017

View reviewed changes

refack approved these changes May 25, 2017

View reviewed changes

http: destroy sockets after keepAliveTimeout

b86b91f

Implement server.keepAliveTimeout in addition to server.timeout to prevent temporary socket/memory leaking in keep-alive mode.

refack approved these changes May 26, 2017

View reviewed changes

aqrln closed this May 26, 2017

jasnell mentioned this pull request May 28, 2017

8.0.0 Release Proposal #12220

Closed

brudil mentioned this pull request Jun 1, 2017

net::ERR_INCOMPLETE_CHUNKED_ENCODING in node v8.0 + chrome 58.0.3029.110 webpack-contrib/webpack-hot-middleware#210

Closed

pbininda mentioned this pull request Jun 2, 2017

Http connections aborted after 5s / keepAliveTimeout #13391

Closed

ChALkeR mentioned this pull request Jun 9, 2017

http: fix timeout reset after keep-alive timeout #13549

Closed

3 tasks

sourcesoft mentioned this pull request Jul 5, 2017

Getting net::ERR_INCOMPLETE_CHUNKED_ENCODING react-boilerplate/react-boilerplate#1840

Closed

cheton mentioned this pull request Dec 2, 2017

Question: The "others" event callback Snapmaker/Luban#2

Closed

ggoodman mentioned this pull request Apr 24, 2018

https?.Server.keepAliveTimeout introduced boundary condition that results in client-side socket hang-ups #20256

Open

		@@ -841,8 +841,25 @@ Note that the socket timeout logic is set up on connection, so
		changing this value only affects new connections to the server, not


		* {Number} Defaults to 5000 (5 seconds).

		The number of milliseconds of inactivity after a server has finished writing

http: fixed socket destruction, timeout and leaking #2534

http: fixed socket destruction, timeout and leaking #2534

Conversation

tshemsedinov commented Aug 25, 2015

Choose a reason for hiding this comment

ChALkeR commented Aug 25, 2015

tshemsedinov commented Aug 25, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Fishrock123 commented Aug 25, 2015

ChALkeR commented Sep 4, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

indutny commented Sep 9, 2015

tshemsedinov commented Sep 10, 2015

indutny commented Sep 10, 2015

indutny commented Sep 10, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ChALkeR commented Sep 10, 2015

indutny commented Sep 10, 2015

ChALkeR commented Sep 10, 2015

aqrln commented May 24, 2017 • edited Loading

aqrln commented May 24, 2017

aqrln commented May 24, 2017 • edited Loading

aqrln commented May 24, 2017 • edited Loading

refack left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aqrln May 25, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aqrln commented May 25, 2017

refack left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aqrln commented May 25, 2017

aqrln commented May 25, 2017

refack commented May 25, 2017

refack left a comment

Choose a reason for hiding this comment

aqrln commented May 26, 2017

gibfahn commented Jan 15, 2018

aqrln commented May 24, 2017 •

edited

Loading

aqrln commented May 24, 2017 •

edited

Loading

aqrln commented May 24, 2017 •

edited

Loading

aqrln May 25, 2017 •

edited

Loading