fix: should not destroy streams #612

Closed
wants to merge 1 commit into
from

Projects

None yet

6 participants

@dead-horse
Member

use black-hole-stream to make sure stream's data has been read.

since user may set this.body to a http IncomingMessage, if we destroy this stream, it will destroy the http socket that make keep-alive failed, and may affect other http requests reusing this socket.
I think maybe we shouldn't destroy streams, only need to insure streams' data will be read and don't leak.

@dead-horse dead-horse added bug 1.x labels Dec 2, 2015
@JacksonTian JacksonTian and 2 others commented on an outdated diff Dec 2, 2015
},
"engines": {
- "node": ">= 0.12.0",
- "iojs": ">= 1.0.0"
+ "node": ">= 0.12.0"
@dead-horse
dead-horse Dec 2, 2015 Member

can we drop the support of node < 4 in koa@1 ? /cc @tj @jonathanong

@JacksonTian
JacksonTian Dec 2, 2015

oh, master is v1.

@juliangruber
juliangruber Dec 4, 2015 Member

this should be its own pull request

@dead-horse dead-horse referenced this pull request in ali-sdk/ali-oss Dec 2, 2015
Open

getStream dont use keepalive feature #40

@fengmk2
Member
fengmk2 commented Dec 4, 2015

LGTM

@juliangruber juliangruber commented on an outdated diff Dec 4, 2015
- "4"
+ - "5"
@juliangruber
juliangruber Dec 4, 2015 Member

this should be its own pull request

@juliangruber juliangruber commented on an outdated diff Dec 4, 2015
test/application.js
@@ -872,6 +876,69 @@ describe('app.respond', function(){
.get('/')
.expect(404, done);
})
+
+ it('should insure stream do not leak', function(done){
@juliangruber
juliangruber Dec 4, 2015 Member

insure => ensure

@dead-horse
Member

@juliangruber all fixed

@juliangruber juliangruber and 1 other commented on an outdated diff Dec 4, 2015
test/application.js
@@ -872,6 +876,69 @@ describe('app.respond', function(){
.get('/')
.expect(404, done);
})
+
+ it('should ensure stream do not leak', function(done){
+ done = pedding(3, done);
+ var app = koa();
+ let stream1 = fs.createReadStream(__filename);
+ let stream2 = fs.createReadStream(__filename);
+ stream1.once('end', done);
@juliangruber
juliangruber Dec 4, 2015 Member

i don't understand why this one would end as well? it's never read from

@juliangruber
Member

hmmm, this seems more like an edge case to me. are there other cases that would benefit from this change?

@dead-horse
Member

The most serious problem now is http keepalive sockets will be destroyed, and we must fix this issue. Not sure any other side effect if we destroy the streams.

@juliangruber
Member

are you sure destroying a http stream will affect other requests on the same socket? sounds more like a node bug to me

@dead-horse
Member

If you destroy an IncommingMessage, it will destroy the socket behind: https://github.com/nodejs/node/blob/master/lib/_http_incoming.js#L92

That will affect other IncommingMessage's that reusing the same socket and will cause socket hang up error. This is reported in ali-sdk/ali-oss#40 (sorry this issue is in chinese).

@dead-horse
Member

any suggestions? I'll merge this PR if no response in the next day. :)

@dead-horse dead-horse self-assigned this Dec 5, 2015
@juliangruber
Member

Hey man, no need to put a deadline on it. If you need it urgently use a fork or find a userland solution.

@dead-horse
Member

I think this is just a bug fix. so if anyone doubt this patch, I'll leave it here and fix in our own framework for now. :)

@juliangruber
Member

this patch fails if the body stream is endless, like for example:

this.body = fs.createReadStream('/dev/urandom');

or more real-worldy

this.body = serverSentEvents(createUpdateStream());

in the cases above it's required to destroy the stream.

@juliangruber
Member

and trying to read it all out will never finish

@dead-horse
Member

although i don't think pipe an endless stream to response meaningful. but maybe there is a solution has less impact. detect the stream in destroy and don't destroy IncommingMessage(a breaking change in destroy).

@dead-horse dead-horse referenced this pull request in stream-utils/destroy Dec 6, 2015
Closed

do not destroy IncommingMessage #2

@dead-horse dead-horse fix: should not destroy streams
use black-hole-stream to make sure stream's data has been read
d9c1ea3
@dead-horse
Member

@juliangruber how about this?

@juliangruber
Member

too edge-casy. koa should not need to worry about this.

Just to be sure, this would work for you, right?

httpStream.destroy = blackHole.bind(null, httpStream);
this.body = httpStream;
@dead-horse
Member

it is a common usage that use koa as a http proxy, and anybody who use this.body = yield request('http://foo.bar/'); //keep-alive will hit this "edge case", and it is really hard to figure out why. so I think we should fix it in framework level, because it is a bug in koa, and we can't let everyone to use httpStream.destroy = blackHole.bind(null, httpStream); hack.

@jonathanong
Member

maybe a better solution is to just wrap the http stream in another stream so you never set the http stream as the actual .body=.

const PassThrough = require('stream').PassThrough

app.use(function * (next) {
  this.body = httpStream.on('error', this.onerror).pipe(PassThrough())
})

would need docs on this though... super edge case

@juliangruber
Member

would need docs on this though... super edge case

what about documenting that a body stream can be .destroyed and showing how to guard against that

@tejasmanohar
Member

what about documenting that a body stream can be .destroyed and showing how to guard against that
Show all checks

+1 @juliangruber

@tejasmanohar tejasmanohar referenced this pull request Feb 25, 2016
Open

Koa 2.0.0 #533

@jonathanong jonathanong commented on the diff Feb 25, 2016
lib/response.js
@@ -161,8 +164,15 @@ module.exports = {
}
// stream
- if ('function' == typeof val.pipe) {
- onFinish(this.res, destroy.bind(null, val));
+ if (val instanceof Stream) {
+ onFinish(this.res, function(){
+ // don't destroy http IncomingMessage, keep `keep-alive` conncetion alive.
+ if (val instanceof http.IncomingMessage) {
+ if (val.readable) val.pipe(new BlackHoleStream());
@jonathanong
jonathanong Feb 25, 2016 Member

why do you need black hole stream? just val.resume() to dump it

@jonathanong
Member

anyways prefer going the documentation route. will close this when i or someone else adds documentation

@dead-horse
Member

It's ok to add some warning in documentation.

@dead-horse dead-horse closed this Feb 26, 2016
@dead-horse dead-horse deleted the fix-destroy-everty-stream branch Feb 26, 2016
@dead-horse dead-horse added documentation and removed bug labels Feb 26, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment