Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: should not destroy streams #612

Closed
wants to merge 1 commit into from

Conversation

dead-horse
Copy link
Member

use black-hole-stream to make sure stream's data has been read.

since user may set this.body to a http IncomingMessage, if we destroy this stream, it will destroy the http socket that make keep-alive failed, and may affect other http requests reusing this socket.
I think maybe we shouldn't destroy streams, only need to insure streams' data will be read and don't leak.

},
"engines": {
"node": ">= 0.12.0",
"iojs": ">= 1.0.0"
"node": ">= 0.12.0"

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

node >= 4

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we drop the support of node < 4 in koa@1 ? /cc @tj @jonathanong

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, master is v1.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should be its own pull request

@dead-horse
Copy link
Member Author

ping @jonathanong @fengmk2

@fengmk2
Copy link
Member

fengmk2 commented Dec 4, 2015

LGTM

- "4"
- "5"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should be its own pull request

@dead-horse
Copy link
Member Author

@juliangruber all fixed

var app = koa();
let stream1 = fs.createReadStream(__filename);
let stream2 = fs.createReadStream(__filename);
stream1.once('end', done);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i don't understand why this one would end as well? it's never read from

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we'll destroy(consume) all stream's that set to this.body here https://github.com/koajs/koa/pull/612/files#diff-d86a59ede7d999db4b7bc43cb25a1c11R166

@juliangruber
Copy link
Contributor

hmmm, this seems more like an edge case to me. are there other cases that would benefit from this change?

@dead-horse
Copy link
Member Author

The most serious problem now is http keepalive sockets will be destroyed, and we must fix this issue. Not sure any other side effect if we destroy the streams.

@juliangruber
Copy link
Contributor

are you sure destroying a http stream will affect other requests on the same socket? sounds more like a node bug to me

@dead-horse
Copy link
Member Author

If you destroy an IncommingMessage, it will destroy the socket behind: https://github.com/nodejs/node/blob/master/lib/_http_incoming.js#L92

That will affect other IncommingMessage's that reusing the same socket and will cause socket hang up error. This is reported in ali-sdk/ali-oss#40 (sorry this issue is in chinese).

@dead-horse
Copy link
Member Author

any suggestions? I'll merge this PR if no response in the next day. :)

@dead-horse dead-horse self-assigned this Dec 5, 2015
@juliangruber
Copy link
Contributor

Hey man, no need to put a deadline on it. If you need it urgently use a fork or find a userland solution.

@dead-horse
Copy link
Member Author

I think this is just a bug fix. so if anyone doubt this patch, I'll leave it here and fix in our own framework for now. :)

@juliangruber
Copy link
Contributor

this patch fails if the body stream is endless, like for example:

this.body = fs.createReadStream('/dev/urandom');

or more real-worldy

this.body = serverSentEvents(createUpdateStream());

in the cases above it's required to destroy the stream.

@juliangruber
Copy link
Contributor

and trying to read it all out will never finish

@dead-horse
Copy link
Member Author

although i don't think pipe an endless stream to response meaningful. but maybe there is a solution has less impact. detect the stream in destroy and don't destroy IncommingMessage(a breaking change in destroy).

use black-hole-stream to make sure stream's data has been read
@dead-horse
Copy link
Member Author

@juliangruber how about this?

@juliangruber
Copy link
Contributor

too edge-casy. koa should not need to worry about this.

Just to be sure, this would work for you, right?

httpStream.destroy = blackHole.bind(null, httpStream);
this.body = httpStream;

@dead-horse
Copy link
Member Author

it is a common usage that use koa as a http proxy, and anybody who use this.body = yield request('http://foo.bar/'); //keep-alive will hit this "edge case", and it is really hard to figure out why. so I think we should fix it in framework level, because it is a bug in koa, and we can't let everyone to use httpStream.destroy = blackHole.bind(null, httpStream); hack.

@jonathanong
Copy link
Member

maybe a better solution is to just wrap the http stream in another stream so you never set the http stream as the actual .body=.

const PassThrough = require('stream').PassThrough

app.use(function * (next) {
  this.body = httpStream.on('error', this.onerror).pipe(PassThrough())
})

would need docs on this though... super edge case

@juliangruber
Copy link
Contributor

would need docs on this though... super edge case

what about documenting that a body stream can be .destroyed and showing how to guard against that

@tejasmanohar
Copy link
Member

what about documenting that a body stream can be .destroyed and showing how to guard against that
Show all checks

+1 @juliangruber

@tejasmanohar tejasmanohar mentioned this pull request Feb 25, 2016
onFinish(this.res, function(){
// don't destroy http IncomingMessage, keep `keep-alive` conncetion alive.
if (val instanceof http.IncomingMessage) {
if (val.readable) val.pipe(new BlackHoleStream());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do you need black hole stream? just val.resume() to dump it

@jonathanong
Copy link
Member

anyways prefer going the documentation route. will close this when i or someone else adds documentation

@dead-horse
Copy link
Member Author

It's ok to add some warning in documentation.

@catamphetamine
Copy link

Didn't understand a thing from the documentation on what's happening here.
It's just like "warning, never set a stream as ctx.body".

@jiajianrong
Copy link

@catamphetamine, they were warning not to set a http-request-alike-stream which has keep-alive enabled. Because the framework will close the stream by default, which will break keep-alive feature.
Keep-alive means to reuse tcp socket connection for two or more http connections.

Generally, it's OK with ctx.body = fs-read-stream.
Just be careful with ctx.body = request('host:port/path').

@rhinoceros
Copy link

@dead-horse
cnpmjs.org中目前用的koa的版本是"koa": "^1.2.0", 如果使用oss作为nfs, 在从oss上get的时候会有一堆的sockets hang up。

有什么好的办法能避免这类错误吗?

cnpm/cnpmjs.org#1401

谢谢

image

@zhangwilling
Copy link

@dead-horse 马哥好,昨天夜里见到这个 issue 忘了跟你打招呼了,白天来踩一下。

----- 骄傲的用中文交流

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

10 participants