Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

HTTP server request socket is dropping data? #2691

Closed
craigyk opened this Issue · 6 comments

3 participants

@craigyk

So I have this strange problem where my http ServerRequest is losing a chunk of data somewhere between the entry point of the handler function, and the point in the function where the request gets piped into a file stream.

https://gist.github.com/1736167

Not sure how to proceed, except maybe hack something out? Do I just pause the request and resume it later explicitly?

update: doing a request.pause() at the top of the function and then resume() after connecting the pipe seems to have worked... but this still seems hackish? Surely the ServerRequest shouldn't be losing/dropping that first chunk when there isn't anyone yet listening for 'data' events?

@koichik
Owner

@craigyk - Please note that as a rule we do not support third-party modules (of course includes CoffeeScript).

Surely the ServerRequest shouldn't be losing/dropping that first chunk when there isn't anyone yet listening for 'data' events?

No. In fact, you should add 'data' listener before the stream emits 'data' event. Otherwise, data is lost. If you use v0.6.x or older, see also #1040.

@koichik koichik closed this
@craigyk

Wait... hold on a sec...
Shouldn't the ServerRequest queue incoming data until it gets at least one listener attached? Since JS is so async driven isn't it a given that there might be a couple of async operations between the arrival of the request and a listener being attached? In my case I'm trying to open a file to pipe the request data into, but it takes a bit of work before the ground is laid to safely 'open' that file for writing: the path to the file needs to be checked, and potentially created, locks, etc.

I would have thought that something as important as ServerRequest would throttle and queue incoming data until at least one function was ready to listenfor data? If not, that's a pretty big gotcha that's not well documented.

Why should

function(req,res) {
    var stream = unsafelyCreateStream();
    req.pipe(stream);
}

work (not always, since it's not safe, but you get the idea), but

function(req,res) {
    safelyCreateStream(function(stream){
        req.pipe(write_stream);
    });
}

not work?, Instead I have to do

function(req,res) {
    req.pause();
    safelyCreateStream(function(stream){
        req.pipe(write_stream);
        req.resume();
    });
}

which I guess is OK, but isn't obvious from the docs...

@koichik
Owner

@craigyk

Shouldn't the ServerRequest queue incoming data until it gets at least one listener attached?

No, because it is by design.

If not, that's a pretty big gotcha that's not well documented.

I agree. We should write it clearly.

Please note that the third snippet does not work with v0.6.x or older (probably, you can use mikeal/morestream).

@koichik koichik referenced this issue from a commit
koichik doc: add the note about 'data' event
Refs #2691.
a2cd31c
@koichik
Owner

@craigyk - documented in a2cd31c.

@isaacs
Owner

@craigyk If we buffered data until a listener is assigned, then you'd have problems in cases where you don't care about the data.

http.createServer(function (req, res) {
  switch (req.url) {
    case '/upload': req.on("data", handleUpload); req.on("end", endUpload); break
    default:
      res.writeHead(404); res.end('Not found'); break
      // leaking all the memory!!
  }
})
@craigyk

Good point, though I was thinking it would buffer, and throttle back the incoming packets, if such a thing is possible... I assumed that was the end-result of pausing() a socket.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.