Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Occasional errors piping CouchDB via node.js #195

Closed
jmarca opened this Issue · 7 comments

4 participants

@jmarca

I apologize for not having a better bug report, but I am noticing an occasional error using the x.pipe(y) pattern to push couchdb data out in response to web requests.

The version of request that I am using is request@2.9.152

The error has two facets: a crash with an error report, and a silent fail with no further activity from Request.

The crashing error is as follows, but I think this is just a symptom of the silent fail, not the actual problem:

Error: This request has been piped before http.request() was called.
    at Request.resume (/home/james/repos/jem/carbserver/node_modules/tracker_rewriter/node_modules/request/main.js:702:29)
    at ServerResponse.ondrain (stream.js:48:14)
    at ServerResponse.emit (events.js:64:17)
    at Socket.ondrain (http.js:1322:27)
    at Object.afterWrite [as oncomplete] (net.js:488:28)

The code that triggers this error is as follows. (BTW, the code is borrowed shamelessly (but with attribution) from maxogden/rewriter, so it should be easy to reproduce.)

var couch = "http://127.0.0.1:5984";
exports.rewriter = function(req,res,next){
    var to = couch + '/vdsdata%2ftracking/'+req.params.id;
    if(req.params.attachment !== undefined){
        to += '/'+req.params.attachment;
    }
    request(to).pipe(res);
}

My app does two things. First it asks for a document, and then pulls out all of the document's attachments (png files) and displays them. that part of the app uses the above service and when it crashes produces the above error.

The app also gets a list of the 5,000 or so document ids in a separate request, asking for 1000 docs at a time. This part of the app is the part that triggers the silent fail (Request freezes up). It uses the following service (very similar to the above).

exports.vdsdetectors = function(req,res,next){
    var to = couch + '/vdsdata%2ftracking/_design/vdsml/_view/mainline';
    var query = req.query
    if(query)
        to += "?" + toQuery(query)
    request(to).pipe(res);
}

As I said, the bug seems to have two manifestations. The most obvious one is the crash, and I have found that it only happens after calling the listing service, but it also only happens from the one-at-a-time service (get a doc or get the doc's attachments). I cannot get that crash to happen by holding down control-r on a stripped down version of my app that just gets one document and its attachments.

The silent fail, in which Request freezes and refuses to work anymore (couchdb is fine) only occurs when triggering the request to get 1000 documents a bunch of times (I hold down control-r in ffox). I can get that one to happen every time, and it will usually fail after working only about 6 times (it varies). If I hit refresh at a more modest pace, the crash doesn't happen.

In case it is relevant, I am using these services within the connect framework, with a bunch of other things turned on...logging requests, session cookies, etc.

I hope this bug report makes sense, but if it doesn't or if you have any suggestions on what else to try, please let me know.

@mikeal mikeal closed this issue from a commit
@mikeal mikeal Removing old checks for self.req, it's ensured if start() is called. …
…Implementing early pause/resume for when streams try to pause/resume before any data is emitted. Fixes #195.
7b35b4f
@mikeal mikeal closed this in 7b35b4f
@jmarca

I am still seeing the bug where request stops working with the new version. I will try to write a test case and open up a new issue.

@garbados

I'm also still seeing this issue with silent failures. Requests never hit CouchDB, and hang forever on the server. GET requests work as expected, but PUT and POST do not. Code in question:

function(req, res){
  var db_url = '...';
  req.pipe(request(db_url)).pipe(res);
}
@mikeal
Owner

are you sure they are hitting the server? you could be exhausting the default pool in nodejs.

@garbados

Aye, console.log statements work up until the pipe.

@garbados

Fixed it, though I don't completely understand the underlying problem. Express' body parser modifies the request object in a way that causes problems for proxies, so you can resolve the issue by making the proxy into middleware that catches before the body parser activates.

Still not sure why the pipe fails silently.

@mikeal
Owner

this sounds like another case where Connect breaks streams. this is why a lot of people don't use Connect.

@badunk badunk referenced this issue in AceMetrix/npm-artifactory
Closed

Connection drops during large tarball GETs #21

@badunk

@garbados Thank you! I had the same problem where my pipes were breaking silently. I, too, was using the bodyParser.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.