Occasional errors piping CouchDB via node.js #195

jmarca opened this Issue Feb 28, 2012 · 7 comments


None yet
4 participants

jmarca commented Feb 28, 2012

I apologize for not having a better bug report, but I am noticing an occasional error using the x.pipe(y) pattern to push couchdb data out in response to web requests.

The version of request that I am using is request@2.9.152

The error has two facets: a crash with an error report, and a silent fail with no further activity from Request.

The crashing error is as follows, but I think this is just a symptom of the silent fail, not the actual problem:

Error: This request has been piped before http.request() was called.
    at Request.resume (/home/james/repos/jem/carbserver/node_modules/tracker_rewriter/node_modules/request/main.js:702:29)
    at ServerResponse.ondrain (stream.js:48:14)
    at ServerResponse.emit (events.js:64:17)
    at Socket.ondrain (http.js:1322:27)
    at Object.afterWrite [as oncomplete] (net.js:488:28)

The code that triggers this error is as follows. (BTW, the code is borrowed shamelessly (but with attribution) from maxogden/rewriter, so it should be easy to reproduce.)

var couch = "";
exports.rewriter = function(req,res,next){
    var to = couch + '/vdsdata%2ftracking/'+req.params.id;
    if(req.params.attachment !== undefined){
        to += '/'+req.params.attachment;

My app does two things. First it asks for a document, and then pulls out all of the document's attachments (png files) and displays them. that part of the app uses the above service and when it crashes produces the above error.

The app also gets a list of the 5,000 or so document ids in a separate request, asking for 1000 docs at a time. This part of the app is the part that triggers the silent fail (Request freezes up). It uses the following service (very similar to the above).

exports.vdsdetectors = function(req,res,next){
    var to = couch + '/vdsdata%2ftracking/_design/vdsml/_view/mainline';
    var query = req.query
        to += "?" + toQuery(query)

As I said, the bug seems to have two manifestations. The most obvious one is the crash, and I have found that it only happens after calling the listing service, but it also only happens from the one-at-a-time service (get a doc or get the doc's attachments). I cannot get that crash to happen by holding down control-r on a stripped down version of my app that just gets one document and its attachments.

The silent fail, in which Request freezes and refuses to work anymore (couchdb is fine) only occurs when triggering the request to get 1000 documents a bunch of times (I hold down control-r in ffox). I can get that one to happen every time, and it will usually fail after working only about 6 times (it varies). If I hit refresh at a more modest pace, the crash doesn't happen.

In case it is relevant, I am using these services within the connect framework, with a bunch of other things turned on...logging requests, session cookies, etc.

I hope this bug report makes sense, but if it doesn't or if you have any suggestions on what else to try, please let me know.

mikeal closed this in 7b35b4f Feb 28, 2012

jmarca commented Feb 29, 2012

I am still seeing the bug where request stops working with the new version. I will try to write a test case and open up a new issue.

garbados commented Jul 7, 2013

I'm also still seeing this issue with silent failures. Requests never hit CouchDB, and hang forever on the server. GET requests work as expected, but PUT and POST do not. Code in question:

function(req, res){
  var db_url = '...';

mikeal commented Jul 7, 2013

are you sure they are hitting the server? you could be exhausting the default pool in nodejs.

Aye, console.log statements work up until the pipe.

Fixed it, though I don't completely understand the underlying problem. Express' body parser modifies the request object in a way that causes problems for proxies, so you can resolve the issue by making the proxy into middleware that catches before the body parser activates.

Still not sure why the pipe fails silently.


mikeal commented Jul 23, 2013

this sounds like another case where Connect breaks streams. this is why a lot of people don't use Connect.

badunk referenced this issue in AceMetrix/npm-artifactory Sep 16, 2013


Connection drops during large tarball GETs #21

badunk commented Sep 16, 2013

@garbados Thank you! I had the same problem where my pipes were breaking silently. I, too, was using the bodyParser.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment