json parser buffer pipe requests

[impronunciable]

I'm using superagent for https://github.com/danzajdband/Tuiter

When using a stream method (like tuiter#sample) I'm using the superagent pipe method. Because the response content-type is json (I think that's the reason) the request object is buffering the stream inside res.text.

This is critical for my lib because this buffer is increasing apps memory usage in ~ 1.5MB / sec.

Do you think that's the reason? Is a Superagent issue or am I doing something wrong?

[tj]

woah hmm, any idea how big the average response is? You can opt-out of buffering with .buffer(false), hopefully later we'll have a lot of these settings at the agent-level instead of per request

[impronunciable]

I solved using a noop custom parser. buffer(false) is not working, that's the problem. The sample method of the Streaming API send ~1MB data per second so res.text grows at the same level.

Again, I solved for the library using .parse(function(){}) but .buffer(false) seems to fail on this situation, I think is an edge case (I'm receiving json data but the connection will persist "forever") but we'll have more cases like this in the near future.

[tj]

hmm interesting, I just remembered that I set this already in .pipe(): https://github.com/visionmedia/superagent/blob/master/lib/node/index.js#L383, if it's not working there's definitely a bug, I'll label this issue and check it out later

[impronunciable]

I tried using .buffer(false) and .pipe() (and both together), none of this work. I think the problem is with the response type signature.

[defunctzombie]

Closing this old issue. If something similar comes up we can revisit. The pipe logic is less than ideal at the moment anyway.