Piped responses truncated when running under node 0.8.x

[natevw]

I'm using this module to forward certain incoming requests to a backend server:

var dbReq = request({uri:dbURL, jar:false, pool:foreverPool});
req.pipe(dbReq).pipe(res);

// debugging code
dbReq.on('data', function (d) { console.warn("DATA FROM API REQUEST", d.toString()); });

However, when the dbReq returns a large amount of data the response often gets truncated running under node 0.8.x. I've confirmed that the upstream server is returning valid data, but the debugging logs on the 'data' events show only what the client is seeing: a bunch of output during the the first part of the dbReq response but then suddenly stopping.

Running under node.js 0.10.x seems to fix (or at least mask) the problem.

[lazdmx]

Same problem!

[joe-mojo]

same problem. piped exemple doesn't work. Node 0.10.22 / Express 3

[mikeal]

is the stream ending without finishing the data or is it getting paused?

[joe-mojo]

My web browser initiate a request. Node log shows the http loggers entries after a long time, and since those logs show up, browser say request is incomplete (no response)

[mbuttu]

I also ran into this issue using node@0.10.28, express@3.0.rc5, and request@2.39.0.

Upgrading to express@3.5.12 fixed the issue for me.

[mikeal]

Is this still an issue?

This is so old I'm closing, if it is actually still an issue just let me know and I'll re-open.

[joe-mojo]

Sorry but nothing changed so this is still an issue. I may be due to POST request excpting a form() parameter ?
Didn't tested if the doc for a custom PUT defining content-type as application/json is working with POST, I got the job done first with nodejs native http module and then with superagent.

[cmbaughman]

This is definitely still an issue! node@0.10.33, request@2.51.0. This was working fine actually using node@0.10.33, request@2.46.0, but after upgrade we started having the issue.

[arnihermann]

This is affecting a project I'm working on as well. I can't find a version combo that resolves the issue.

[arnihermann]

Ah, it seems if gzip is set to true, the response is truncated. Works fine with gzip false.

[rodw]

I'm running into what I think is the same issue.

Specifically I've got a program that writes the response to an HTTP(S) request to stdout using request(params).pipe(process.stdout).

When writing directly to the console (node foo.js) , things work fine.

When writing to a file (node foo.js > file.txt), things work fine.

When piping to another process (eg. node foo.js | cat or node foo.js | json_pp), the stream seems to be truncated.

The truncation appears to happen consistently after 1024 bytes (or characters, at least).

Strangely enough, while:

• node foo.js | cat seems to be truncated at 1024 bytes
• both node foo.js and node foo.js > foo; cat foo displays all 8814 bytes (for example)

each of:

• node foo.js > foo; wc foo
• node foo.js > foo; cat foo | wc
• node foo.js | wc
• and node foo.js | cat | wc

yield the same result (specifically 1 582 8814).

In other words, it may be the case that the truncation is only a visual one. While not all the characters appear on the terminal, the wc command sees them as the same (and correct) size.

Is it possible an extra \x00 byte is being added to the end of the block or something like that?

Note that all of this is independent of the specific data in my file -- this is not a function of the particular character found at the 1024th position in the file.

This is also independent of the command to which the stream is being piped, or at least there is more than one command that sees the truncated stream (e.g., the aforementioned cat and json_pp).

Setting gzip:false didn't seem to help me.

Versions:

• request@2.53.0
• node v0.10.31
• OSX Darwin Kernel Version 13.4.0 (MacBook Pro)

[nylen]

@rodw I suspect that it is just an intermittent issue rather than something related to visual display or which command comes next in the chain. Any chance you could give us a self-contained test case that either reports success or failure (and fails sometimes)?

[rodw]

@nylen There's really nothing intermittent about it - in the cases in which this "fails", it fails every time in exactly the same way and at exactly the same place. (Suspiciously right at 1024 bytes.)

I'll try to pull together a simple test case, but I don't think I'm doing much more than this example from the README request('http://google.com/doodle.png').pipe(fs.createWriteStream('doodle.png')) save that I'm using an HTTPS URL that returns JSON and piping it to process.stdout.

[rodw]

@nylen Oddly enough, it appears that the underlying issue is related to an interaction between request and a module called term-list.

For example, using this program:

var request = require('request');
request.get("https://github.com/request/request/issues/511").pipe(process.stdout);

the command:

node issue-511.js 

and

node issue-511.js | cat

give the same results.

However, if we load the term-list module:

var TermList = require('term-list');
var request = require('request');
request.get("https://github.com/request/request/issues/511").pipe(process.stdout);

Now:

node issue-511.js 

works the same as before, but:

node issue-511.js | cat

is truncated after 1024 characters and throws an error, as in the following output:

<!DOCTYPE html>
<html lang="en" class="">
  <head prefix="og: http://ogp.me/ns# fb: http://ogp.me/ns/fb# object: http://ogp.me/ns/object# article: http://ogp.me/ns/article# profile: http://ogp.me/ns/profile#">
    <meta charset='utf-8'>
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta http-equiv="Content-Language" content="en">


    <title>Piped responses truncated when running under node 0.8.x · Issue #511 · request/request · GitHub</title>
    <link rel="search" type="application/opensearchdescription+xml" href="/opensearch.xml" title="GitHub">
    <link rel="fluid-icon" href="https://github.com/fluidicon.png" title="GitHub">
    <link rel="apple-touch-icon" sizes="57x57" href="/apple-touch-icon-114.png">
    <link rel="apple-touch-icon" sizes="114x114" href="/apple-touch-icon-114.png">
    <link rel="apple-touch-icon" sizes="72x72" href="/apple-touch-icon-144.png">
    <link rel="apple-touch-icon" sizes="144x144" href="/apple-touch-icon-144.png">
    <met
stream.js:94
      throw er; // Unhandled stream error in pipe.
            ^
Error: write EPIPE
    at errnoException (net.js:904:11)
    at Object.afterWrite (net.js:720:19)

I don't know which module is doing something wrong (if either), but this sort of makes sense as term-list does work with stdout, possibly in a non-standard way.

I'm not sure if this is exactly the same issue as others on this thread are seeing, but the basic symptoms are quite similar.

[rodw]

Actually, it's not even term-list, the problem seems to be related to module that term-list loads called keypress.

The following program creates the issue:

require("keypress")(process.stdin);
var request = require('request');
request.get("https://github.com/request/request/issues/511").pipe(process.stdout);

but taking out line 1, or just changing it to require("keypress")(); does not create the issue.

[michael-ts]

Seeing this issue under node v4.4.7, request v.2.75.0 on Windows 10 (anniversary edition, under the Linux Subsystem). Same identical code runs fine on node v6.6.0, request v2.75.0 on OS X.

Have never used the term-list and keypress modules, and neither are present on the system.

I am only seeing it on fairly large sized resources.

For a specific URL where I expect a bit over 3344 * 1k bytes back, if I pass the option encoding:null for a specific URL (expecting ) it always stops at the same exact spot (2976 * 1k), but if I omit this it varies slightly - but it is still always on a 1k boundary. For instance in four runs it returned 2832, 2928, 2944 and 2976 times 1k bytes.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.