maxBuffer error when streaming logs from a remote host

[saidimu]

Specifically when streaming Docker container logs, the following error is encountered:

Error: stdout maxBuffer exceeded.
    at Socket.<anonymous> (child_process.js:678:13)
    at Socket.EventEmitter.emit (events.js:117:20)
    at Socket.<anonymous> (_stream_readable.js:746:14)
    at Socket.EventEmitter.emit (events.js:92:17)
    at emitReadable_ (_stream_readable.js:408:10)
    at emitReadable (_stream_readable.js:404:5)
    at readableAddChunk (_stream_readable.js:165:9)
    at Socket.Readable.push (_stream_readable.js:127:10)
    at Pipe.onread (net.js:528:21)

Any particular reason maxBuffer is capped to 1MB?

shipit/lib/shipit.js

Line 188 in feb2804

maxBuffer: 1000 * 1024

[vvo]

@neoziro You should use .spawn which is aimed at launching things that you don't know how many output it will have.

Exec is good for small programs with little output.

[gregberge]

Yes @vvo is right.. But for simplification I use exec, because it's very convenient to get all the output in a var stdout in 99% of cases.

@saidimu I added this limit because if your program output more than 1MB I think it's interesting to be alerted. You can modify this limit by yourself using options.

[vvo]

@neoziro it's really easy to use spawn and also output program/capture it and stream it on console or anywhere else. This will happen many times

[gregberge]

@vvo stream it to the console is not the problem, the problem is for the user who want to get the stdout result easily. It could be an option, but I think it must not be the default choice. Shipit is not designed to be a tool for power users who master streams.

[vvo]

Of course you must abstract it maybe with a logger object but the thing is that you have a unknown length output of commands (you cannot know how much time it will take, how much output it will have). So you must use spawn.

It's really a small change but I don't care in the end I don't use shipit :)

[luizfilipe]

Would be good get this working from a large output
Since there's no way to customize it, this make shipit < capistrano

[coder36]

I've hit this issue when doing a rsync task. Any work arounds ?

[gregberge]

Probably solved in v4, I need time to work on it :(.