mux-demux streams can't be fed buffers

[garthk]

Per gist 3253513, piping streams involving Buffers rather than strings to mux-demux results in the other end of the mux stream sending data events with buffer-shaped JavaScript objects rather than anything you could pipe further, e.g. into a socket or file stream.

[dominictarr]

This was a problem with stringify in event-stream. I've patched it in event-stream@2.1.9. can you reinstall and test?

[garthk]

I've installed mux-demux@2.0.0 with event-stream@2.1.9, but mux_demux_buffer_demo.js still shows buffer-like-objects rather than actual buffers.

[garthk]

I'm not sure converting the buffer to a string is the right approach: it presumes the other end is happy with the default utf8 encoding and knows when/if the source has been encoded. My patch to mux-demux is somewhat grotty, but ensures that Buffer objects always come out the other end as Buffer objects.

[dominictarr]

oh, of course, my fix didn't work in mux-demux because the buffer is put inside an array, so it's stringified as normal.
for some strange reason Buffer().toString() === '' inside a JSON reviver function. weird!

I'm not gonna merge this patch, because as you say it's somewhat grotty,
but I have another solution.

I've patched mux-demux to support custom serialization formats.
you just need to pass in a function that wraps the internal js object stream
in a stream that serialization. some examples are provided in the readme & tests.

using msgpack-stream should work for your purposes.

also, what I'd like to have a stream interface for the redis protocol.
it would fit perfectly for sending messages embedded in arrays,
as I do in a number of my stream modules including mux-demux.

here is a redis parser.
https://github.com/mranney/node_redis/blob/master/lib/parser/javascript.js

[dominictarr]

oh, you want mux-demux@2.1.0

[dominictarr]

this issue is related nodejs/node-v0.x-archive#3848

[juliangruber]

I have that usecase where some of the streams are binary and others aren't. What should I do? I converted binary to string streams and that works.

[dominictarr]

you could use msg-pack instead of json. here is an example https://github.com/dominictarr/mux-demux/blob/master/test/msg-pack.js
that will preserve buffers, even within js objects.

[tlrobinson]

Shouldn't mux-demux support buffers out of the box? I'd really like to be able to just pipe binary data around if I want. A binary protocol that's more efficient than JSON would be nice too.

EDIT: Nevermind, I didn't see the msgpack version. It works beautifully!

A note in the README about buffers would be nice.

[dominictarr]

@tlrobinson good point - fixed in 3.7.9