Binary protocol to avoid JSON encoding/decoding typed arrays (performance, bandwidth) vs. duplex-emitter

[deathcap]

duplex-emitter JSON encodes everything, even Uint8Array's (binary data, exploding from 1 byte up to 4 bytes. 255 => 255,)

http://nodejs.org/api/stream.html - supports 'object mode', possibly useful?

[deathcap]

Trying to understand the reason for duplex-emitter exactly..

https://npmjs.org/package/duplex-emitter
https://github.com/pgte/duplex-emitter

describes it as "Turns a duplex stream into an event emitter".

But http://nodejs.org/api/stream.html says "All streams are EventEmitters" - so why do duplex streams have to be turned into event emitters if they already are?

Looking at the code, appears that duplex-emitter serializes/deserializes the events into JSON then pipes to/from the streams. Uses https://npmjs.org/package/stream-combiner and https://npmjs.org/package/split to parse NDJ - Newline Delimited JSON. Two problems:

• Both WebRTC data channels and WebSockets are already message-based (onmessage events), so having duplex-emitter additionally handle its own message framing/unframing is redundant, possibly harmful
• JSON vs typed array encoding inefficiency (per above) - efficient alternative? Maybe BSON http://bsonspec.org/ - supports binary. https://npmjs.org/package/basalt native decoder + pure-JS. https://npmjs.org/package/msgpack "NodeJS Buffer values map to MSGPACK_OBJECT_RAW", but unpacks to string(?) https://github.com/msgpack/msgpack/blob/master/spec.md#formats-bin . For modern browsers, with types for storing buffers: https://npmjs.org/package/msgpack-js and https://github.com/juliangruber/msgpack-browserify for buffer in node, typed arrays in browsers. Reportedly slower than native JSON.parse. A custom packet format unpacked/packed manually (ala MC) would also be an option, but is undesirable. The only problem with JSON is binary data... (store it separately?)

[kumavis]

Streams are event emitters, but they only emit events about the stream (got data, went into error state). The duplexEmitter allows you to emit your own events (set a voxel, new client joined, avatar position update). I found this to be an easy way to program client-server interaction. That said, network throughput is a significant limiting factor, so we should squeeze as much perf as possible out of it.