Using standard node.js based streams

[AdrianRossouw]

Hey David,

Very interested in what you've been doing here, and it's great to see that you are still building stuff like this.

I personally love streams as an abstraction, and I was wondering if you considered using straight node.js streams (as per the stream handbook ). There's a really large ecosystem for this kind of thing already, and because a lot of this stuff handles data too.

I think if this system coould allow you to easily wire up data sources into the graph, there could be some really simple abstractions to be made.

[dmvaldman]

Hi Adrian, nice to hear from you again!

I can see where you're coming from. I'm kind of reinventing the wheel here, but there are good reasons for it. I may consider making the API compatible with Node in the future. Currently, EventEmitter has the same API as Node's. The reason streams are different is several-fold. One, there's no need for read and write streams as we're not doing filesystem stuff. The UI focus also needs me to have a start event in addition to Node's data (update) and end events. I've also gone for subscribe over pipe, which is a difference of "pull" versus "push" and has some performance advantages. For one, there's no need for backpressure logic. But I may support pipe in the future as there are use cases for it. It would definitely be a huge win to be able to piggy-back off the node ecosystem.

Do you know of data-agnostic libraries that use Node streams? I haven't done much research here.

[AdrianRossouw]

Well. for each point.

1. node streams are specialised event emitters, so i'd recommend starting out by just using the event emitter as a base to begin with.
2. the read/write distinction is a valid concern, but the majority of streams you deal with end up being Transform/through streams or Duplex streams. Which can both be read and written from.
3. While not called start, there is a 'readable' event which gets fired when a stream starts.
4. Custom stream implementations are completely free to add additional events to emit.
5. Node streams can be both push or pull streams.

Well, I guess it depends on what you mean by "data agnostic". All node streams don't particularly care about what you are streaming through them. The biggest distinction is whether they are running in binary/text, or in objectMode.

You should probably check out highlandjs, which is by the guy who wrote the async.js library. My favorite part of it is that you can use promises, callbacks and generators directly. It is not exactly compatible with node streams tho.

If you wanna chat about this stuff sometime, i'd be open to it. My interest in famous came about because I was looking to use it in a stream-oriented architecture as part of my graft project, so i've done a lot of research on the subject.

[dmvaldman]

As this is not an issue with the code itself, I'd prefer it be raised in the google group: groups.google.com/forum/#!forum/samsarajs

Feel free to repost there!