compatibility with through and native streams in pipeline/pipe

[michelle-becker]

Using es.pipeline to join together several es.pipe s of different streams seems to have some event based issues when more than one of the streams in pipe are created with through or es.through and the rest are subclasses of nodes native stream.Stream (Streams I).

The pipeline will work with all of the streams being subclasses of stream.Stream, and only one through stream, but where I put that through stream in the pipe seems to matter (has to be last). If I add more than one in that pipe, it breaks - the last node stream never gets an 'end' event.

I was wondering if there was a known compatibility issue with how these interact with each other - though I don't understand why this would be the case. Is it preferable that everything that goes through the pipeline be of one type of stream? i.e. all node streams or all through?

[michelle-becker]

After running some simple test cases, It seems like if I simply remove the intermediate es.pipes from the pipeline and simply put in the streams, the flow is much more stable.

However, I did notice that it seems it does make a difference if doing all through vs all node streams + one through stream, when there is an IO operation that needs to be preformed in the middle through stream. In the latter case, if a through stream that makes an async call (using pause/resume logic) is introduced into a pipeline of node streams, it doesn't look like things go missing from the final result set. Using all through streams seems to solve this problem.

A better solution, however, seems to be to use es.map for the async operation. When doing this, both of the pipelines work exactly the same. Because of this I am assuming the problem that I was seeing is in the way the pause and resume logic was implemented when doing the async operation.

For reference, this was the test file I used: https://gist.github.com/michelle-becker/6528230

Since It doesn't seem there was really and issue I'm going to close this. But I think it is interesting that the node streams handled the async problems less gracefully than the through streams did.