You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I have tested complex multiple-core (using node.cluster) data pipeline at around 20k rows per second and am very happy with the results so far. The only issue (hence this ticket) that I have experienced so far is nextTick warnings and maximum recursion errors when there is an aggregation of data somewhere in mid-pipeline (possibly coming from an SQL endpoint).
I
The working solution is to use setImmediate on the callbacks, allowing pending IO to finish (in the _transform prototype for streamz):
See pull-stream/stream-to-pull-stream#1 (comment) for example streamz production use. @bnoon I'd be very intested to hear how this is going and if you have been facing any immediate problems?
I have tested complex multiple-core (using node.cluster) data pipeline at around 20k rows per second and am very happy with the results so far. The only issue (hence this ticket) that I have experienced so far is nextTick warnings and maximum recursion errors when there is an aggregation of data somewhere in mid-pipeline (possibly coming from an SQL endpoint).
I
The working solution is to use setImmediate on the callbacks, allowing pending IO to finish (in the _transform prototype for streamz):
I wonder whether this is a potentially more fundamental issue with the @joyent Node Readable, where setImmediate might be more appropriate than nextTick, i.e. https://github.com/joyent/node/blob/master/lib/_stream_readable.js#L532-L534
Anyway, this needs further investigation.
The text was updated successfully, but these errors were encountered: