Hitting Node.Stream limit constantly

[justinwoo]

I reported this upstream also (purescript-node/purescript-node-streams#15), but I regularly run into problem where if I try to send more than 16KB of data, the node streams will fail to write because of their "boolean flag for successful write" with event listener attaching monstrosity.

Is there an easy way for me to work around this somehow?

[justinwoo]

Ah, sorry, I just found out that I could create and write to my own buffer (e.g. justinwoo/vidtracker@60a51e5)

[owickstrom]

OK, so before you passed a String, which resulted in a Buffer of size over 16KB? Hyper should handle that, you shouldn't need to drop down to creating your own Buffers for that reason, IMO.

[justinwoo]

Yeah, I had a string that was barely 16KB so it would fail to write using the normal String respond method. Despite what it looks like the Node docs say, I think trying to write 16KB+ strings to streams/buffers will always return false, so it needs to be chunked afaict

[owickstrom]

OK, I'll have a look at this. It is indeed a nightmare to work with these NodeJS APIs. You never really know what you're doing, and if you're doing it right.

This would be a nice place to throw in some QuickCheck style tests, if the Node buffer writing code can be tested without too much hacking.

[frasermince]

What's the latest update on this? Is creating your own buffer still the best solution? Or is there a better way?

[owickstrom]

The PR #38 haven't been merged, but it seems to fix this issue in a somewhat hacky way. We need some more functionality exposed in Node.Stream to make it work better. I haven't had the time to send any PRs upstream.

[paluh]

We need some more functionality exposed in Node.Stream to make it work better.

It seems that there are no plans to expose streams events in purescript-node-streams currently. Maybe I'm wrong but it seems to me that it is hard to capture them in simple wrapper API.

I'm guessing that probably we should rich for some more powerful abstractions (from purescript-pipes or purescript-coroutines) to handle streaming with these events in an elegant manner.

[paluh]

It should not be a problem now: #55

Please reopen this issue if you are still experiencing this problem.