Add Readable and Writeable Stream APIs

[ARitz-Cracker]

Come on, man. I'd love to just, create a stream, pipe that steam into streams, pipe streams into files, pipe streams into network ports, pipe streams into pipes, stream pips into files, pipe....

oh also, how about you make write return a promise for when the internal buffers fill up? That'd be neat, in fact, that's one thing I hate about NodeJS's streams, for some reason they don't mix with async functions well and you gotta make/use a wrapper with it and that makes me sad.

I'm sure with enough effort, spiderfire will be the key to ending all my sadness. Wouldn't that be cool?

Oh, and files and networks stuffs shouldn't be the only thing that's streams, stdin/stdout/stderr should also have streams! Streams everywhere! Just like UNIX intended.

[ARitz-Cracker]

streams are the only thing keeping me from using this in prod btw, so implement streams and I'll help you END the V8 monopoly 😎

[Redfire75369]

As of Bug 1795914, the Streams implementation in SpiderMonkey has been removed. Since streams are a relatively complex spec, this means they will delayed for now, while more important features are implemented.

[guest271314]

Streams API (ReadableStream, WritableStream, TransformStream, ReadableStreamBYOBReader, ReadableStreamBYOBRequest) are implemented on Mozilla Firefox 116.0a1 20230703204942. It looks like WHATWG Streams are implemented in DOM: Streams https://bugzilla.mozilla.org/buglist.cgi?product=Core&component=DOM%3A%20Streams.

[Redfire75369]

I've started work on ReadableStream in https://github.com/Redfire75369/spiderfire/tree/feature/readable-stream.
It will take a while, but it should be compliant with the WHATWG Streams spec.

[Arshia001]

@Redfire75369 Rather than reimplement the streams spec from scratch, which will take considerable time, Spiderfire could use this fork of mozjs (which we're currently using for WinterJS): https://github.com/wasmerio/mozjs/tree/wasi-gecko

The fork uses this version of SpiderMonkey that has implementations for both readable and writable streams.

We also have an implementation for TransformStream (missing back-pressure ATM), TextEncoderStream and TextDecoderStream in WinterJS that could end up being useful to have in Spiderfire.

[Redfire75369]

Although this is certainly an option, I have chosen not to go with that approach, since even if the relevant patches are reverted, the implementation is still unmaintained and may lead to issues when upgrading to more recent mozjs versions.

Also, last I checked, they never completed the WritableStreamimplementation.

[sagudev]

It would be interesting if servo and spiderfire could share streams impl (at least part of it).

[guest271314]

Isn't this all implemented using Rust? Deno's WHATWG Streams implementation works as expected. Just use what Deno uses?

[Redfire75369]

Deno's implementation is written in JS. (https://github.com/denoland/deno/blob/d1f4d81dcf6a03463d82a89d058337acecca0588/ext/web/06_streams.js)

[guest271314]

Use that? Complie to WASM if necessary? Here's txiki.js' implementation in C https://github.com/saghul/txiki.js/blob/9cad93b8299cb61b6bcff081ac5a83b97c9d2404/src/streams.c.

[Redfire75369]

The issue is not the implementation being written in JS per se. The issue is that its difficult to interface with streams and to have things like native sources through the Rust code.

[guest271314]

I thought Rust was very flexible at JavaScript/WASM <=> Rust interface?

Anyway, you'll figure it out eventually.

[Arshia001]

@guest271314 you're probably thinking of wasm_bindgen, which is an entirely different story.

[guest271314]

What I'm saying is various JavaScript runtimes have figured out how to implement WHATWG Streams. You any means necessary. wasm_bindgen, whatever. Just make sure the streams are transferable, the same as they are in the browser.

[guest271314]

Additionally, make sure the streams actually stream. Bun had an issue with the ReadableStream implementation where in the server would read the entire stream before serving the data to the client. CloudFlare's Workerd implementation does not allow, by deliberate decision of the maintainers, an uploaded stream to be served to a subsequent request. Don't do that. That is one way we can implement close to full-duplex streaming in the browser https://gist.github.com/guest271314/4847c70be203ee7cd4c8a6d6a9bca1d3, save for the edge case of between a ServiceWorker and a WindowClient on Chromium, as WHATWG Fetch still has not spelled out the mechanics of full-duplex streaming in the browser, while deno and node both support full-duplex streaming using fetch() https://github.com/guest271314/telnet-client.

[guest271314]

Pardon, the telnet-client repository is a different experiment, is one way I implement full-duplex streaming from the browser

• https://github.com/guest271314/native-messaging-deno/tree/fetch-duplex
• https://github.com/guest271314/native-messaging-nodejs/tree/full-duplex

[guest271314]

I'll be checking back here periodically. FWIW just added spiderfire to this list https://gist.github.com/guest271314/bd292fc33e1b30dede0643a283fadc6a. Good luck!

[Redfire75369]

Streams: Writable & Transform