Use a BYOB readable stream to optimise the QUICStream readable stream

[tegefaulkes]

Specification

Currently the readable stream in QUICStream uses a single buffer that stream_recv writes to and gets sliced, copied and enqueued into the readable stream. Previously this was just sliced and enqueued. There was a fix in #63 to copy the sliced buffer to prevent all enqueued buffers referencing the same memory but this has a performance cost associated with it.

We want to look into changing the readable stream to a BYOB readable stream. This will allow us to cut down on copies by allowing us to pass in buffers to read into. This should remove the need to copy or create a new buffer for each read by using the provided buffer.

We'll need to check if BOYB readable streams can be used the same way as normal readable streams.

Additional context

• Related: ci: merge staging to master #63
• Commentary: ci: merge staging to master #63 (comment)

Tasks

1. replace the QUICStream readable stream with a BYOB readable stream.
2. Check that this stream can be used the usual way and also by passing in a buffer to write to.

[CMCDragonkai]

Relevant discussion here: 5fce16d#r1349859926

Basically copying the chunks is good for UX, but it's not good for performance, since it's not part of the streams spec then I think it is implementation-specified.

So by supporting BYOB, we allow the user to be able to recover the more efficient sharing of buffers.

The same thing should be done in js-ws @amydevs.

[tegefaulkes]

After experimenting with the BYOB ReadableStream, I'm starting to think it might not be a good fit for the following reasons.

1. It will only function as BYOB if you use the 'byob' mode on the reader. byob.getReader({mode: 'byob'}). so we can only use the 'byob' otherwise we'd have to implement both styles of handling pull. The byob mode works with a pull that uses the normal controller.enqueue style, but that will mean we can't re-use a single buffer still.
2. You can't re-use the same buffer when reading in byob mode. First read mutates the buffer to <Buffer > and 2nd read complains about the 0-length buffer. This kind of defeats the point since you can't re-use 1 buffer for the whole process.

I can go into more detail. but from what I've seen this doesn't fulfil our needs.

That said, we need to fix the problem where we have a 1MB buffer for each readable stream. I think we just need to make that buffer much smaller on the order of a few KB.

[CMCDragonkai]

Is this something that can be solved with an alternative - something that is running on the rust side?

Also just to sanity check this, can you throw your requirements into ChatGPT (mention what you have tried), ask it for alternative solutions.

[tegefaulkes]

Chat GPT didn't have any more insight than I had. https://chat.openai.com/share/5323e80c-f2af-4a8c-9c65-8394c3c965f7

Moving forward we're just going to significantly reduce the buffer size used by the streams. It should be a simple change. We'll need to do some bench-marking to see the difference. I think performance will be the same unless we have very large messages. But for our RPC usage that really shouldn't be the case.

I'm focusing on other stuff now so I'll get back to this.

[CMCDragonkai]

I am kind of confused here.

During some benchmarks we saw that the longest time was doing the CPU operation of reading or writing into the stream on the rust side.

Due to passing control between JS side and Rust side this slowed down other operations due to single threadedness but also FFI cost.

One idea is to introduce multithreading to the rust side.

If that occurs then we still have to copy data between JS side and Rust side right?

Reducing the number copying that occurs would a good idea.

BYOB was supposed to help reduce of number of copies. However we have to beware of the possibility of asynchronous clobbering.

So I'm not entirely understanding the constraints right now.

[tegefaulkes]

There are 3 main constraints

1. reduce the amount to held memory needed
2. reduce the number of new buffers created
3. reduce the amount of copied data.

Ideally the BYOB stream was meant to solve all 3. But from experimenting with it, it falls short in 2/3 ways.

1. When doing a read we won't know how many bytes are available to read. So each each read takes a buffer of a set size without know how many bytes will be read. Not a problem if we only have the 1 buffer.
2. To reduce the number of new buffers we can just re-use the 1 buffer to read each message. but from experimenting, you pass in the buffer to be written to, and a new buffer is returned, presumably the same buffer you passed in. But the buffer you have passed in is now modified to be 0-length and can't be re-used for the next message. So we have to create a new buffer for each read.
3. It would actually reduce the number of copies by 1 for each message. But the extra copy per message is really fast and not the bottleneck here.

It light of these 3 things I don't think using the BYOB would be worth the extra complexity. On top of that it only functions as a BYOB if you're using the BYOB mode reader. So it's very limiting.

For comparison, reducing the main buffer for the QUICStream only addresses constraint 1. But it's the main problem right now.

[tegefaulkes]

Closed as wontfix, replaced with #83

[tegefaulkes]

Whoops, closed it the wrong way.