Stream data when syncing

[dmonad]

Currently. Yjs sends a big json object..

_{Huly®: YJS-90}

[t-mullen]

I'd be willing to take a shot at this if you'd like.

The initial sync message is the main concern, since it contains the entire model. Every time a peer joins, everything grinds to a halt trying to serialize this massive object.

[dmonad]

This would be great! But first we should agree on a streaming interface.

• Websockets support streaming, and there are plugins that allow streaming using socket.io.
• WebRTC should also allow streaming.

It would be great if you could first find efficient ways for streaming. Maybe propose an API that can be implemented with socket.io and simplewebrtc (actually I'd prefer simple-peer). But it is important that the streaming API is optional (there are communication protocols that don't support streaming).

I once tried to split up large messages into several smaller ones, and experienced several cases of data-loss using socket.io. But I'd be willing to try that out again.

[t-mullen]

We can just extend readable-stream's Duplex class, then users can pipe the stream any way they want, or listen to 'data' events (which we could wrap so that the old API is still present).

simple-peer is definitely the best option. I've built a connnector using it for my own project, so that will work.

I'm not sure why you would've seen data loss with socket.io, it's been pretty reliable for me. One thing to note is that it doesn't guarantee in-order delivery with the long-polling transport, so that would need to be disabled for YJS.

As for serialization, I was thinking we could send two types of messages. First, the normal JSON-serialized messages with all of YJS metadata but with no content, followed by chunks of content in raw bytes. This would skip any serialization of large strings and would be much faster. Basically a wire protocol.

[dmonad]

I'm not sure why you would've seen data loss with socket.io, it's been pretty reliable for me. One thing to note is that it doesn't guarantee in-order delivery with the long-polling transport, so that would need to be disabled for YJS.

I didn't know that

As a bottleneck seems to be the parsing and encoding, what do you think about JSONStream? You can implement it in every connector that supports node streams. Just adapt the send & broadcast methods to pipe the big JSON message through the stream.

[t-mullen]

That might work. It's still serializing everything, but not all at once, so it should be an improvement.

[dmonad]

I'm curious if there are going to be any significant performance changes. I also think its going to be a good step in the right direction.

[t-mullen]

It seems JSONStream only streams keys and values. This might not help us much as it seems the issue here is large values, not many values.

Would it be possible to define a fixed schema for messages and use something like Avro or schemapack? They are generally much faster, and wouldn't require much work to switch out.

[t-mullen]

I just noticed that the connectors actually handle serialization, so no code changes would be required at all, my bad.

As for a streaming API, for a start we could just use an object stream, then serialization and message splitting could remain in the connector.

[dmonad]

Streaming data would only improve the performance of the 'sync step 2' message. It contains a os property, which is an Array of possibly thousands of values. Each value is very small. I assumed that JSONStream could solve that.

JSON.stringify is not very performant. If this is not possible with JSONString (I don't know..), then it would make sense to write something similar. I think it would be rather easy to write a String transformer that transforms a JSON on the fly to string and back. Do you think that is possible?

[emreeren]

I'm using websocket connectors and had a problem with socket io ping timeouts. If sync step 2 does not complete in 30 seconds (or whatever timeout it set to) connection resets. So I added socket io streaming support on top of original websocket client and server implementation. My implementation is specific (typescript & hardcoded stuff) to my project for now but if anyone interested I released them under https://github.com/emreeren/y-websockets-streaming-server and https://github.com/emreeren/y-websockets-streaming-client.

And I have a question.

Server sends the size of buffer before streaming starts and client compares the size of the received data with it. If it matches client calls self.receiveMessage('server', buffer);. The size may not match because of connection issues. What can we do in this case? Should we just ignore it or somehow force it to reconnect?

https://github.com/emreeren/y-websockets-streaming-client/blob/master/src/y-websockets-streaming-client.ts#L95