add protocol documentation

PROTOCOL.md

@@ -0,0 +1,101 @@
+
+# Protocol Spec
+
+y-protocol implements different binary communication protocols for efficient exchange of information.
+
+This is the recommended approach to exchange Awareness updates, and syncing Yjs documents & incremental updates.
+
+## Base encoding approaches
+
+We use efficient variable-length encoding where possible.
+
+The protocol operates on byte arrays. We use the `•` operator to define concatenations on array buffers (i.e. `[1, 2] = [1] • [2]`). A "buffer" shall refer to any byte array.
+
+* `varUint(number)` - encodes a 53bit unsigned integer to 1-8 bytes.
+  - unsigned integers are serialized 7 bits at a time, starting with the
+    least significant bits.
+  - the most significant bit (msb) in each output byte indicates if there
+    is a continuation byte (msb = 1).
+  - A reference implementation can be found in lib0: [encoding.writeVarUint](https://github.com/dmonad/lib0/blob/1ca4b11f355c5ccec25f20ac8d3e2382e6c4303c/encoding.js#L243) [decoding.readVarUint](https://github.com/dmonad/lib0/blob/1ca4b11f355c5ccec25f20ac8d3e2382e6c4303c/decoding.js#L235)
+* `varByteArray(buffer) := varUint(length(buffer)) • buffer` - allows us to read any buffer by prepending the size of the buffer
+* `utf8(string)` - transforms a string to a utf8-encoded byte array.
+* `varString(string) := varByteArray(utf8(string))`
+* `json(object) := varString(JSON.stringify(object))` - Write a JavaScript object as a JSON string.
+
+### Sync protocol (v1 encoding)
+
+The Sync protocol defines a few message types that allow two peers to efficiently sync Yjs documents with each other. For more information about Yjs updates and sync messages, please look at [Yjs Docs / Document Updates](https://docs.yjs.dev/api/document-updates).
+
+We initially sync using the state-vector approach. First, each client sends a `SyncStep1` message to the other peer that contains a `state-vector` (see Yjs docs). When receiving `SyncStep1`, one should reply with `SyncStep2` which contains the missing document updates (`Y.encodeStateAsUpdate(remoteYdoc, sv)`). Once a client receives `SyncStep2`, it knows that it is now synced with the other peer. From now on all changes on the Yjs document should be send to the remote client using an `Update` message containing the update messages generated by Yjs.
+
+#### Message types
+
+* `SyncStep1MessageType := 0`
+* `SyncStep2MessageType := 1`
+* `UpdateMessageType:= 2`
+
+#### Encodings
+
+* `syncStep1(sv) := varUint(SyncStep1MessageType) • varByteArray(sv)` - Initial sync request. The state vector can be received by calling `Y.encodeStateVector(ydoc)`.
+* `syncStep2(documentState) := varUint(SyncStep2MessageType) • varByteArray(documentState)` - As a reply to `SyncStep1`. The document state can be received by calling `Y.encodeStateAsUpdate(ydoc, sv)`.
+* `documentUpdate(update) := varUint(UpdateMessageType) • varByteArray(update)` - Incremental updates the Yjs event handler `Y.on('update', update => sendUpdate(update))`. The receiving part should apply incremental updates to the Yjs document `Y.applyUpdate(ydoc, update)`.
+
+### Awareness protocol
+
+The Awareness protocol synchronizes the pure state-based Awareness CRDT between peers. This can be useful to exchange ephemeral data like presence, cursor positions, etc..
+
+Each peer is allocated a unique entry in the Awareness CRDT that only they can modify. Eventually, this state is going to be removed by either a timeout or by the owner setting the state to `null`. 
+
+Since the Awareness CRDT is purely state-based, we always exchange the whole state of all locally known clients. Eventually, all Awareness CRDT instances will synchronize. The Awareness CRDT must remove clients from clients that haven't been updated for longer than `30` seconds. With each generated update, the `clock: uint` counter must increase. The Awareness CRDT must only apply updates if the received `clock` is newer / larger  than the currently known `clock` for that client.
+
+`awarenessUpdate(clients) := varUint(clients.length) • clients.map(client => varUint(client.clientid) • varUint(client.clock) • json(client.state))`
+
+
+## Combining protocols
+
+The base protocols can be mixed with your own protocols. The y-protocol package only defines the "base" protocol layers that can be reused across communication providers.
+
+* `SyncProtocolMessageType := 0`
+* `AwarenessProtocolMessageType := 1`
+
+A message should start with the message-type (e.g. `SyncProtocolMessageType`) and be appended with a specify protocol message (e.g. `SyncStep1MessageType`)
+
+* E.g. encoding a SyncStep1 message over the communication protocol: `varUint(SyncProtocolMessage) • syncStep1(sv)`
+* E.g. encoding an awareness update over the communication protocol: `varUint(AwarenessProtocolMessageType) • awarenessUpdate(clients)`
+
+A communication provider could parse protocols as follows:
+
+```js
+
+import * as decoding from 'lib0/decoding'
+import * as encoding from 'lib0/encoding'
+import * as sync from 'y-protocols/sync'
+import * as awareness from 'y-protocols/awareness'
+
+const messageTypes = {
+  [SyncProtocolMessageType]: sync.readSyncMessage,
+  [SyncProtocolMessageType]: awareness.readAwarenessMessage,
+  [YourCustomMessageType]: readCustomMessage
+}
+
+function readMessage (buffer) {
+  const decoder = decoding.createDecoder(buffer)
+  const messageType = decoding.readVarUint(decoder)
+  const replyMessage = encoding.createEncoder()
+
+  const messageHandler = messageTypes[messageType]
+  if (messageHandler) {
+    messageHandler(decoder, encoder, ydoc)
+    if (encoding.length(encoder) > 0) {
+      // the message handler wants to send a reply (e.g. after receiving SyncStep1 the client should respond with SyncStep2)
+      provider.sendMessage(encoding.toUint8Array(encoder))
+    }
+  } else {
+    throw new Error('Unknown message type')
+  }
+}
+
+```
+
+
+