Store documents in a stable format

[ClementEXWiki]

Currently, Docs stores all documents in a Y.js buffer format, which does not have any stability guarantee.
Y.js messages are only supposed to be stored in order to be sent to currently-offline users.

If Y.js breaks their format, or if BlockNote breaks their own (which happens from time to time), many documents will break.

Documents should be stored in a stable format, be it Markdown, HTML, anything really, even a standardized JSON object would be fine :)

[AntoLC]

It is important to save a YDoc to have the collaboration working correctly.
More informations here: https://docs.yjs.dev/api/document-updates

[ClementEXWiki]

I recently had the opportunity to discuss with some of the (lovely) persons working on BlockNote, and I understand that my initial comment was wrong ; Docs doesn't store update buffers but rather snapshots that can be used to rebuild the original Y.js document.

But, you can absolutely make realtime work without storing Y.js documents. That's what we do in our app that also uses BlockNote, and it works flawlessly.

Y.js is designed to enable collaboration even with documents that are built directly from the "source" content, without any history.

[AntoLC]

Hi, initially we saved the Blocknote json but when users are collaborating and a new user arrives, it didn't sync correctly.

Curious to see how you are doing so, could you show me how you initialize your provider ?

[ClementEXWiki]

Basically the way we do it in our project is:

• If the client is the first to connect, the editor is directly initialized with the document's content
• If there are other players in the realtime session, we sync the current user with them

You can achieve this by using storing some configuration in the provider's document, e.g.:

provider.on("synced", () => {
  const initialContentLoaded = provider.document
    .getMap("configuration")
    .get("initialContentLoaded");

  if (!initialContentLoaded) {
    provider.document
      .getMap("configuration")
      .set("initialContentLoaded", true);

    console.debug("Setting initial content for realtime first player", {
      content,
    });

    // here we directly set the editor's content
  }

  // here the editor is ready
});

[AntoLC]

Hi @YousefED! Would be nice to have your opinion about it !
It is true that it could be easier to just save the json instead of the yDoc base64.

Here the implementation I think: https://github.com/xwiki-contrib/cristal/blob/082c068e51128a5e2684de1ddf39cce46870ec4c/editors/blocknote-headless/src/react/BlockNoteView.tsx

[YousefED]

Nice discussion! To get the complete picture; is the requirement for this purely about stability of the format, or are there other requirements as well?

My main immediate concern would be with offline users. How do you know whether or not any clients are making changes to the document while offline?

Cc @nperez0111

[nperez0111]

Hey all, allow me to clear up some things here:

Currently, Docs stores all documents in a Y.js buffer format, which does not have any stability guarantee.

The Y.js update encoding has been stable & backwards compatible since it began as a project (~2011), many systems only use this format for storage, and expect to continue to do so. It is always extract-able to json, if you can read the document in clear text.

Side note, it isn't technically a "format", a format determines the structure of some data, whereas Y.js is merely an encoding for document content in this case (since you can go from Y.js <-> JSON interchangeably).

Docs doesn't store update buffers but rather snapshots that can be used to rebuild the original Y.js document.

Yes & no, updates can both encode a snapshot of a document at a point in time (e.g. recreating the entire document & it's history), or can encode as a "patch" which is the diff of the document content that 1 peer has that another doesn't. If you change a character in an editor that emits an update in this "patch update", which is not distinguished from a full document snapshot, since a full snapshot is technically just when you have no initial content & need the patch of initialState + whole doc.

But, you can absolutely make realtime work without storing Y.js documents. That's what we do in our app that also uses BlockNote, and it works flawlessly.

This is an "it depends" scenario, if you can make guarantees like "no one else has this document open", "no one is offline with this document", then yes it could work.

The problem is that document merging properties only work on Y.js documents which share history, this gives each document enough context to know what the "intended change" was & converge on that same edit across the documents. If you have one user which is using only a JSON object & manipulates that, you lose all guarantees of Y.js, as when you go to merge that solution it becomes either last-write-wins (like your solution has right now), or at best you can attempt to make a diff.

Summary

While we can talk more about the formats that docs should support (especially in regards to durability when projects are changing like BlockNote), Y.js should be treated as the source of truth for collaborative documents (since it's merging properties are only valid against other Y.js update). It is okay to break this rule of thumb in specific scenarios like single user editing but, not in the general case.

One approach often recommended is to store not just Y.js updates in the database, but also to store some secondary format alongside it. This makes it easier to retrieve the document content for things which are read-only, but all writes should be made by Y.js updates.

There is definitely some changes I would make for how Docs does snapshotting (should be server-side), and the format they store to their database (should be binary not base64), but this is not really relevant for this issue.

I hope this clears some things up!