Content resent after hocuspocus server restarts

[zeljko-bulatovic]

Hi,

I occurred one more issue during collaboration setup :/ After hocuspocus restarts, existing content from tiptap v2 is sent again, which results doubling the data in the content.

Steps to reproduce:

1. Open tiptap v2 editor
2. Enter some text, for example 'Test'
3. Stop hocuspocus server
4. Start hocuspocus server again

Error: Front end sends 'Test' content again to hocuspocus, which results with two 'Test' content paragraphs in database / file.

NB: After you restart it again, two 'Test' paragraphs will be sent, which results with 4 'Test' items in the database..

Front end is configured in the following way:

const ydoc = new Y.Doc();
const token = user.token;
const uri =  'ws://localhost';
this.provider = new WebsocketProvider(uri, 'tiptap-example', ydoc, {
	params: {
		token
	}
});
this.provider.on('status', event => {
	this.status = event.status;
});

this.editor = new Editor({
	extensions: [
		...defaultExtensions().filter(extension => extension.config.name !== 'history'),
		Underline,
		Image,
		Document,
		Paragraph,
		Text,
		CollaborationCursor.configure({
			provider: this.provider,
			user: this.currentUser,
			onUpdate: users => {
				this.users = users;
			},
		}),
		Collaboration.configure({
			provider: this.provider,
			document: ydoc
		}),
	],
});

Thank you for help in advance!

[kriskbx]

Hey Zeljko. Thank you so much for your bug report. Can you share your backend/hocuspocus code as well?

[kriskbx]

Do you use the @hocuspocus/extension-rocksdb as primary storage? If not, you definitely should.
If you do, do you check in your implementation of the onCreateDocument hook if the document is already present in the primary storage?

const hocuspocus = Server.configure({
  async onCreateDocument(data) {
    const fieldName = 'default'

    // Check if the given field already exists in the given document.
    // Important: Only import a document if it doesn't exist in the primary data storage!
    if (data.document.isEmpty(fieldName)) {
      return
    }

   // import the document from somewhere else…
   return new Doc()
   }
}

The background is: Serializing the Y-Doc to Prosemirror JSON in the onChange hook and deserializing it in the onCreateDocument hook, doesn't save the collaboration history. And thus, Yjs doesn't know that those changes are the same and just merges them.

Because saving a Y-Doc with the related collaboration history can be hard, we built the @hocuspocus/extension-rocksdb which is meant to be your primary storage and makes this incredibly easy (it's just one line to implement).

You can read more on the need for a primary storage here: https://www.hocuspocus.dev/guide/documents#using-a-primary-storage

[zeljko-bulatovic]

Hi Kris, here is the collab server code:

class CollaborationServer {
	_server = null;
	_debounced = null;
	_debounceTime = 4000;

	constructor() {
		this._server = Server.configure({
			port: process.env.PORT || 80,
			// allows up to 5 connection attempts per ip address per minute.
			// set to null or false to disable throttling
			throttle: false,
			// bans ip addresses for 5 minutes after reaching the throttling threshold
			// banTime: 500,
			onConnect: this.onConnect,
			onCreateDocument: this.onCreateDocument,
			onChange: this.onChange
		});
	}

	async listen() {
		return this._server.listen();
	}

	async onConnect(data) {
		const { requestParameters } = data;
		const token = requestParameters.get('token');
		if (!token) {
			throw new Error('Auth token not provided');
		}

		const user = await getUserByToken(token);
		if (!user) {
			throw new Error('Authentication failed');
		}

		// You can set contextual data to use it in other hooks
		return {
			user: {
				id: user.id,
				name: user.fullName,
			},
		}
	}

	async onCreateDocument(data) {
		// The tiptap collaboration extension uses shared types of a single y-doc
		// to store different fields in the same document.
		// The default field in tiptap is simply called "default"
		const fieldName = 'default';

		// Check if the given field already exists in the given y-doc.
		// Only import a document if it doesn't exist in the primary data storage
		if (data.document.get(fieldName)._start) {
			return;
		}

		const namespace = data.documentName.split('/');
		if (namespace.length !== 2) {
			return;
		}

		const [ projectId, itemId ] = namespace;
		const prosemirrorDocument = await getItem(itemId);
		if (!prosemirrorDocument) {
			return;
		}

		// When using the tiptap editor we need the schema to create
		// a prosemirror JSON. We must list all extensions that are used on the front-end
		const schema = getSchema([...defaultExtensions(), Image.configure({inline: false}), Link, Underline]);

		// Convert the prosemirror JSON to a ydoc and simply return it.
		return prosemirrorJSONToYDoc(schema, prosemirrorDocument, fieldName);
	}

	async onChange(data) {
		const save = async () => {
			// Get the underlying Y-Doc
			const ydoc = data.document;

			// Convert the y-doc to the format your editor uses, in this
			// example Prosemirror JSON for the tiptap editor
			const prosemirrorDocument = yDocToProsemirrorJSON(ydoc, 'default');
			const [ projectId, itemId ] = data.documentName.split('/');;
			const newContent = prosemirrorDocument;
			// Updates database
			await updateItemContent(itemId, newContent);
		};

		if (this._debounced) {
			this._debounced.clear();
		}

		this._debounced?.clear();
		this._debounced = debounce(() => save, this._debounceTime);
		this._debounced();
		save();
	}
}

export {
	CollaborationServer
};

It seems that i miss-understood the purpose of the rocksDb, i thought that it can be used instead of PostgreSQL for example (which i am using). I didn't find any tutorial regarding the mix of rocksDb with other databases, that's why i didn't include it and used PostgreSQL directly (is that wrong?).

Also, i can't find any example how to configure UI for using IndexedDB with v2.. I found in example that it is imported, but not set anywhere (or importing is enough? i was looking https://next.tiptap.dev/examples/collaborative-editing)

[kriskbx]

I didn't find any tutorial regarding the mix of rocksDb with other databases, that's why i didn't include it and used PostgreSQL directly (is that wrong?).

It's not wrong per se, but it will lead to issues like the one you described above. Your setup looks fine, just include the rocksdb extension and it should work. :)

Theoretically, you could also store the collaboration history in your PostgreSQL database, but it's hard. Especially because there is no documentation at the moment on how to do it. We ourselves just kinda brute-forced our implementation: https://docs.yjs.dev/tutorials/persisting-the-document-to-a-central-database

Also, i can't find any example how to configure UI for using IndexedDB with v2.. I found in example that it is imported, but not set anywhere (or importing is enough? i was looking https://next.tiptap.dev/examples/collaborative-editing)

I recently removed the indexedDB from tiptaps collaboration example because we had some issues with it and hocuspocus. Long story short: We completely changed how hocuspocus worked a few weeks back and it resulted in corrupted data which was stored in the clients indexeddb and completely destroyed our collab-system because Y-js always wanted to merge the corrupted data… So, nothing to worry about for you! It's pretty simple to set up, just import it, create a new instance and pass it the Y-Doc:

import { IndexeddbPersistence } from 'y-indexeddb'

const ydoc = new Y.Doc()
const indexdb = new IndexeddbPersistence('tiptap-collaboration-example', ydoc)

[kriskbx]

One comment on your implementation:

async onChange(data) {
  const save = async () => { 
    // ...
  }

  // you can remove this, it's basically the same as this._debounced?.clear()
  if (this._debounced) {
    this._debounced.clear();
  }

  this._debounced?.clear();
  this._debounced = debounce(() => save, this._debounceTime);
  this._debounced();

  // debouncing is meant to take load off your database server
  // calling save here directly will just make the whole debouncing useless
  save();
}

[kriskbx]

Make sure to use the latest version of @hocuspocus/extension-rocksdb (>= 1.0.0-alpha.42) as we fixed some issues with the underlying rocksdb adapter in this release. If you experience errors after updating, delete your database folder and restart the server.

[zeljko-bulatovic]

Hi Kris,

It works very good now! Thank you for suggestions!

[kriskbx]

Very nice! I'm happy it works now. 🙌