Real-time collaboration on self-hosted

[brnpl89]

We have the application self-hosted in our own infrastructure and we want to understand if real-time collaboration is possible as it's not currently working. Is this real-time feature implemented with Pusher API? If so, is it possible to enable if we create our own account?

[davidjgraph]

We don't have RT working in the stand-alone project. Where are you saving your diagram to?

[davidjgraph]

OK, that's not a simple architecture to support. Building the RT in a generic way for the open project isn't simple. We use some very specific services, namely cloudflare workers, legacy GAE with memcache and pusher. Apart from being system specific the problems with creating an RT for this project that is generic are 1) It's a lot of work and 2) we wouldn't be using it in production so it'd simply break.

[davidjgraph]

But, it probably is possible to enable a slow collaboration that uses Google Drive directly.

[brnpl89]

What do you mean by using Google Drive directly? Can you point us in that direction?

[davidjgraph]

I can't currently, sorry, I'm working.

[reinoutkoninkx]

I have also implemented a custom real time collaboration. You can use the eventlisteners to listen to 'message', then filter on 'merge', here you can intercept the xml from the drawing, send this to all other users using some sort of real time server. Making your own RTC can be tricky though

[VendettaMike]

I asked a similar question but wasn't quite clear in the title (/rt and /cache implementations). In our instance, we're hosting internally (Apache/Tomcat) and using OneDrive so we have the storage side ready, just not the RT side.

From a quick look at the code, it appears that /cache is used with Pusher.js, but P2PCollab.js seems to be based on Simplepeer but I haven't found any documentation on it so I'm not sure of it's status - is it still in testing or is it stable enough to use as a replacement for Pusher? I assume I need to setup a signaling server and set the RT_WEBSOCKET_URL as a start. Is there a specific signaling server that should be used and/or code to help implement this in a self-hosted environment?

[VendettaMike]

I dug into the code a bit more to figure out how everything works as it appears I didn't fully understand it and I'm probably still a bit fuzzy on some of the finer details.

DrawioFileSync.js. manages the real-time file changes. When a diagram is modified, a notification message is sent to the /cache endpoint which is handled by Tomcat. Tomcat uses memcache and Pusher Java classes to manage these notifications and send them to the Pusher API. Each client has a Pusher websocket to receive the notifications and respond accordingly.

P2PCollab.js is used to initiate the Pusher process and process notifications received from the Pusher websocket and call DrawioFileSync.js as needed. It also uses a separate websocket service (the /rt endpoint in the hosted version) and SimplePeer to create a peer-to-peer connection between browsers for the real-time cursor and selection display.

As David mentions, it's quite a complex system and not something that can be easily implemented. There's the cache and Pusher portion that's needed - I do see this servlet in an archived project but not sure if it's too out of date to be used or not. I may try and see if I can get this to work.

The cursor and selection websocket server looks simple enough that it wouldn't be too difficult to create your own.

It would be great to have a self-hosted Pusher service but that's not a simple task.

Hope this helps.

[jprtf]

@VendettaMike are you still pursuing RT? Have you seen this?

[VendettaMike]

@miki-hmt mentioned Laravel Websockets which I'm guessing was how it was implemented for the demo that was shared https://live.csdn.net/v/325551

I did review Laravel Websockets which is PHP based, and found Soketi which is Node.js based, but we don't have PHP or Node.js implemented on our Diagrams.net internal server right now so I need additional time to install and test to determine which one will fit our needs. Based on our deployment schedule for our internal tool, there just wasn't enough time but I should be looking at this again in 3-4 weeks.

[davidjgraph]

There are three parts, the fast (real time RT) part. That's written in JS and runs on cloudflare workers. That's a sub/pub fast channel where updates go out immediately.

The cache on GAE is slower. It is a correctly ordered sequence of changes that are only recorded once the client correctly writes the file. The result returned from there can be used by the client to playback changes in a different order to that received from the fast RT. https://gist.github.com/davidjgraph/9e19257bfc0d0535db0d6ab601d649dd is the cache code.

The last fallback is just the file itself. If we can't resolve something, we load the currently file and apply the diff-sync.

[VendettaMike]

Appreciate the clarification. I haven't gone thoroughly enough through the code to fully understand how they work together.

And thanks for the link to the cache source.

[miki-hmt]

Thank you for the ideas and code provided by the author. I have implemented the function of local privatization and multi person collaboration：
https://live.csdn.net/v/325551

[jprtf]

Hi! Is the code of your implementation available on GitHub?

[miki-hmt]

not，it just works in onedrive

[VendettaMike]

Are you using Pusher or did you implement your own private service? We're hoping to eliminate Pusher so everything is using an internal/private system.

[jprtf]

@miki-hmt My question was if you will share your code on Github.
我的问题是你是否会在Github上分享你的代码。

[miki-hmt]

@VendettaMike I recommend you privatize the deployment of laravel websockets

[brnpl89]

@m-mohamedin We were looking into the docker-drawio repository and we noticed that the self-contained implementation supports real-time collaboration using AWS IoT: https://github.com/jgraph/docker-drawio/tree/dev/self-contained.

In one of the steps, it is mentioned to follow the instructions in the etc/mxPusher folder to setup a lambda function for temporary keys as well as setting a role for that lambda function but we cannot locate this folder neither on the repository nor the container files. Where can we find these instructions?

[m-mohamedin]

This was an old project that is cancelled and not maintained anymore. We'll remote it from the docs

[jprtf]

@m-mohamedin what are the options to enable RT on self-contained?
Is the RT service single handed by Pusher? If so, would be possible to configure in a similar way to the EMF converter, where we create an account and setup the API key?

[m-mohamedin]

I think so, the account info goes in pusher.properties file. Please check David reply above for more details