Self hosting Excalidraw - Umbrella issue #1772
Comments
|
Worth mentioning that atm:
|
|
just to pitch in an alternative approach. I have experimented with this, and created a bundle - single binary - which will work for selfhosting (on mac/win/linux/rasperry pi) on local network. (currently only link sharing is implemented) https://github.com/kjellkvinge/excalidrawserver Is this interesting? It was a fun hack :) regards |
|
Made a comment in https://github.com/excalidraw/excalidraw-json/issues/76#issuecomment-650463667 but will re-post here for visibility:
|
|
Thank you @NMinhNguyen for sharing. Indeed the socket server needs only a Dockerfile and it would be ready to be self hosted along the main Excalidraw client. For the storage one, we will have to think the current storage solution and storing diagrams in the filesystem makes sense I think. So with a Dockerfile and shared volume on the host, people will be able to self host that as well. Thank you again |
|
@NMinhNguyen did you have any issues when serving the site over HTTP? Or do you plan he use HTTPS in all cases? |
I don't actually have much Docker experience but we have a PaaS solution (somewhat similar to GAE in some ways) that understands Node.js so all I had to do was push the source code + dependencies but yeah :) I'll need to think about the best way to parameterise endpoint URLs in the If you have any suggestions (or links to good Docker resources), I'm willing to look into those too.
The PaaS (Cloud Foundry) is able to serve via HTTPS so I didn't have any issues but you're right in that |
|
So I've been thinking a bit more about this and I'm wondering if the easiest way to make the backend URLs configurable, is to have some kinda REACT_APP_BACKEND_V2_GET_URL=... npx @excalidraw/cli build
# this then creates a `public` folder or somethingThe CLI itself could take flags like
Let me know if you think this approach is wack 😛 |
|
@NMinhNguyen, from my observations, the by far most common way of configuration for docker images is by environment variables. Sometimes there are also some command line parameters or config files involved, but not a manual build. |
Essentially the problem here is that environment variables are read only once when you do |
|
in my experiment i can do: But this require that the backendurls use this variable. I solve this with a patch like this, which I apply in a build step: diff --git a/src/data/index.ts b/src/data/index.ts
index f9c3bec..793d28e 100644
--- a/src/data/index.ts
+++ b/src/data/index.ts
@@ -24,12 +24,18 @@ export { loadFromBlob } from "./blob";
export { saveAsJSON, loadFromJSON } from "./json";
export { saveToLocalStorage } from "./localStorage";
-const BACKEND_GET = "https://json.excalidraw.com/api/v1/";
-
-const BACKEND_V2_POST = "https://json.excalidraw.com/api/v2/post/";
-const BACKEND_V2_GET = "https://json.excalidraw.com/api/v2/";
-
-export const SOCKET_SERVER = "https://excalidraw-socket.herokuapp.com";
+// get backend host from env. default excalidraw
+const HOST = process.env.REACT_APP_HOST
+ ? process.env.REACT_APP_HOST
+ : "https://json.excalidraw.com/";
+const BACKEND_GET = `${HOST}api/v1/`;
+
+const BACKEND_V2_POST = `${HOST}api/v2/post/`;
+const BACKEND_V2_GET = `${HOST}api/v2/`;
+
+export const SOCKET_SERVER = process.env.REACT_APP_SOCKET_SERVER
+ ? process.env.REACT_APP_SOCKET_SERVER
+ : "https://excalidraw-socket.herokuapp.com";
export type EncryptedData = {
data: ArrayBuffer; |
|
@kjellkvinge yup that is precisely what I did so I could rebuild it. But just wanted to agree on the approach (do we rebuild? do we not?) first before making any changes, although I suppose it doesn't harm to introduce those environment variables and an |
Ok, got your point. I was not aware of the limitations imposed by the build process and just had a look from the sysadmin perspective. Rebuilds are ugly to handle (in my opinion), but if there is no workaround :| |
|
Thank you all for your ideas. I think a good small step would be to containerize the socket server. Please feel free to make a PR for that and we can then talk about integrating with the main app Docker image. :) |
|
@kbariotis I've raised a PR that makes it simpler to |
|
From https://github.com/excalidraw/excalidraw-json/issues/76#issuecomment-652832953
|
@kbariotis I've raised excalidraw/excalidraw-room#71 although I've left publishing out of the PR for now. |
|
By the way, I wanted to ask about scalability - how many instances of |
|
Right now we have a single instance for both, but they both are really far away from max utilization. The design of both were with scalability in mind. For json, it's just uploading / serving static files. For room, it's just moving opaque binary opaque across all the people in a single room, it doesn't do anything else and doesn't keep any state outside of the list of people in the room. |
|
Amazing everyone, @NMinhNguyen thank you for containerizing Now I am wrong to believe that the client is not configurable to use a specific URL for that service right? So what we can do now is make it configurable somehow so when someone builds the |
|
@lietu as far as I know, Excalidraw has mixed saving ways where one uses json store, and the other one uses firestore which is most likely for collaboration rooms, best way would be if this repository migrated to supabase and dropped json store support, that seems like ideal solution. |
|
I was looking for a fully self hosted excalidraw solution.. many were still using Firebase for the storage. |
|
Seeing also the need for self-hosting, I did start a self-containing Golang implementation providing collaboration and storage options (S3, SQLite, filesystem, in-memory) I simply don't see any reason for pasting together several docker images, when I can simply put a single binary somewhere and can even cross-compile it for Mac. While it works with the official excalidraw UI without patching - besides setting endpoints, I am also struggling with the firebase dependency. |
I like your attempt!
At the moment, there are only two ways to get rid of the Google Firebase/-storage usage when using a self-hosted Excalidraw version:
I would recommend to update the readme of your project, that in the end, your self-hosted version uses Googles Firestorage (in collaboration mode or when "Save to..."->"Export to Link" is used). A way to test, that Googles Firebase storage is still used, can be found here. |
|
Wow! Thanks for that information! I was unaware of that. Is dislike patching many places in the front-end code (I am not a front-end guy). As the go binary serves the static files anyway, one can patch those calls in the assets on-the-fly and provide an API stub that somehow covers the functions needed for those calls.
Here is a prototype that has both API endpoints (documents:commit, documents:batchGet) It removes most of the error messages but is merely a proof-of-concept in the current stage. I haven't fully grasped yet that |
|
I have created the following diagram to better understand the architecture and dependencies of self-hosted Excalidraw primarily from a feature perspective (saving, link sharing, collab) based on my current understanding. What we see here is the vanilla case, where all components are from github.com/excalidraw/* repos. Hope this helps others getting started. Any suggestions are welcome. 
|
|
The official docker image has not been updated in a very long while. |
|
Hi PR is welcome to update docker image |
Is there any information on how the docker image is built? I tried setting it up myself, but haven't been able to get it working. Is it the dockerfile and compose file that's already available in the repo? |
Yes the files are already available in the repo, the build is not. being published as its failing so you will need to check why the publish docker gh action is failing and fix it |
You mean in the GitHub actions? I don't really know how they work, but it seems like there is some server error when fetching the packages for the build. I'll try to fork and look into it. |
|
@AlphaCraft9658 you should be able to run those commands in local and test it out as well in |
I got it. Now I'll see what's going on. I added a workflow_dispatch trigger for the workflow, for testing. |
The docker build workflow is still failing. That's the current status: https://github.com/AlphaCraft9658/excalidraw/actions/runs/8970588981/job/24634460317 It cannot resolve index.html. |
How is it even supposed to be build? I see it's trying to use vite and cross-env. Something makes vite unable to resolve the index.html file. |
There are various open PRs to fix the docker build, non have been merged so far (for unknown reasons).
Related Issues:
|
TLDraw doesn't have an up-to-date docker image either. |
The |
There is also an index.html in excalidraw-app. And where would I set this path? There are all kinds of different configs. |
|
@AlphaCraft9658 yes you are right, it should be |
But do you know how to specify that path? I haven't worked with vite ever. The error happens during the actual build process with vite, which fails due to the "missing" index.html file. |
can you update the command to 👇🏻 and try |
That causes a loop, as it seems. But we're talking about another PR which may solve it, you know. |
|
Hi All, |
|
The concept of standalone is at odds with the Firebase requirement, IMHO. |
|
Honestly its easier to make an app that wraps excalidraw package and selfhost modified excalidraw json for own custom storage. |
|
Nextcloud Whiteboard is based on Excalidraw and should be relatively easy to self-host. |
and others
So I'm opening the self-hosting thread.
Current architecture
First an architecture overview. Excalidraw is made of three main parts at the moment.
excalidraw.comwhich is this reposocket.ioAt the moment, we are planning on publishing a Docker image of the Excalidraw client only. That means that people will be able to self-host the client but they won't be able to use sharing/collaboration features because the client will still have to talk to our cloud-hosted backend servers on which there are CORS restrictions.
For now
For the image that we are planning on publishing soon (in the next few days), we have already stripped out the google tag manager but I see the visible sharing/collaboration icons as an issue. People will be frustrated by not understanding why they don't work.
I suggest we strip them out too on the Docker image only. Temporarily until we have a plan for the rest two services. It's the best UX we can offer IMO.
For the future
Moving forward we would like people to be able to self-host a full-fledged Excalidraw environment. My suggestion is to publish a Docker image for each of the two backend servers so people can self-host these as well. Having separate images promotes better scalability and also follows the one service/per container practice. They will be easier to maintain and also people can opt-out of a feature if they don't want to support, e.g. they only want collaboration features but not the storing feature.
Then we will be able to point the client image to specific URLs and it will simply work.
For that to work, the socket server will be much simpler as it has no dependencies. On the other hand, the storing server has a DB dependency which we will have to consider. Ideally, it should be something easy to use and maintain, like a postgres or a redis instance.
We could consider how the future of that service looks like, but for now, we can focus on what it actually needs.
Let me know your thoughts and we can start planning right away.
I should also mention the embedded Excalidraw instance we are working on that would also benefit from being able to connect to a self-host backend server but also have a better UX without icons that don't work.
Todo
The text was updated successfully, but these errors were encountered: