Selfhosting suggestions/ideas

[Dima-Kal]

Hi, thanks for this awesome tool, I’ve used it with docker to try to selfhost it but can’t currently find it useful (no plugins and relies on local client’s storage)

What would be amazing in my opinion is:

1. Selfhost the container and provide it with a persistent storage variable for notes folder (on the server itself or remote location, doesn’t matter)
2. When opening the web app from any client it will already be on the same notes folder
3. Support plugins on that same instance of web app, so any client will have the same experience

[Dunrar]

I think this would be paramount for the self-hosted solution. Currently, it feels more like an afterthought or a second-class citizen without it's own featureset.

I guess the official web version deployed on their servers could not (yet?) allow for the use of plugins, for a variety of reasons. This is even more true for on-server storage. But a self-hosted version does not have these limitations.

If self-hosting is really to be an option going forward, it needs to be it's own thing. And I don't think that it would be that hard as plugins are already possible on the native versions and one could basically just get rid of any syncing solutions, at least if Logseq is used on the web exclusively.

[bdklahn]

I would like this, as well. I was thinking about an integrated knowledge graph for my work group. It seems like i could be a simple setting, to use server-side storage, vs. local. But, then again, maybe not so much because. . . .

• It seems the server files are just there to serve static files for full client side javascript running. So it might be tricky to enable (smooth) authentication for sever side writing.
• They might be keeping it a little "closed", with the view to offer their own paid hosting as their business model.

But, I guess, it could be a third option: If they want to continue to promote this as "privacy first", it would make it tough to claim that in cases where someone tries to sell their own "self hosting" to others. In that case, it would be like Roam, et. al.: The self host-er would have access to others' data.

One thing I have done is to put my files inside MEGAsync-tracked directories (MEGAsync is end-to-end encrypted), then let those sync across computers where I had Logseq running. But I've run across issues where the metadata got corrupted when two machines were writing at the same time.

So maybe a solution is to have it deal better with concurrent writes. Maybe design the metadata update to be more modular/atomic or simply use temporary lock files. But maybe that is already handled. Because I recently tested modifying the same page (in local storage), in two separate browser tabs. I did get an alert which asked me which version of a conflicting change I wanted to keep. I haven't recently tested writes from separate computers, via MEGAsync'd directories, lately.

[Dima-Kal]

I would like this, as well. I was thinking about an integrated knowledge graph for my work group. It seems like i could be a simple setting, to use server-side storage, vs. local. But, then again, maybe not so much because. . . .

• It seems the server files are just there to serve static files for full client side javascript running. So it might be tricky to enable (smooth) authentication for sever side writing.
• They might be keeping it a little "closed", with the view to offer their own paid hosting as their business model.

But, I guess, it could be a third option: If they want to continue to promote this as "privacy first", it would make it tough to claim that in cases where someone tries to sell their own "self hosting" to others. In that case, it would be like Roam, et. al.: The self host-er would have access to others' data.

One thing I have done is to put my files inside MEGAsync-tracked directories (MEGAsync is end-to-end encrypted), then let those sync across computers where I had Logseq running. But I've run across issues where the metadata got corrupted when two machines were writing at the same time.

So maybe a solution is to have it deal better with concurrent writes. Maybe design the metadata update to be more modular/atomic or simply use temporary lock files. But maybe that is already handled. Because I recently tested modifying the same page (in local storage), in two separate browser tabs. I did get an alert which asked me which version of a conflicting change I wanted to keep. I haven't recently tested writes from separate computers, via MEGAsync'd directories, lately.

Currently the solution i went with is using VS Code server running purely with markdown extensions where the workspace is notes vault, so when i have logseq desktop client i use it, and if i need to quickly edit something and i dont have logseq desktop client i go to the VS Code server

[bdklahn]

So when you say "logseq desktop client", are you referring to the standalone packaged Electron app? Or do they now have a client-server model? Those desktop packages are basically little web browsers running the Javascript app inside them.

I've looked at what the Docker file does to deploy. It basically just creates static Javascript files, then copies them into a web server root directory, then starts an nginx webserver.
I started that Docker container, with a volume mounted to outside, then simply copied over the files under the "static" directory. I uploaded that to a host running nginx to serve static files out of a directory. I was able to then go to that url subdir and "run logseq". From there, I wondered if it could actually just be set to store the graph(s) on the webserver. But because that is actually a "static" site, I wasn't sure if it would be a challenge to write back to the server.

[Dima-Kal]

So when you say "logseq desktop client", are you referring to the standalone packaged Electron app? Or do they now have a client-server model? Those desktop packages are basically little web browsers running the Javascript app inside them.

I've looked at what the Docker file does to deploy. It basically just creates static Javascript files, then copies them into a web server root directory, then starts an nginx webserver. I started that Docker container, with a volume mounted to outside, then simply copied over the files under the "static" directory. I uploaded that to a host running nginx to serve static files out of a directory. I was able to then go to that url subdir and "run logseq". From there, I wondered if it could actually just be set to store the graph(s) on the webserver.

I dont think that their desktop client is the same as their web client, because the web client is unable to take plugins -> and thus rather useless

[bdklahn]

Hmmm. It could just be that . .

• The assets packaged into the DMG, AppImage, etc. might be different than what is put into the static dir.
• Those local packages allow for easier storage access for plugins.

I'm not sure what you mean by "web client". In that case the "client" is your web browser.

I don't know absolutely for sure. But I read through their deployment/design doc. They basically compile/package clojure (LISP-like which runs in JVM) to javascript (clojurescrript). So that can be served from any static site, and run in any browser.
Then they package that as an Electron app, to make it like a regular desktop app (without having to use a different graphical toolkit, like Qt, etc., to create the same app for desktop). A lot of apps take this shortcut: VSCode, GitKraken, many messaging apps. That way they can use the same code for desktop and web. These apps are usually very bulky, and often slow, but they save a lot of developer time in "cross-compiling".

[Dima-Kal]

Hmmm. It could just be that . .

• The assets packaged into the DMG, AppImage, etc. might be different than what is put into the static dir.
• Those local packages allow for easier storage access for plugins.

I'm not sure what you mean by "web client". In that case the "client" is your web browser.

I don't know absolutely for sure. But I read through their deployment/design doc. They basically compile/package clojure (LISP-like which runs in JVM) to javascript (clojurescrript). So that can be served from any static site, and run in any browser. Then they package that as an Electron app, to make it like a regular desktop app (without having to use a different graphical toolkit, like Qt, etc., to create the same app for desktop). A lot of apps take this shortcut: VSCode, GitKraken, many messaging apps. That way they can use the same code for desktop and web. These apps are usually very bulky, and often slow, but they save a lot of developer time in "cross-compiling".

Yes I do understand that the Electron app is just a web browser engine, but it does not matter for me what it is based on since the Electron app and the self hosted web interface functions differently (Even if the underlying architecture is the same), if there is a workaround to use plugins on the self hosted web interface then great, but currently I don't think there is. The plugin support is a huge must for me and I cannot use logseq without it

[bdklahn]

Ok. I just got a little confused by "I dont think that their desktop client is the same as their web client,"

I'm thinking "The client part is trivial: they're both just browser engines which can run Javascript".

Yeah . . . sorry . . .I kinda lost track of the plugin part being a deal-breaker for original Issue you raised here.

I simply would be happy just to be able to store graphs server side.
But, in a way, it is probably the same/similar issue: centralized/server storage of user-customized data. Plugins are also user-modified/added data. So I wonder, if we get an idea of how Logseq handles access to plugin files . . . or graph files . . . or other config files, if one can see the way how to do all of this.

But, I guess, if the plugin handling is somehow very closely tied to platform packaging (AppImage, DMG, exe, etc.), maybe that would be a pain. But that would seem a poor design, actually.
Do you know where plugins are stored locally?

A typical design for, say, configuration files which can be over-ridden is to have a central file, which gets superseded if there is a local file. Logseq must do something like this with the example graph that is present with the web app: that is on the server, but gets "masked" after you link in a graph from your local filesystem. Presumably plugins (and a global default config) could be handled the same way. So maybe dig into this git repo and find find the code that handles loading the default graph (and see where/how that graph is stored). It seems similar code could handle using any server-side app-available data.

Sigh . . . but I guess that default graph is read-only. It gets loaded to browser memory, then changes go away after the session closes.

Looks like it's not even loaded from example markdown files, but instead baked into js files.

$grep -lr 'A dummy tutorial' *
js/main.js
js/publishing/main.js

[HALOCORE]

If not considering plugins, is it possible for the webApp to access remote file system over HTTPS?
Because there exists websocket-based remote file system webfuse and corresponding javascript adapter webfuse-js. If anyone knows how to polyfill the webpage to proxy the File System Access API to self-hosted webfuse before logseq webapp is loaded, then it should be able to automatically open self-hosted remote file systems instead of the local one.
Will it work?