Logseq in docker - ability to open directory on server

[l-t-k]

I'm running logseq in docker on a small server in my home network. If I try to open a directory through the webbrowser on my laptop (within my home network) I am only able to select a directory that's on my laptop but I want to select a (local) directory on my server.

[sdyarnell]

I also would like to self host my data on my home network. I do not want to rely on third party data syncing external to the application.

[retiarylime]

I also facing the same problem. Does this issue has resolved?

[onlyacat]

Same here. Guess a 'VSCode' way is enough, thus we would be able to switch between the local and the remote dirs :)

[Dunrar]

I'd also really like to see this!

[Dunrar]

@andelf I would have a go at this, if this is something that is even wanted in the project. Will probably take some time though, as I have not worked with Clojure/Lisp before. Also, enabling plugins should not be a problem in the case of selfhosting. Would you ever consider merging something like this?

[JamesAtIntegratnIO]

This is the main thing in my way of using logseq. I think some of the easier solutions would include an environment variable to select storage option, such as local to the server or local to the user. If local to the server is set then another env var to set the storage location on boot of the container. This will make configuration of the container settings easy and is generally a best practice when exposing configuration outside of the container. I would also recommend the ability to set the location of config files and vault files to separate paths. This is big for self hosters as they may want separate backup and management configuration of the different types of data.

[fivestones]

Also there would need to be a way in the mobile app to set the server, so the data comes from there (the server instance running logseq) instead of locally on the phone (e.g., iCloud storage on iPhone).

[ondrejpales]

As the Logseq's own sync solution is reportedly being already tested internally, perhaps it would suffice to release this solution as a docker container as well, so that us self-hosting home-labbers are satisfied. :-)

[fivestones]

The problem is that logseq's own sync solution is going to be behind a paywall of a monthly subscription (I'm assuming) but I and many other homelabbers aren't going to sign up to pay for a monthly subscription. And if you're running Logseq from a server designed to be connected to from a browser, or the iPhone app, you already have your sync solution. Given that this probably conflicts with Logseq's business interests, I have little hope they will implement such a thing. But I'm happy to use a fork.

[ianjs]

The problem is that logseq's own sync solution is going to be behind a paywall of a monthly subscription (I'm assuming) but I and many other homelabbers aren't going to sign up to pay for a monthly subscription.

As you said, a keen homelabber probably isn't going to sign up anyway, so there's no loss in sharing the code and keeping them as enthusiastic contributors. That said, as a homelabber, I wouldn't automatically not use their hosting, depending on reliability and price.

Given that this probably conflicts with Logseq's business interests, I have little hope they will implement such a thing. But I'm happy to use a fork.

I'd be interested to hear a policy on this from the devs. Forking the project is never a good look.

There are lots of Open Source projects that provide a hosted service for (the majority of, I suspect) users, even though they support self-hosting. Not everyone has the skills or inclination to self host and they'd be doing themselves a favor opening up that code for all the usual open source reasons.

[fivestones]

and they'd be doing themselves a favor opening up that code for all the usual open source reasons.

Agreed, with that and everything else you (@ianjs) said.

[queeup]

Waiting this for self hosting my logseq.

The problem about desktop app on Linux is electron + wayland is not good along well with fractional scaling. All text blurry or no title bar. So I like to deploy this to my self with docker/podman and use it from browser. But right now File System API is not supported with many browsers (Firefox :( ). I want to use logseq on Firefox.

Also here:
https://discuss.logseq.com/t/local-on-server-storage-for-self-hosted-logseq/6613
https://discuss.logseq.com/t/allow-webapp-container-to-support-user-defined-graph-directory-environment/9884

[cnrpman]

@queeup @fivestones @ianjs
We are welcome to more deployment options. I'm happy to do code-review if you'd like to submit PRs addressing this :)
But it won't be easy, basically it's something similar to VSCode-remote / Codespaces.
To achieve this, may need to implement the fs protocol and a head-less server.

[crawfordlong]

I'm enthusiastically in support of this concept. Compared to the ease of using original logseq across devices in-browser (though hosted by logseq), the current model is not optimal. I spend more time than I care to admit resolving (self-hosted) git conflicts because of the way the application works now. I'm also in a bind because multiple devices I depend on do not permit application installation. I don't want to go with a non-markdown solution because (a) the one that is primarily browser-based is entirely proprietary, and (b) the one that isn't does not have a fully baked browser option yet; both store their data in a format other than plaintext. I'm happy to pay logseq for this functionality, btw. I just can't pay for the sync feature. And I'll cheerfully send some funds to dev(s) who work on implementing this.

To summarize: self-hosted, browser-based logseq ftw.

[ianjs]

I'm enthusiastically in support of this concept.

To summarize: self-hosted, browser-based logseq ftw.

++1

I'm not sure if GitHub is the place to be discussing the pros and cons at length, so I've posted some more thoughts here.

[S0und]

It would be great if we could set a volume to store the settings and our files externally. All you have to do is to store your files in exp. ./docker/logseq/data folder which would be the external volume for logseq.

[SystematicError]

This would be an incredibly useful addition for homeservers and such, is there currently any workaround to achieve this?

[S0und]

@SystematicError Currently i'm using Syncthing, this is the closest "selfhosted" solution. Works well on Desktop, but can be cumbersome on mobile. Syncthing is a continuous file synchronization program. So what i did, i setup one at Home, on my Pi server and one at Work. I'm using my Pi server as a repository, Home <-> Pi <-> Work, so the clients are syncing only to the server and not to each other (Syncthing can sync between all clients). The cool thing is, if you use Zerotier (i'm assuming this works with Tailscale too), Syncthing will automatically find the clients/server during the configuration, this way you don't have to open ports on your router and forward them.

[SystematicError]

@S0und thanks for your input, whilst this is definitely a practical solution, this involves running the desktop app on a client device. I'm looking into running the actual app on a server and accessing on the client via the web browser, this may seem odd but has a couple of benefits such as being able to use the app on locked down company devices, being able to update the app for all clients, etc.

[ianjs]

I'm looking into running the actual app on a server and accessing on the client via the web browser, this may seem odd but has a couple of benefits such as being able to use the app on locked down company devices, being able to update the app for all clients, etc.

In fact this solves most of the problems of "sync".

By having the option of accessing a self-hosted server, Logseq can be accessed from anywhere on anything with a browser. For those willing to do the setup it's the perfect option, especially when the security issues can be overcome just by having an incoming VPN.

[crawfordlong]

In fact this solves most of the problems of "sync".

As well as being what logseq originally was: a browser-based, use-anywhere, zero-friction note-taking solution. I am really impressed with (and grateful for) the work that has been done on the app, but there is no question my engagement has dropped in the app-based logseq world since I have to stop what I'm doing and find a device with the app installed, rather than just firing up a browser on whatever device I'm using.

[fivestones]

Back in May, @tiensonqin did some work on integrating yjs into logseq (see the commits here](master...feat/crdt-rtc) on the logseq crdt-rtc branch; also a bunch of crdt work was done earlier this year year in the file-sync-crdt branch--not sure if this code is what is now being used for sync with the not-open-sourced logseq backend).

Yjs implements a CRDT datatype, which basically is a way to have some information sync to multiple places without any conflicts, and without any user intervention needed. The way CRDTs work is explained pretty well in the CRDT wikipedia article.

Using this with logseq would allow you to have logseq run as a local-first app, with all the benefits that entails: the speed of an app with a local file system, but the ability to have your own backend, self hosted which could do as much as store your whole set of logseq data, or as little as nothing--with syncing happening by p2p between the clients you use logseq on (by means of webrtc or libp2p, both of which are included as sync providers in the Yjs project). You also would get the conflict-free nature of the way CRDTs work (@crawfordlong no more resolving git conflicts!) and it could run in both an app for your phone/computer, or in a browser. In both cases your data would be local but synced with CRDTs either with a server or by p2p (or both) to your other devices. Another benefit of CRDTs is that using logseq this way would mean that when your internet connection is down or you're on a plane or in a subway without internet, you would still be able to use logseq, and as soon as the internet connection is restored, your changes sync without any conflicts. You could even have two different people on different devices both editing the same file at the same time while using the offline mode and not connected to the internet, and when the internet is restored for them, all of both of their changes would sync to each other and no data would be lost.

I don't know that logseq's financial model is currently especially compatible with all of this; but I think for someone who knows the code it wouldn't take a ton of work to integrate yjs. And looking at the file-sync-crdt branch it seems like much of this work has already been done, and if so I think it would mean that I think all the things that everyone in this thread is wanting would be realized--I just would love to see the backend open sourced (as the logseq readme.md used to say would happen as soon as it was up to standard security wise--something that has since disappeared from the readme.md). With the backend code, and the front end running in either an app or a browser using local storage and syncing with CRDTs to a self-hosted server, I'd be one happy camper.

I've contributed financially in the past to logseq, and will almost certainly contribute more in the future. But I'd love to see this stuff happen.

What do you all think?

[zephyros-dev]

Personally, I'm more interested in a centralized, on-server solution. This is due to the fact that I embed attachment to logseq (pdf, pictures, etc...), and syncing them would be a waste of storage on local device and bandwidth.
Of course the centralized solution would not be compatible with offline usage, so syncing would works better for this usecase.

[fivestones]

and syncing them would be a waste of storage on local device and bandwidth.

What if these kinds of things were not synced/replicated locally by default, and while the text was synced, those kinds of attachments/assets were only synced if you marked them specifically to sync? Alternatively for people who wanted them to sync there could be a setting to set them to all sync by default. Or maybe sync only assets below some size limit by default, and leave larger ones out of the sync. Would that make it work in your use case?

[ianjs]

What if these kinds of things were not synced/replicated locally by default, and while the text was synced, those kinds of attachments/assets were only synced if you marked them specifically

This seems to be addressing a very niche case where a user has unusually large attachments. There’s no reason why it couldn’t be an option but the default should be to have a complete sync.

The remote access to an instance with an external data store discussed above would be a better solution to this use case though.

[ianjs]

Yjs implements a CRDT datatype, which basically is a way to have some information sync to multiple places without any conflicts, and without any user intervention needed.

What do you all think?

This sounds ideal. What we’re aiming for is seamless access from any device without distractions about whether the syncing might be screwing up the data.

A single central instance, remotely accessible is one solution, but this seems even more useful: multiple synced instances so the data is replicated and editable offline.

Yes please.

[crawfordlong]

I don’t want to lose track of the concern that many use cases do not allow for local installation of software. A browser-based solution (whether involving a local data store synced in this way, if that is possible) or centralized, unsynced data would both work. Local app installation would not. I cannot, for example, access (legitimate) software distribution websites or plug in a USB drive containing an installer or portable app, even though I have no restriction against visiting a website hosting a note-taking tool.

…

On Nov 29, 2022 at 11:47 PM, <Ian Slinger ***@***.***)> wrote: > > > Yjs implements a CRDT datatype, which basically is a way to have some information sync to multiple places without any conflicts, and without any user intervention needed. > > > > What do you all think? > > This sounds ideal. What we’re aiming for is seamless access from any device without distractions about whether the syncing might be screwing up the data. A single central instance, remotely accessible is one solution, but this seems even more useful: multiple synced instances so the data is replicated and editable offline. Yes please. — Reply to this email directly, view it on GitHub (#4107 (comment)), or unsubscribe (https://github.com/notifications/unsubscribe-auth/ATWGBHLUW3YYULC7T7L2JXTWK3L5RANCNFSM5NKO26AQ). You are receiving this because you were mentioned.Message ID: ***@***.***>

[ianjs]

I don’t want to lose track of the concern that many use cases do not allow for local installation of software.

I completely agree, in fact that's exactly why I suggest having a central server option is almost essential.

At the moment, if I understand correctly, logseq instances are distinct installations with various clunky and error-prone methods for syncing between them. Ideally a note-taking app should be accessible on the nearest device, for example your phone or tablet, rather than waiting till you get back to, say, your desktop. Otherwise you risk losing that momentary insight or noting down something to remember.

If syncing was transparent, fast and error-free then we'd be done, but none of the solutions I've seen so fast are any of these. A central server option is one way of bypassing the whole sync issue, but if the CRDT option can make that work for multiple installed instances then bring it on.

[fivestones]

@crawfordlong the great thing about a CRDT syncing solution is that it can work both in the browser on a website you load or in an app. And once you've created it for the browser it's easy to port that to the app, or vice versa.