Nextcloud sync on Linux takes long time #3004
Comments
|
The following also has some great info and problem determination: |
Hmm, ok it's that good old bug. I had a look at it a few years ago and none of the fixes I've tried worked. I think we'd need an Linux Electron or GUI expert to make sense of this weird bug. |
|
@bedwardly-down, have you ever heard of this kind of bug in Linux? And do you know of any workaround? Essentially, in some cases, sync is stuck on Linux till something updates the UI (for example, by switching notes, open menus, etc.). The fixes I've tried:
Also I've never managed to replicate this bug so it seems to only happen in certain environments. |
|
@laurent22, so far resetting the UI by enabling and disabling Webclipper in settings every time the client freezes during sync on 5.5 is the only thing that’s been found to work so far. And the kernel patch that’s supposed to fix the issue has no effect on Electron so far. The issue is just as bad if not worse on 5.6. @ebayer, can you try to downgrade to kernel 5.4 to see what the results are? |
|
Also, i just moved my Nextcloud instance from an Ubuntu Server to an Archlinux one. Arch is now on 5.6 and I’ve noticed that since the move, the Linux client and DAV stuff in general has substantially slowed down. My main system is still on 5.4, but i wonder if the issue here could be the kernel and OS of the server the Nextcloud instance is running on. I also know that the kernel commit that messed all of this up deals exclusively with asynchronous io calls, so i wonder if it can cause issues server side. |
|
I also encountered this bug. I must change on my computers the sync settings to local folder connected to my nextcloud server. |
|
Hi! I'm using Joplin on Arch Linux (two devices) and Android (one device). This bug does not happen only on Nextcloud sync and I'm not sure it is related with the synchronization. I was using Nextcloud sync since December 2018, but today I tried to change it to file system (using Syncthing) hoping it would be better, since in the last few weeks the desktop has been unusable: for example copying & pasting something would hang the UI (I could continue writing the note but the preview pane would not update and I could not even click other note, as nothing would happen). After changing to file system sync it kept hanging (after synchronizing some 4 items of 2029 total items); tried to rename the How can I help debugging this further? Thanks! |
|
@afv , are you able to install and boot into the lts-linux kernel and have the same issues on both Arch devices? |
|
Using file system sync.
Edit: |
My nextcloud instance is running in Centos7 (kernel 3.10.0) and latest nextcloud minor version (18.0.2) I have done a full reset, nothing changed. I will try to find a 5.4 kernel to try. |
Sorry! I was not using 5.4; on a second reboot I ended up on 5.6 again without noticing (edited my previous comment to explain that). On 5.4 everything looks fine (using file system sync). |
|
I have done more testing. First of all, this problem has nothing to do with Nextcloud sync. I have set up a copy of Nextcloud dir in my home directory (cp -a) and tried to sync. Initial sync did not start at all without the afforementioned disable/enable web clipper workaround. All sync operations hanged after initial sync is finished, and I could not find any way to cause a trigger to update the UI and finish syncing. The UI rendering completely stopped, ie. when switching between notes the note list would show the new note highlighted but the editor and display area does not get updated. Menus are working, I can check for application updates, but editor and note display stops working until sync finishes (can happen only with enable/disable webclipper). When I check the sync folder access with inotify at this moment, I can see that no process is trying to access it. Here is the inotifywait output when a manual sync is started and hangs: After I enable/disable the web clipper, sync finishes and UI goes back to normal: I have installed a 5.4 kernel and tried Nextcloud sync and nothing changed. Exact kernel version I used was 5.4.14-9-default. However local sync hang is fixed with 5.4 kernel. When I have some time, I will try older Joplin versions to try to pinpoint when this problem was introduced and test if this is Joplin related at all. Funny thing is, when trying to find a temporary workaround, I discovered that if I use an external editor to edit a note and then save the note inside the external editor while syncing, the sync finishes. I think the reason for this is saving the note in external editor refreshes the note content pane, which triggers UI update, which triggers sync operation to carry on like manually updating the UI. |
|
@ebayer, can you try the following to see if it solves your issue?
If everything worked as intended, Joplin should start updating and resyncing all of your data. Let that fully complete and you should be good to go as long as your Nextcloud instance is always mounted. There are two caveats here, though:
|
Does using davfs bring something new to this issue? I already have a copy of my nextcloud folder synced with nextcloud linux client and tried filesystem sync on Joplin on this folder before (more details a couple of comments up). I will be happy to test your suggestion if it will bring some new information though. |
|
It’s a bit more convenient. I found that using the terminal client for Nextcloud had to be set on a cron tab for it to stay consistently synced properly. The upside, though, if your net goes out, your folder will stay mounted locally and can still be synced with Filesystem sync. You just won’t get to update it upstream until your network comes back. I’ve also found that it’s substantially faster on lower end hardware. |
|
I am using Nextcloud Desktop Client and have it sync automaticly when a file changes on either locally or on the server. As I said before, using an external editor causes the UI to be updated at every note save and causing the UI update lets the sync operation carry on without hang. I am using this temporary glitch until the issue is fixed. |
|
That definitely is not applicable to my situation then. With 2.6.13, the UI isn’t freezing at all, so resetting isn’t viable. But, switching to Filesystem Sync with my Nextcloud instance mounted seems to have solved my weird issue on my last test on #3114 So, let’s assume your issue is not the same as my second one then since my solution does not fix your problem. I use very few GUI apps and try to go terminal where able because that’s more comfortable for me and i can quickly go about my business. |
|
I’ve slept for a little bit and had a thought: can you please share a recording of what’s happening because your bug is confusing to work through. I feel that there may be some conflicting information and your bug may not be easily reproducible in its current form. Thanks |
|
Here is the video: Note that this behaviour is different than what you discuss in #3114. IN #3114, the app gets completely frozen and does not recover without completely shutting down the application. In my case, only sync operation gets frozen and only for a couple of minutes. Also note that using davfs will not solve this problem, as filesystem target is also affected by this hang in recent kernels. Using 5.4 kernel fixes this issue on filesystem target, but not on Nextcloud target. |
|
Thanks for the video. So, 5.6.13 does not solve it for you on Filesystem sync, right? You never once stated that you tried that kernel release because the UI issue was fixed there. If you’re still having to use other methods to get out of it (like using external editor), there’s something else going on that could possibly be distro related or quite a few other things. I don't have OpenSuse but could attempt to try it in a virtual machine to see if your issue shows up for me. |
|
hmm, I did not try using 5.6.13, Opensuse Tumbleweed still uses 5.6.12 as latest. I will try to find that kernel from somewhere and try that as soon as possible and report back. |
|
I don't know how Opensuse handles the kernel building process, but once you get that down, it's like riding a bike. There's not a massive amount you'll need to know about rolling a custom one. If you know your hardware well and know what future hardware you may be wanting to get, you can even trim down your build to the bare minimum and just build it from source every time a new release is available. That's how I've been testing new releases since they sometimes took up to a week to come in. |
|
I'm running OpenSUSE Tumbleweed with 5.14 Kernel, and it is glacially slow syncing with a Nextcloud server. Updating from 5.12 to 5.14 made no difference at all. I don't really even think syncing is slow once it actually gets started. Joplin just seems to spin its wheels a long time before it starts the process. If I make a change to a note after starting sync, it "breaks it loose," and syncing is fast after that. |
|
@sentientnebula have you tried switching to Filesystem sync and using the Nextcloud app to handle the actual syncing? Just setup a local Nextcloud folder and make sure the GUI app runs at startup. Then set Joplin to sync to its location in that folder |
|
Sync is slow for me as well. I currently sync my Joplin files over to Fastmail via Webdav. To get around this issue, I started using the command line tool rclone. I set up my webdav connection, wrote a script that runs the rclone mount command (with the Here's the script - which essentially is just an rclone command: #!/bin/bash
rclone mount --daemon fastmail:joplin /home/mxjonny/joplin
Here's what my crontab looks like: I did not use I use Ubuntu 18.04. So far it works pretty well. There is some lag between editing a file on my phone and syncing it on my laptop but my test was changing a file on my phone and checking to see if it would update on my laptop immediately. I imagine this has something to do with rclone syncing/detecting changes. I never use my phone and laptop in this fashion, if I do edit Joplin on my phone probably by the time I get to Joplin on my laptop there won't be any lag. EDIT: I recommend setting the flag For some reason though, when I pointed Joplin to sync with the directory it recognized all the files as new items and how to download everything. I did not lose any data though and it didn't look like anything changed. After that my other devices had to re-download too - again, nothing changed and this wasn't a big deal since sync is only slow on Linux and nothing else. |
|
Today, I realized that there is a new version released 11 days ago (I really need to find a better way to check release announcements :) ) and this issue seems to be fixed with 1.0.216 version. I checked the release notes and maybe upgrading to Electron 8.2.5 might have fixed the problem. Thanks everyone for their time and effort, especially @bedwardly-down . I think we can close this issue. |
|
@ebayer there we ho. Thanks for your help too. |
|
Ah I definitely need to upgrade my version then, I'm on 1.0.210.
EDIT: Nevermind! I'm silly. My rclone mount wasn't set up. Looks like I need to work on my script since it should mount on startup. Thanks guys! |
Hello
Nextcloud linux client takes too long to finish sync. I have ~1000 notes and ~4000 resources and without encryption. I am using Nextcloud as sync target and Joplin Android version with same nextcloud settings takes 10 seconds to finish sync. However Linux desktop client takes more than 2 mins to sync.
Funny thing is, if I randomly change the visible note or toggle note list or sidebar a couple of times during the sync, operation finishes in 10 seconds like mobile sync.
It is as like there is an internal timer or tick waiting to trigger next request but get stuck until the UI gets another (unrelated) event.
I know this sounds wierd but I can record a video or something to better show what happens. I have already mentioned this in the forum: https://discourse.joplinapp.org/t/sync-with-nextcloud-fast-with-mobile-extremely-slow-on-linux/4017/23
This looks like #312 but I am not sure.
Environment
Joplin version: 1.0.197
Platform: Linux 5.5.7 x86_64
OS specifics: Opensuse Tumbleweed, Joplin installed with Joplin_install_and_update.sh
Steps to reproduce
Logfile
Enabled debug console and logs as described in https://joplinapp.org/debugging/ but no errors during or after the sync. I am attaching the log file containing two sync operations, first unattended and second time toggling note list during entire sync operation.
log.txt
The text was updated successfully, but these errors were encountered: