Syncthing ignores files that have been deleted on other devices

[ncfavier]

What happened?

The syncthing-android app running on my phone reports that a folder is in sync, but the folder still contains files that were deleted from other devices.

The folder is on an exFAT filesystem on an SD card. It is set to receive-only mode on the phone, and send-receive on other devices. The "ignore permissions" flag is enabled on the phone. There are no ignore patterns.

On the web UI, the global state reports 3050 files while the local state reports 3062 files; the app UI just says "3050/3050", so it is inconsistent with the web UI.

The extra 12 files present on the phone are files I've deleted from other devices a while ago. They're regular MP3 files and I can play them fine on the phone. I can't see anything obvious that would set them apart from the other files.

The expected behaviour is to either delete the files on the phone, or at least report that the folder is out of sync and allow me to override the local changes. Instead there is no indication that anything is out of sync, and the web and app UIs report different file counts.

I've tried rescanning the folder and restarting syncthing.

Screenshots

Syncthing version

v1.27.3

Platform & operating system

Android 8.0.0

Browser version

No response

Relevant log output

No response

[tomasz1986]

Looks like a bug but it is impossible to say what could have caused it without a specific set of steps to reproduce the problem or debug logs from when the issue first occurred. You may want to open a new topic at https://forum.syncthing.net with the same information as it will likely not be possible to do anything about it here at this point.

If you just want to reset the broken folder states, you can remove the folder from Syncthing, then add it again. Syncthing should then rescan the local files and the excessive ones should be marked as "local additions" which you can revert via the GUI.

[ncfavier]

I did that; after rescanning, syncthing correctly showed the extra files as local additions, and after discarding them I am back to the exact same situation, which leads me to think that it's failing to delete the files from the filesystem.

I can repeat this process with debug logging enabled; which debug logging facilities should I enable, apart from fs?

Could this be an ownership/permission issue? Using Termux, I can see that the shared folder and every file inside it are owned by u0_a130:everybody, with permissions rwxr-x---. But exFAT apparently does not support permissions, so I'm not sure how this all works.

[tomasz1986]

I did that; after rescanning, syncthing correctly showed the extra files as local additions, and after discarding them I am back to the exact same situation, which leads me to think that it's failing to delete the files from the filesystem.

The reason why I think this is a bug is that you should never see a larger local state than the global in a Receive Only folder with the folder still marked as just "Up to Date" with no "Local Additions". If Syncthing failed to delete the files, you should be presented with a error message about it, but there is nothing like that in the screenshot.

Can you reproduce the problem in a completely new folder with just a handful of files? If yes, then I would suggest to pause all other folders first, restart Syncthing, enable db and model logging, reproduce the issue, and finally upload the logs here.

Could this be an ownership/permission issue?

If permissions were the culprit, you should still see them marked as "local additions" in the folder in question. You should still probably enable the "ignore permissions" setting in the Advanced tab of the folder anyway.

[ncfavier]

Nope, cannot reproduce in a new folder. :( In fact even with the music folder, if I delete a file from computer then it gets correctly deleted from phone, without affecting the 12 ghost files. So probably some database somewhere is screwed.

[ncfavier]

I re-added the folder again with fs, db and model enabled; here are the lines of the log that mention just one of the ghost files: https://gist.github.com/ncfavier/00dc2c71afb34588e704b08458ea37da. Note in particular Deleting file ... but the file is definitely still there.

I also tried removing one of the ghost files from the phone. Syncthing detected this as a local change and prompted me to discard it, which I did, after which the file was gone for good, so there are only 11 ghost files left.