Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WebDAV: Don't allow two-way file sync #1785

Closed
daniel-callejas-sevilla opened this issue Dec 1, 2021 · 10 comments
Closed

WebDAV: Don't allow two-way file sync #1785

daniel-callejas-sevilla opened this issue Dec 1, 2021 · 10 comments
Assignees
Labels
enhancement Refactoring, improvement or maintenance task released Available in the stable release ux Impacts User Experience

Comments

@daniel-callejas-sevilla
Copy link

What does not work as expected?

When setting up a remote sync WebDAV server (Settings → Sync) with both "Download remote files" enabled and also "Upload local files" enabled, Photoprism will first download remote files as expected, but then will upload those same files back to the WebDAV server, creating duplicates.

Additionally, there is an issue with the upload path: The sync server is configured to download files only from WebDAV subfolder A/… The spurious uploads then happen on path A/A/… (duplicated A). Example: Photoprism downloads file A/some/sub/folder/picture.jpg and then reuploads it as A/A/some/sub/folder/picture.jpg.

How can we reproduce it?

Set up as described above.

What behavior do you expect?

Only files which were not downloaded from WebDAV sync server should be uploaded to a WebDAV sync server.

Can you provide us with example files for testing or screenshots?
N/A

What version you are using?

Running PhotoPrism® 211010-83b4f783-Linux-x86_64

Any other helpful information?

Due to #1781 I had to remove the sync target several times during the download phase. Maybe that has had an influence afterwards?

@daniel-callejas-sevilla daniel-callejas-sevilla added the bug Something isn't working label Dec 1, 2021
@daniel-callejas-sevilla
Copy link
Author

In my setup, the WebDAV server is a Nextcloud instance where I intend to keep the master copy of all my pictures, perform backups, etc… All my "picture sources" (mobile phones, dslr cameras, etc…) are already configured to push their pics to Nextcloud, and I expected to use the "Upload local files" option of the Photoprism sync target to handle Photoprism as an additional "picture source" for whatever gets directly uploaded/imported into Photoprism.

@lastzero
Copy link
Member

lastzero commented Dec 1, 2021

While we've tested two-way sync, it's a hard problem in software engineering, and there undoubtedly are cases when duplicates are created. Also, files cannot be deleted or moved remotely, which guarantees duplicates in the long run anyway. The exact result may additionally depend on when you enabled the options and if you had connection issues in between (which seems to be the case).

The proper solution to your problem seems mounting your Nextcloud files with davfs which is included in our Docker image:

https://manpages.ubuntu.com/manpages/impish/man8/mount.davfs.8.html

Consistency, availability, and partition tolerance are other hard problems in computer science we can't simply solve as a reaction to your issues (wish we could, but no... there's tradeoffs everywhere):

https://en.wikipedia.org/wiki/CAP_theorem

@lastzero lastzero added the declined Cannot be merged / implemented at this time label Dec 1, 2021
@lastzero lastzero closed this as completed Dec 1, 2021
@lastzero lastzero changed the title Bug: WebDAV sync uploads copies of whatever it downloads WebDAV: Avoid creating file duplicates when enabling two-way sync Dec 1, 2021
@lastzero
Copy link
Member

lastzero commented Dec 1, 2021

As a general rule, you should enable "preserve filenames" when using two-way sync:

preserve

@daniel-callejas-sevilla
Copy link
Author

Preserve filenames was enabled when this bug happened.

I will consider the davfs recommendation, it may help in my specific scenario.

@lastzero
Copy link
Member

lastzero commented Dec 1, 2021

How can it create duplicates if names are the same local and remote?

@daniel-callejas-sevilla
Copy link
Author

See the note "Additionally, there is an issue with the upload path" in my initial entry.

@lastzero
Copy link
Member

lastzero commented Dec 1, 2021

I see, so it's not a duplication issue, but the path simply isn't configured / resolved correctly?

@daniel-callejas-sevilla
Copy link
Author

I see, so it's not a duplication issue, but the path simply isn't configured / resolved correctly?

The main issue is that the upload should have never happened in the first place. The secondary issue is that the upload, instead of overwriting every file with a (hopefully) identical copy of itself, resulted in copies of the files being created at a separate location.

@daniel-callejas-sevilla daniel-callejas-sevilla changed the title WebDAV: Avoid creating file duplicates when enabling two-way sync WebDAV: Avoid uploading downloaded files when enabling two-way sync Dec 1, 2021
@lastzero
Copy link
Member

lastzero commented Dec 1, 2021

Obviously it worked as designed when we developed & tested WebDAV sync with Nextcloud. Edit: Maybe just a slash in the remote path name is missing and the URL parser doesn't fix it automatically?

Since Nextcloud is based on PHP, even one-way sync isn't really great in terms of scalability and performance. Compared to other options, it's particularly bad for cloning your entire library.

I'm still not sure if something broke, if it's a more complex timing issue, or if the feature just needs to be better documented. To fully test it, we need hours, which we don't currently have.

That being said, two-way sync is generally not a good idea, so it may be best not to offer it at all. If you need it, there are more advanced tools where maintainers have done nothing but catch all edge cases and optimize performance over the last few years.

@lastzero lastzero changed the title WebDAV: Avoid uploading downloaded files when enabling two-way sync WebDAV: Don't allow two-way file sync Dec 1, 2021
@lastzero lastzero reopened this Dec 1, 2021
@lastzero lastzero added enhancement Refactoring, improvement or maintenance task ux Impacts User Experience and removed bug Something isn't working declined Cannot be merged / implemented at this time labels Dec 1, 2021
@lastzero lastzero self-assigned this Mar 27, 2022
@lastzero lastzero added the in-progress Somebody is working on this label Mar 27, 2022
@lastzero lastzero added please-test Ready for acceptance test and removed in-progress Somebody is working on this labels Mar 27, 2022
@lastzero
Copy link
Member

The checkboxes have been replaced with radio buttons so that only upload OR download can be selected at the same time:

remote-sync-radio-buttons

I will start a new development preview build later today so it can be tested. Any help is greatly appreciated!

@graciousgrey graciousgrey added released Available in the stable release and removed please-test Ready for acceptance test labels May 17, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Refactoring, improvement or maintenance task released Available in the stable release ux Impacts User Experience
Projects
Status: Release 🌈
Development

No branches or pull requests

3 participants