WebDAV: Don't allow two-way file sync

[daniel-callejas-sevilla]

What does not work as expected?

When setting up a remote sync WebDAV server (Settings → Sync) with both "Download remote files" enabled and also "Upload local files" enabled, Photoprism will first download remote files as expected, but then will upload those same files back to the WebDAV server, creating duplicates.

Additionally, there is an issue with the upload path: The sync server is configured to download files only from WebDAV subfolder A/… The spurious uploads then happen on path A/A/… (duplicated A). Example: Photoprism downloads file A/some/sub/folder/picture.jpg and then reuploads it as A/A/some/sub/folder/picture.jpg.

How can we reproduce it?

Set up as described above.

What behavior do you expect?

Only files which were not downloaded from WebDAV sync server should be uploaded to a WebDAV sync server.

Can you provide us with example files for testing or screenshots?
N/A

What version you are using?

Running PhotoPrism® 211010-83b4f783-Linux-x86_64

Any other helpful information?

Due to #1781 I had to remove the sync target several times during the download phase. Maybe that has had an influence afterwards?

[daniel-callejas-sevilla]

In my setup, the WebDAV server is a Nextcloud instance where I intend to keep the master copy of all my pictures, perform backups, etc… All my "picture sources" (mobile phones, dslr cameras, etc…) are already configured to push their pics to Nextcloud, and I expected to use the "Upload local files" option of the Photoprism sync target to handle Photoprism as an additional "picture source" for whatever gets directly uploaded/imported into Photoprism.

[lastzero]

While we've tested two-way sync, it's a hard problem in software engineering, and there undoubtedly are cases when duplicates are created. Also, files cannot be deleted or moved remotely, which guarantees duplicates in the long run anyway. The exact result may additionally depend on when you enabled the options and if you had connection issues in between (which seems to be the case).

The proper solution to your problem seems mounting your Nextcloud files with davfs which is included in our Docker image:

https://manpages.ubuntu.com/manpages/impish/man8/mount.davfs.8.html

Consistency, availability, and partition tolerance are other hard problems in computer science we can't simply solve as a reaction to your issues (wish we could, but no... there's tradeoffs everywhere):

https://en.wikipedia.org/wiki/CAP_theorem

[lastzero]

As a general rule, you should enable "preserve filenames" when using two-way sync:

[daniel-callejas-sevilla]

Preserve filenames was enabled when this bug happened.

I will consider the davfs recommendation, it may help in my specific scenario.

[lastzero]

How can it create duplicates if names are the same local and remote?

[daniel-callejas-sevilla]

See the note "Additionally, there is an issue with the upload path" in my initial entry.

[lastzero]

I see, so it's not a duplication issue, but the path simply isn't configured / resolved correctly?

[daniel-callejas-sevilla]

I see, so it's not a duplication issue, but the path simply isn't configured / resolved correctly?

The main issue is that the upload should have never happened in the first place. The secondary issue is that the upload, instead of overwriting every file with a (hopefully) identical copy of itself, resulted in copies of the files being created at a separate location.

[lastzero]

Obviously it worked as designed when we developed & tested WebDAV sync with Nextcloud. Edit: Maybe just a slash in the remote path name is missing and the URL parser doesn't fix it automatically?

Since Nextcloud is based on PHP, even one-way sync isn't really great in terms of scalability and performance. Compared to other options, it's particularly bad for cloning your entire library.

I'm still not sure if something broke, if it's a more complex timing issue, or if the feature just needs to be better documented. To fully test it, we need hours, which we don't currently have.

That being said, two-way sync is generally not a good idea, so it may be best not to offer it at all. If you need it, there are more advanced tools where maintainers have done nothing but catch all edge cases and optimize performance over the last few years.