Synchronising Symlink-Files (just the reference, not the content of the folder)

[Diegox]

I read plenty of comments on synchronizing symlink contents, yet there is also an issue on failing to synchronize just the references. This is exactly what I need: to have my symlink-reference-files synchronized, so I can reconstruct a one to one structure/content on a second computer. I understand that not all people want that, so it should be an option to be set for each base directory.

Well, maybe it is just a question of configuring owncloud accordingly. In that case: does somebody know how?

---

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[oktayacikalin]

+1 for this

[ghost]

👍

Owncloud: Your data, your way ... web services under your control ...

To my opinion, this includes symlinks too ...

Now i have to stick to version 7 ... version 8 undermines my data structure ....

:-(

[Diegox]

I agree: The issue with symlinks is crucial for synchronisation, and cannot be talked away. What kind of synchronisarion is this, if I have to manually reconstruct a part of the data, like the reference files (= symlinks). Weather to synchronize or not these files should be a decision of the user, he knows best what is useful.

[crysman]

+1 for implementing symlinks being synchronized. Missing this!

Currently (client version 1.7.0~beta1+really1.6.4+dfsg-1) states: Symbolic links are not supported in syncing.

[dragotin]

@crysman oh please can you update asap?

[danimo]

@dragotin Btw: that's what's in Debian. I guess many people are not aware that newer versions are important :/

[dragotin]

why does debian package & ship beta version?

Maybe we should also on linux display an update notification like "A new version is available, you can install it as described here" pointing to a proper explaining we page.

[danimo]

@dragotin We do, but debian patches that out, as it has a "no-phone-home" policy, which includes vendor update notifications.

[crysman]

This version is what has come with Xubuntu 15.04 (3.19.0-16-generic #16-Ubuntu SMP Thu Apr 30 16:09:58 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux)

I have no unofficial PPAs enabled so far...

[crysman]

PS: are you telling me "hidden in the lines" that if I installed newer (newest) version, symlinks would work? If so, then I will be very motivated to update!

[LukasReschke]

@dragotin We do, but debian patches that out, as it has a "no-phone-home" policy, which includes vendor update notifications.

At least with 8.1.0 we can start shaming distributions as we can block old clients 🙈

[ghost]

PS: are you telling me "hidden in the lines" that if I installed newer (newest) version, symlinks would work?

Nope, newer versions are also not syncing symlinks.

[brevilo]

+1 for this. Without it a sync just isn't complete...

[RunasSudo]

+1

On the discussion at #1887, it was noted that WebDAV does not support symlinks, but RFC 4437 defines an extension that allows for ‘redirect-y’ WebDAV resources.

[brevilo]

OmniGroup's OmniPresence syncs symlinks as they are (no dereferencing), via WebDAV. Maybe their partially open OSX frameworks can shed some light on how they do it. IMHO, ownCloud is just incomplete without this...

[ksuquix]

+1

[malixsys]

+1

[LPTT]

+1

[drgrumpy]

+1

[dragotin]

We understood that this is a totally awesomely important feature. Before adding more +1 can somebody actually do something useful and sniff the WebDAV traffic of the OmniPresence tool and paste it here? I am interested to see how they tell the WebDAV server that a files type is SymLink rather than file or dir. Thanks.

[RunasSudo]

@dragotin I don't know how OmniPresence does it, but as I mentioned above, RFC 4437 defines a method for creating and serving symlink-like resources.

[TEsders]

+1

[guruz]

As per the discussion in #5019 we should maybe make it a setting and try it with Windows-only first (junction points)

[doits]

(originally on #665, but this place seems to be better)

I'm using owncloud to store a lot of my files and also program data. Some of those programs use (relative) symlinks, so I'd be happy if those were synced, too.

I'm sorry if it has been brought up like the following already, but why not introduce symlinks - having in mind that only "power users" need them and "normal users" (99 %) will never have any symbolic links - like this:

• general:
  • store information on server that this file is a symlink with it's target (e.g. relative or absolute path)
• sync clients:
  • detect symlinks, do not dereference them for syncing
  • when syncing to a file system that does not support symlinks, simply do not sync them (ignore them, file is missing)
    • when a new file is created with the same name like an existing symlink on server, display a sync error telling "file exists on server as a symlink" and do not sync that new file
  • when syncing to a file system that does support symlinks, sync them and restore them exactly like they were (relative or absolute)
• in webui:
  • display a link with a different icon and it's target (relative or absolute)
  • that's it, you cannot click on it or so, it is just displayed, never dereferenced
• webdav: ❓
  • is there any spec how symbolic links should be handled in webdav?

By not dereferencing them it should be easier to implement this (but still hard, I know). Dereferencing is done on file system level by OS using those files.

Of course with that solution you cannot simply drop a symlink into your owncloud folder and it will sync the linked object (only the symlink itself), but in my opinion the sync client should always sync exactly what I drop into the folder and not manipulate it in any way (e.g. on one device it is a symlink and on the other a regular file/folder which was dereferenced).

Was this way thought of already?

[fthommen]

We use symlinks within the synchronized directory tree to avoid data multiplication and data inconsistencies. Having the links synced is crititical for us.

[callegar]

@doits

I think that this can be made even simpler.

1. invent a magic for identifying files representing symlinks. For instance: #@# Symlink

2. Assure that when the sync program sees a symlink on the client transforms it into a file with this magic and sends that to the server via webdav. For instance. Suppose that on the client there is a symbolic link foo pointing at bar/foo. The sync software should see it and assure that the server gets a file named foo with the following content

   #@# Symlink
   bar/foo
   

3. Assure that when the sync client, on platform that support symlinks, checks for the symlink magic at the head of all the files it syncs. When it detects on the server a file with the sync magic, it converts it automatically into the corresponding symbolic link

This assures

• Compatibility with webdav
• That symbolic links are supported without ever dereferencing them
• That symbolic links are managed correctly even when the sync includes hosts with platforms that do not support them.

Additionally, the web client could be modified to allow symbolic link files to work as http links for links to files that are inside the hierarchy on the server.

[doits]

@callegar The only difference I see is that there is a special file with that magic syntax synced to clients that do not support symlinks? I think that might be a good way to do it, so then the file exists for clients that don't support it, too, but of course can be changed by them, too (by editing the text inside that file) - which is a corner case.

So in general files would have to be transformed on the fly - is there already such an infrastructure in the clients? Eg. some transformations to run to the file content before uploading and after downloading a file? If it was, I think this would be easy to implement client wise, otherwise such an infrastructure must be created first (in every syncing client), which might not be an easy step for every client.

[callegar]

One of the ideas behind the proposal is that if you sync with clients that do not support the symlink transformation, almost nothing breaks. So, in case the proposal is accepted, you can start implementing the support client by client (e.g. first linux, then macos, etc.). Furthermore, if different client versions are used against the same server (some supporting the transformation and some not supporting it), nothing serious should happen.

Even in the worst possible scenario where a symlink gets replaced on a client by a file with the symlink magic, a two line script should be able to fix everything quite rapidly.

[9662]

I'm sorry if it has been brought up like the following already, but why not introduce symlinks - having in mind that only "power users" need them and "normal users" (99 %) will never have any symbolic links

What are your sources for the above assertion regarding “power” vs “normal” users and their respective symlink usage? Symlinks are often created behind the scenes by software, so a user does not even need to be aware of the existence of symlinks to be using them.

Another problem with the approach described above is that it introduces a security hole. Symlinks should be copied only when the target is also part of the tree being synchronised, otherwise it is relatively trivial for someone with access to your cloud account (perhaps even via a public shared directory with write access) to add a symlink to /etc, ~/.bash_history or whatever and sniff out sensitive information out of your local machine.

With that said, I'm 100% in favour of introducing symlinks (and tags) sync support.

[callegar]

Please, no... do not start this in this way again.

There is no single "introduce symlinks" request. There are 2 different kinds of "introduce symlink" requests which are mutually exclusive (i.e., even if both models are implemented, they cannot be used simultaneously).

• One is "with symlinks dereferenced". This allows folders outside the folders marked for synchronization to be synced too. It is insecure and also unreliable (because the client cannot be properly notified of changes in a directory it does not directly watch). Yet, there are many users wanting this, because they have learned to use this approach with dropbox (even if dropbox itself deprecates it loudly, though). I think it makes little sense to introduce something like that because the owncloud/nextcloud sync clients can now deal with multiple sync folders. However, others may have other sensitivities.

• The other way is "with symlinks stored as they are, aka never dereferenced". This introduces no fundamental security hole. Symlinks are not copied. They are stored as they are. This is what the current enhancement request is about.

Before claiming security holes, please note what this enhancement request is about. I am almost sure that there is another enhancement request where to discuss the "symlinks dereferenced" model and its security implications.

[9662]

@callegar I understood what is being discussed the first time, thank you. I also understood that the intention is to synchronise the symlinks not to follow them. That sets the stage for a number of attacks. If you implement symlink sync without control sphere restrictions I would be happy to demonstrate.

[callegar]

@9662, yes, I would be glad if you could clarify what you intend with "attacks" when symlinks are stored and not dereferenced, since symlinks, when treated in this way, are basically rather similar to data. In this case, I would say that the symlink synchronization rather than constituting a security hole itself, would rather help expose security holes already existing on the system or with the practices used on the system (e.g. users working with unnecessarily high privilege or running suid executables, privileged programs mis-dealing with symlinks, etc.) that would probably be quite exploitable by other means (as simple as convincing a user to untar a tar.gz file, etc.).

[AntoniMarcinek]

Is there any progress on this? I can't believe that such an important feature is missing completely already for 6 years.

What is missing to implement this? From the earlier comments I got an impression that the problem is in deploying this in all contexts simultaneously. But can't it be done step by step and with switches in the sync client?

For example on Linux allow to sync the symlink as is with a second option being ignore (default). Then on Windows or in the WebUI the file would be useless, but it is still better than unconditionally not present. I just checked that the windows .lnk files are synced. I just cannot use them on Linux, but I find no problem with this in my use case.

Then you could go with further options like

• reinterpreting to allow symlink functionality across systems as in Synchronising Symlink-Files (just the reference, not the content of the folder) #1440 (comment)
• dereferencing as in Dereference symlinks #1887

Right now it looks like you cannot make up your mind how to satisfy everybody, so you decide to satisfy nobody. Or am I wrong or unfair?

[lenalebt]

I just learned that nextcloud does not sync symlinks, and additionally does not warn about them not being synced. This explains a few "data losses" I had in the past where I just thought I did mess something up, where it was the software.
So, not syncing symlinks now means: I need ANOTHER sync solution IN ADDITION to nextcloud/owncloud -.-