symlink support for Linux

[makomi]

seafile server v1.7.0.1
client v1.7
found no log entries related to the issue on my system

Issue
Creating a symlink with "ln -s" creates a symlink on my local machine, but everybody I am sharing the link with gets a copy of the actual file instead of a symlink. I think this is related to issue #130.

Is following symlinks the wanted behavior, i.e. is this a feature and not a bug?

Why I need this feature
In many cases I find it difficult to structure my data sufficiently just by sorting it into folders, because e.g. it would be easier to browse my data if pictures in one folder would also be accessible in another folder.

[killing]

Yes this is the expected behavior. Syncing the symlink (including its target) is not possible since it may not be possible to create the target folder on another client.

[makomi]

Do you think, it would be a good idea to detect whether a symlink links to a file or folder inside the shared library (case 1) or to something that is not part of the library (case 2) and then, based on that destinction, either (1) create a symlink to the file or folder already present in the library or (2) follow the symlink and copy the external data into the library?

[blablup]

For me it would be enough if the symlinks would be synced without the files behind it. If the Target maschine do not have the files, the symlink would be dead (That is totaly ok for me). Otherwise I would have Problems in some situations where I have files wich are out of date for me.

Example:
/home/me/seafile_synced_dir/

• fileA
• fileB -> /mnt/some_mount_on_a_fileserver/ProjectDirA/fileB

Now if I change my Computer and sync my "seafile_synced_dir" to the new Computer there would be no link to the file-servers file. If I change "fileB" It wouldn't be changed on the file-server and If the file on a file-server changes I wouldn't recognize it.
Also I have links to dirs on servers which shouldn't be synced (In Example VM templates or ISO Files. It would be TB of Data). Or even sensitive Informations which should never be synced to any mobile device.

And at the moment I'm not shore what happens, If I change fileB on another PC (which was synced by seafile, no symlink) on my first PC. Is the symlink replaced? Would the change be written to file-server?

[makomi]

I propose the following behaviour:

Detect whether a symlink links to a file or folder inside the shared library (case 1) or to something that is not part of the library i.e. external (case 2) and then, based on that destinction, either (1) create a symlink to the file or folder already present in the library or (2) follow the symlink and copy the external data into the library respectively create a potentially broken symlink, if the user chose that option in the Seafile client settings.

[bobbaluba]

I agree with blablup. Please let the symlinks be symlinks, that's the least ambiguous solution. I have tons of projects with relative symlinks, this would cause duplication and/or infinite recursion when synced.

[blakerohde]

I too would like to see symbolic links blindly kept, regardless if the files exist or not on the other client.

[killing]

OK. Since many people are requesting this feature, we'll add an option in the next release.

[ghost]

I actually ran into this symlink problem (infinite loop)

May I advice you to :

• treat symlink as symlink
  and/or
• use the content of the exclude/ignore file and do not try to follow any link/folder that is excluded

Actually the daemon seems to check all the files/dirs including the ignored ones... (as the seafile.log tells about wt-monitor-linux.c(102))

[dschoepe]

Has there been any progress on this issue? common/fs-mgr.h mentions a metadata type for links (SEAF_METADATA_TYPE_LINK), but that doesn't seem to be used anywhere yet.

[ibeardslee]

I'm against following symlinks by default. I'll give an example of something that just blew up for me.

On my Ubuntu desktop (~/Desktop) I have links to things like Dropbox (840MB), Downloads (800MB) our internal network shares, my Music (18GB, on another partition and as a separate Seafile Library) and a link to ~/Seafile where a number of other Libraries are located. I also have just under 200MB of in-progress files and directories sitting on my Desktop that I thought would be good to have sync'd into Seafile.

I ran out of diskspace (started with 16GB free) in /home as Seafile tried to follow the links on the desktop, including the link to ~/Seafile which has the index cache.

Seafile shouldn't be following the symlinks as it syncs.

[johanwilfer]

Any updates on this?
I would also expect a symlink to remain a symlink and not to be followed.
I installed the linux clients version 4.0.6 and server 4.0.5 and they are syncing the target, not the link.

[ibeardslee]

We have our own dev looking at adding some functonality around symlinks as options in the client .. see #130

[MurzNN]

Is there any progress about symlink support in next seafile version? Will be good to have per-library option how to handle symlink files:

• copy target file (current behavior)
• copy as symlink (new feature)

[funkyfuture]

my two cents: i rather enjoy this as a feature to aggregate folders that are in differrents paths outside a library into one library. this makes it easy to work on same fs-objects in different contexts.

[ikcalB]

@funkyfuture then, a syncing problem comes up - when linking to the same file from two or more libraries: which library to sync to or from? (the same reason, why seafile does not allow creation of libraries inside other libraries)

@killing what about the symlink support now? (Pls read above)
I'd suggest to add an option (maybe even on a per library-basis) to:

• always copy files (this is the current behavior)
• copy files if the link points outside the library and preserve library-internal symlinks
• preserve symlinks in any case

[funkyfuture]

@ikcalB i don't get your argument. it will sync from and to any library that links the object in question.

[MurzNN]

Seems that Windows also have symlink support http://www.howtogeek.com/howto/16226/complete-guide-to-symbolic-links-symlinks-on-windows-or-linux/
So we can do support not only in Linux, but on Windows, Mac, Android too.

[ikcalB]

@funkyfuture I re-read your two cents and am unfortunately not able to get what you mean.

@MurzNN nice, thanks for sharing!
@freeplant As MurzNN pointed out, symlinks are indeed available on all supported platforms. People who sync from linux, should be well aware of symlinks. (Or should at least be informed by the creators of the library)

What do you think about configurable options like:

• always copy files (this is the current behavior)
• copy files if the link points outside the library and preserve library-internal symlinks
• preserve symlinks in any case

[vin-c]

Hello,
I've again made a test concerning this problem. In my (particular) environment, I use seafile to sync some folders of my home_dir, some of them are wine environments with symlinks to the root_fs (/). This does lead to infinite loops of symlinks as shown below. I need to sync the environments but not the root_fs.

Test case:

• create a library inside a test_dir folder
• add a symlink inside that point just one or two level up (the sync process will be in a loop) :
  lrwxrwxrwx test_dir/library_name/link_to_top_dir -> ../../test_dir
• add a seafile-ignore.txt excluding "link_to_top_dir/"
• sync to server

the logs are showing :
repo-mgr.c(1211): Failed to stat /(...)/test_dir/library_name/link_to_top_dir/library_name/link_to_top_dir/library_name/link_to_top_dir(...) Too many levels of symbolic links

I don't care about the "right" behaviour against symlinks (follow them or not) but please make sure that at least seafile-ignore.txt work with them !

Sure if users have options for each library or globally, it will help prevent this behaviour.

Regards

Server : 4.4.6
Client : 4.3.4

Will try with client v5

[freeplant]

What client version do you use? v5.0 should have fixed the ignore problem.

[vin-c]

@freeplant : Updated client to v5.0.0, same behaviour on client side

For the record, I cleared all configuration and .seafile-data, starting from scratch with v5.0.0 and followed the test case explained before.

The client's logs are still showing the error "Too many levels of symbolic links", but now, the server library does not contain the linked folder at all.
Before (client v4.x), the directory structure was created on the server and I could follow the many directories created (library_name/link_to_top_dir/library_name/link_to_top_dir/...)

This is better but I will not try with a root_fs link... as I don't know where the error is thrown and don't want to mess up my system now...

[zhanghai]

Please add a symlink-as-file option, just as how git handles this accross platforms.

See http://stackoverflow.com/a/954575/2420519

git just stores the contents of the link (i.e. the path of the file system object that it links to) in a 'blob' just like it would for a normal file.

[ikcalB]

@killing more than 2 years ago, you said that you were implementing a solution - still I cannot spot anything in the roadmap.

will you implement configurable symlink support, finally!?

1. decide if the target is inside -> A or outside -> B of that library
2. and for either case A or B add a seperate option to:

• recreate that symbolic link, regardless whether the target exists
• copy the contents (if possible)
• ignore that symlink
• leave some kind of explanatory file behind

(admittedly, the last 2 are not optimal, and might need rethinking)

[blubberdiblub]

I'm also waiting for non-broken symlink support for quite some time. Obviously, syncing and creating symlinks as actual symlinks (not the files/directories they point to) is the most preferable solution in the majority of cases. Windows can do symlinks, too. They would even be interchangable between operating systems if they are relative and you normalize them (regarding the path separators, for instance) before putting them into a seafile Library.

[killing]

Sorry we didn't have time to change the code in the past :-(
I did some investigation on Dropbox's behavior. It seems Dropbox follows symlinks by default. The rationale is to support syncing a folder outside of the Dropbox folder. However, since Seafile supports syncing any folder in the system, this use case is not for Seafile.
So I think it's good to at least provide an option to let the client keep the symlink. We'll still keep following symlink the default behavior, for compatibility.

[CrazyPandar]

please add keep symlink option in client settings. make it default to false for compatibility.

[deragon]

I add my voice to those who would like symlinks to be preserved. Though an global option should exist, the user should be able to override it on a per library basis. Options should be:

• Follow symlinks always (copy data) - Default
• Preserve symlinks at all cost - Symlinks or nothing; no data if platform does not support it.
• Preserve symlinks whenever possible. Follow symlinks (copy data) if platform does not support it.

In a more advance feature, if symlinks could be converted to Windows shortcuts and vice-versa, that would be great. Though, it might not be necessary... Apparently, symlinks exist in Windows (https://msdn.microsoft.com/en-us/library/windows/desktop/aa365680(v=vs.85).aspx). I am not a Windows guy, pure Linux here. I cannot tell if these Windows symlinks are any good.

[blubberdiblub]

@deragon As far as I can tell, Windows symlinks work fine from a pure file system abstraction point of view. However, Windows Explorer was not able to cope with them very well. I'm not sure if this is still the case on Windows 8 or 10, as I haven't tried them there.

[MurzNN]

@xarinatan can you check and describe how those symlinks are represented in Webdav, web interface and SeaDive mounted folders?

[shoeper]

As files without extension containing the target path.

[ygramoel]

FYI: Dropbox announced support for symlinks today. https://help.dropbox.com/installs-integrations/sync-uploads/symlinks

[shoeper]

The post says it is from mid 2019, so not new.

[andifunke]

Just started with Seafile and I'm a bit surprised to learn about this now 7 years old issue. Seafile still seems to follow symlinks in 2020. Is there a TL;DR for the state of symlinks in Seafile 7.1.x with respect to development plans and/or workarounds?

[KarlZeilhofer]

I use this seafile-ignore.txt file in the root ouf our synced directory:

# see https://www.seafile.com/en/help/ignore/


# symlinks werden von seafile synchronisiert, indem alle Datein in 
# dem Zielverzeichnis synchronisiert werden. 
# Dieses Verhalten ist of unerwünscht. 
# In unserem Fall sind symlinks zu den git-repos oft hilfreich. 

# Alle datein und Ordner, die mit "dontsync" oder mit "symlink" beginnen oder enden, 
# werden mit folgenden Regeln ausgeschlossen:

*/dontsync*
*/*dontsync
*/dontsync*/
*/*dontsync/

*/symlink*
*/*symlink
*/symlink*/
*/*symlink/


# Git Repos sollten eh in einem eigenen Pfad liegen, doch manchmal eben nicht. 
*/.git

So files starting or ending with symlink will not be synced.

[andifunke]

I don't think I will be able re-arrange my paths so that all symlinks can follow a common naming scheme. Guess I'll have to include them all manually to the seafile-ignore.txt, which is obviously quite cumbersome.

# Git Repos sollten eh in einem eigenen Pfad liegen, doch manchmal eben nicht. 
*/.git

As a side note: I don't think this line will be effective as folders need to end with a /. Just tested this and as far as I can tell, only */.git/ would work as a pattern for git folders.

[KarlZeilhofer]

Thanks for that hint!

[gmanley]

I, for one, use symlinks to link folders outside the Seafile folder so that their contents are synced. It lets me sync items while still having them exist outside the Seafile folder. How would I do that without this current symlink behavior? Would hard links work?

Fixing this isn't just a matter of changing the behavior. There needs to be some sort of way to roll this out without just silently breaking the existing behavior as some people do use symlinks for this specific use case. While it may seem like consensus in this thread that everyone just wants the behavior changed, there's probably some selection bias going on.

Dropbox has been around a lot longer than Seafile, and they just made the change about a year ago. I think that goes to show it's not just a no-brainer change and requires some thought on how to migrate to the new behavior. Their document explains how they make a backup copy of the last contents of the symlink and then deselect it from sync.

[dirdi]

@gmanley A file is a pointer to a location on disk. A symlink is a pointer to a file. A hardlink is a pointer to a location on disk, i.e. a file! So yes, hardlinks would work and therefore could replace - most probable - symlinks for your use case. However, hardlinks are unable to cross filesystem boundaries. However, I think using bindfs would be a much more elegant and clean solution in your case, anyway.

[gmanley]

@dirdi Right, I understand the distinction. But from my understanding, hard links generally don't work with folders, correct? I'll take a look at bindfs, thanks.

[dirdi]

@gmanley you can not create a hardlink to a folder, that is true. But I doubt there is reason for you to hurry finding an alternative solution. This bug is over 7 years old and was the very reason I replaced seafile years ago.

[dreua]

@gmanley You can just use a symlinks in the reverse direction. Put the files in Seafile and link to files/folders from wherever you need them.

[xarinatan]

@gmanley multiple solutions actually, for one inside seafile you could make separate repositories in one account, and if you really want everything to be in a single repo, you could instead make a repository and create folders in that repository, which you then individually sync with the folders you want (sub-repositories).

You could also use hardlinks, although I did realize that folder hardlinks do have issues because the filesystem would get in the same kind of loops that Seafile gets when its trying to parse recursive symlinks the wrong way. That's why you use symlinks as symlinks and not hardlinks.......
Still there should be ways to do it if you really have to, you can definitely force it (Linux: ln -F, Windows has 'directory junctions' which you could use), but beware of the recursion issues that could cause your filesystem to hang the same way Seafile does when it meets a recursive symlink (should only be a problem if you refer to a folder higher up in the same tree).

The entire reason Seafile has multiple repositories (and sub repositories) is so you don't NEED to use backwards hacks like this to aggregate data into a single account.. I much love that part of Seafile.
And that's why I really don't get why they still had to use symlinks this way, when they're not supposed to be used this way, as they already solved the issue of not using it that way, by having multiple repositories per account..

Finally I don't see why we can't just have both versions by having a checkbox in the settings that decides how symlinks are handled. I am 100% fine if the DEFAULT behavior is broken for compatibility reasons, but being completely unable to fix it without having to resort to a patch that was written by a good hearted samaritan that simply brought out a feature that seems essentially already in place, is IMHO not a healthy way to deal with it, and am I still considering this to be a large deficit/bug in an otherwise amazing product.

[monotok]

Would be great to have the option to ignore symlinks!

[ikcalB]

@gmanley you can not create a hardlink to a folder, that is true. But I doubt there is reason for you to hurry finding an alternative solution. This bug is over 7 years old and was the very reason I replaced seafile years ago.

What did you replace seafile with?

[dirdi]

@ikcalB:
Since I only need fast and reliable file sync (including symlinks) and no Wiki, Office, Markdown, other crap nobody really asked for ... I use syncthing. Just install it on two (or more) devices, scan a QR code (or type a code) and it just works. You do not even have to set up a server.

The funny thing about this is, that there was a bug report syncthing/syncthing#1776 over at the syncthing repo where smb. asked to follow symlinks (like Seafile does), but they pretty soon figured out that this is a bad idea and that one who really needs this feature, can still achieve this behavior by leveraging a bind mount.

[MurzNN]

Syncthing is cool, but it isn't support "sync on demand" feature, that implemented in SeaDrive client, so this is killer-feature of Seafile and I can't find any alternative to it... :-(
Second missing feature in Syncthing is Web access and sharing of single file or directory for even anonymous access.

[xarinatan]

It feels funny to talk about alternatives here but, NextCloud is a pretty widely accepted alternative, including the web access and sync client, plus a whole heap of other features, I've been considering switching back to it (I used it back when it was still called OwnCloud), but I really like the lean and mean long term git-like history that seafile has, combined with really sweet performance at a scale, I just wish they made the live garbage collector available in the opensource version, and fixed that incredibly annoying symlink bug that still makes Seafile look silly. I mean come on how is hanging and using 100% CPU at all a desirable state, and the symlink support is almost in there? It seemed to be really easy to patch the feature on the client end strangely enough, but perhaps they run into some issue that prevents them from rolling it out? I can only think of reasons which imply using software in general the wrong way though, but maybe I'm mistaken. I'm just salty to have this wart of a bug on an otherwise basically flawless product.

edit: for the seadrive feature, look into WebDAV, NextCloud and Seafile can both support that, it's somewhat similar to a network drive :)

[monotok]

@xarinatan Going off topic a bit but wanted to highlight. I run both Nextcloud 19 and Seafile on my server in lxd containers with attached storage over ISCSI from Freenas. Both solutions have their advantages and disadvantages as you probably know.

However I wouldn't recommend Nextcloud for syncing a lot of small files or even thousands of photos as its sync is incredibly slow compared to seafile. I am talking extreme differences here for some scenarios; a test I did was something like 4000 odd files (cherrytree notes git project and 1000 photos) and seafile was something like 12 seconds and nextcloud was over 7 mins. I commented my test on this issue nextcloud/server#16726 regarding owncloud switching to Go but nextcloud seem more interested in adding stuff like dashboards, talk clients etc than fixing their products primary purpose of syncing files.

Also I always got loads of sync conflicts probably because don't do delta sync.

Not meant to be a nextcloud bashing as they both have their strengths so I use both; seafile for syncing laptops together while nextcloud I use for mainly webdav apps on android and auto uploading photos.

[xarinatan]

@monotok yea exactly, Seafile's performance has always been my favorite part about it, you can run a fully functional server of it on a raspberry pi and expect it to perform well enough to keep a family's photo albums and projects and schoolwork etc backed up and shareable, and I didn't specifically care about the extra features that Nextcloud offered except maybe their well developed calendar.. So far Seafile has been my favorite, but it can be a real pain in the ass with those specific quirks around symlinks (and slow offline garbage collection if you use the community edition).

[ikcalB]

what about a reward at bountysource? I for one'd be willing to contribute (hmm)?

[michaelcadilhac]

For what it's worth, my temporary solution in this thread (#288 (comment)) is still working. It's a handful of commits behind master, though.

[dirdi]

@ikcalB since no consensus has been reached how the current behavior should be alerted, it is unlikely that smb. is able to provide a patch that gets merged, no matter if there is a bounty or not.

BTW I have heard about some portals that have alerted their TOS and bounties that have not been awarded within X month will now expire without a refund to the sponsor.

[kong13661]

hxd,这个新功能还会加吗？7年了＝ ＝。太困扰了，加一个链接同步十几G文件

[ikcalB]

@michaelcadilhac thanks for reminding me of your fork!

if that is still working for you, I might as well try it out.

[liam-k]

There really should be an option where you can choose whether to follow symlinks or not, ideally at client level. It can create problems with other software and recursion. An example:

Final Cut Pro X (video editing suite) uses "databases" with an .fcpbundle extension. They are presented as files on macOS, but they are basically folders and are treated as such on other OS'. Inside those folders, there’s an .fcpcache file which is a symlink to the .fcpbundle folder above it – so FCPX knows it’s supposed to save caches inside the main folder and not elsewhere, which is possible too (in that case, the symlink would point to another location).

Now what happens if you try to sync a Final Cut Pro Database via Seafile? This:

Seafile is now caught in a recursive loop. This goes on forever, hogs a lot of CPU, and, more importantly, uploads an infinite amount of copies of that library onto the server.
Now, even if you set the cache to be outside the main database, it would still create a syncing problem, because Seafile would now, again, not sync the symlink to the location but the content of it, creating larger database folders than necessary and potentially breaking things when you open it in FCPX again. There’s no real solution to this except just not syncing the cache symlink – which sucks if you do want to use an external cache location.

This might be a special case, but FCPX is not the only software to use symlinks in its process, and, as this thread illustrates, there’s personal use cases for symlinks too. I haven’t encountered this problem with any other cloud software, and I don’t understand the decision behind it (I mean, this issue has been open since 2013, not giving that option must be a conscious decision by now). At least there should be a fix that handles recursion elegantly and either throws an error or doesn’t follow those links indefinitely till the last circle of hell...

[linuxturtle]

How can it be that this heinous bug still exists in the client after being reported/discussed for 9 years? I just ran into it because I rsync'd a directory which had a recursive symlink in it "archive -> ./" to a seafile-synced directory. The current behavior of following symlinks and treating them as regular files/directories is utterly insane! Even more insane is the behavior incurred when deleting a recursive symlink, as seafile deletes everything inside the directory (i.e. the current directory) too! Insanity!

[liam-k]

How can it be that this heinous bug still exists in the client after being reported/discussed for 9 years?

Seafile’s support is not bearable anymore at this point. I used to pay for the pro version but I now switched back to Nextcloud. Has it’s own set of problems (especially with large amounts of data) but at least it’s actually actively maintained and improved every day. It’s sad because I really liked Seafile and I find it a lot more usable than Nextcloud, but above all a cloud service needs to be reliable. Seafile has multiple completely no-go bugs which can make you lose data, corrupt or freeze libraries or simply make your life much harder like this symlink thing. It’s partly a community product so I’ve been patient with it, but there’s a limit.

[linuxturtle]

I used to pay for the pro version but I now switched back to Nextcloud.

I also have a nextcloud instance, and nexcloud just ignores symlinks. That's not as useful as doing something intelligent with them, or just syncing them verbatim to clients where they're supported, and ignoring them on clients where they're not supported (hey, just like rsync does!), but it's a LOT better than turning them into some insane kind of hard-link/symlink hybrid like seafile does. I honestly can't figure out any scenario where the current behavior is sane, especially with absolute-path symlinks. I saw one guy who uses the current behavior as a hack to sync stuff without actually putting it in a library, but that's the kind of hack that ought to be ruthlessly broken, not encouraged.