New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

symlink support for Linux #288

Open
makomi opened this Issue Jun 30, 2013 · 73 comments

Comments

Projects
None yet
@makomi

makomi commented Jun 30, 2013

seafile server v1.7.0.1
client v1.7
found no log entries related to the issue on my system

Issue
Creating a symlink with "ln -s" creates a symlink on my local machine, but everybody I am sharing the link with gets a copy of the actual file instead of a symlink. I think this is related to issue #130.

Is following symlinks the wanted behavior, i.e. is this a feature and not a bug?

Why I need this feature
In many cases I find it difficult to structure my data sufficiently just by sorting it into folders, because e.g. it would be easier to browse my data if pictures in one folder would also be accessible in another folder.

@killing

This comment has been minimized.

Show comment
Hide comment
@killing

killing Jun 30, 2013

Member

Yes this is the expected behavior. Syncing the symlink (including its target) is not possible since it may not be possible to create the target folder on another client.

Member

killing commented Jun 30, 2013

Yes this is the expected behavior. Syncing the symlink (including its target) is not possible since it may not be possible to create the target folder on another client.

@makomi

This comment has been minimized.

Show comment
Hide comment
@makomi

makomi Jun 30, 2013

Do you think, it would be a good idea to detect whether a symlink links to a file or folder inside the shared library (case 1) or to something that is not part of the library (case 2) and then, based on that destinction, either (1) create a symlink to the file or folder already present in the library or (2) follow the symlink and copy the external data into the library?

makomi commented Jun 30, 2013

Do you think, it would be a good idea to detect whether a symlink links to a file or folder inside the shared library (case 1) or to something that is not part of the library (case 2) and then, based on that destinction, either (1) create a symlink to the file or folder already present in the library or (2) follow the symlink and copy the external data into the library?

@blablup

This comment has been minimized.

Show comment
Hide comment
@blablup

blablup Jul 10, 2013

For me it would be enough if the symlinks would be synced without the files behind it. If the Target maschine do not have the files, the symlink would be dead (That is totaly ok for me). Otherwise I would have Problems in some situations where I have files wich are out of date for me.

Example:
/home/me/seafile_synced_dir/

  • fileA
  • fileB -> /mnt/some_mount_on_a_fileserver/ProjectDirA/fileB

Now if I change my Computer and sync my "seafile_synced_dir" to the new Computer there would be no link to the file-servers file. If I change "fileB" It wouldn't be changed on the file-server and If the file on a file-server changes I wouldn't recognize it.
Also I have links to dirs on servers which shouldn't be synced (In Example VM templates or ISO Files. It would be TB of Data). Or even sensitive Informations which should never be synced to any mobile device.

And at the moment I'm not shore what happens, If I change fileB on another PC (which was synced by seafile, no symlink) on my first PC. Is the symlink replaced? Would the change be written to file-server?

blablup commented Jul 10, 2013

For me it would be enough if the symlinks would be synced without the files behind it. If the Target maschine do not have the files, the symlink would be dead (That is totaly ok for me). Otherwise I would have Problems in some situations where I have files wich are out of date for me.

Example:
/home/me/seafile_synced_dir/

  • fileA
  • fileB -> /mnt/some_mount_on_a_fileserver/ProjectDirA/fileB

Now if I change my Computer and sync my "seafile_synced_dir" to the new Computer there would be no link to the file-servers file. If I change "fileB" It wouldn't be changed on the file-server and If the file on a file-server changes I wouldn't recognize it.
Also I have links to dirs on servers which shouldn't be synced (In Example VM templates or ISO Files. It would be TB of Data). Or even sensitive Informations which should never be synced to any mobile device.

And at the moment I'm not shore what happens, If I change fileB on another PC (which was synced by seafile, no symlink) on my first PC. Is the symlink replaced? Would the change be written to file-server?

@makomi

This comment has been minimized.

Show comment
Hide comment
@makomi

makomi Jul 11, 2013

I propose the following behaviour:

Detect whether a symlink links to a file or folder inside the shared library (case 1) or to something that is not part of the library i.e. external (case 2) and then, based on that destinction, either (1) create a symlink to the file or folder already present in the library or (2) follow the symlink and copy the external data into the library respectively create a potentially broken symlink, if the user chose that option in the Seafile client settings.

makomi commented Jul 11, 2013

I propose the following behaviour:

Detect whether a symlink links to a file or folder inside the shared library (case 1) or to something that is not part of the library i.e. external (case 2) and then, based on that destinction, either (1) create a symlink to the file or folder already present in the library or (2) follow the symlink and copy the external data into the library respectively create a potentially broken symlink, if the user chose that option in the Seafile client settings.

@bobbaluba

This comment has been minimized.

Show comment
Hide comment
@bobbaluba

bobbaluba Jul 31, 2013

I agree with blablup. Please let the symlinks be symlinks, that's the least ambiguous solution. I have tons of projects with relative symlinks, this would cause duplication and/or infinite recursion when synced.

I agree with blablup. Please let the symlinks be symlinks, that's the least ambiguous solution. I have tons of projects with relative symlinks, this would cause duplication and/or infinite recursion when synced.

@blakerohde

This comment has been minimized.

Show comment
Hide comment
@blakerohde

blakerohde Aug 7, 2013

I too would like to see symbolic links blindly kept, regardless if the files exist or not on the other client.

I too would like to see symbolic links blindly kept, regardless if the files exist or not on the other client.

@killing

This comment has been minimized.

Show comment
Hide comment
@killing

killing Aug 7, 2013

Member

OK. Since many people are requesting this feature, we'll add an option in the next release.

Member

killing commented Aug 7, 2013

OK. Since many people are requesting this feature, we'll add an option in the next release.

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Dec 11, 2013

I actually ran into this symlink problem (infinite loop)

May I advice you to :

  • treat symlink as symlink
    and/or
  • use the content of the exclude/ignore file and do not try to follow any link/folder that is excluded

Actually the daemon seems to check all the files/dirs including the ignored ones... (as the seafile.log tells about wt-monitor-linux.c(102))

ghost commented Dec 11, 2013

I actually ran into this symlink problem (infinite loop)

May I advice you to :

  • treat symlink as symlink
    and/or
  • use the content of the exclude/ignore file and do not try to follow any link/folder that is excluded

Actually the daemon seems to check all the files/dirs including the ignored ones... (as the seafile.log tells about wt-monitor-linux.c(102))

@dschoepe

This comment has been minimized.

Show comment
Hide comment
@dschoepe

dschoepe Mar 23, 2014

Has there been any progress on this issue? common/fs-mgr.h mentions a metadata type for links (SEAF_METADATA_TYPE_LINK), but that doesn't seem to be used anywhere yet.

Has there been any progress on this issue? common/fs-mgr.h mentions a metadata type for links (SEAF_METADATA_TYPE_LINK), but that doesn't seem to be used anywhere yet.

@ibeardslee

This comment has been minimized.

Show comment
Hide comment
@ibeardslee

ibeardslee Apr 30, 2014

I'm against following symlinks by default. I'll give an example of something that just blew up for me.

On my Ubuntu desktop (~/Desktop) I have links to things like Dropbox (840MB), Downloads (800MB) our internal network shares, my Music (18GB, on another partition and as a separate Seafile Library) and a link to ~/Seafile where a number of other Libraries are located. I also have just under 200MB of in-progress files and directories sitting on my Desktop that I thought would be good to have sync'd into Seafile.

I ran out of diskspace (started with 16GB free) in /home as Seafile tried to follow the links on the desktop, including the link to ~/Seafile which has the index cache.

Seafile shouldn't be following the symlinks as it syncs.

I'm against following symlinks by default. I'll give an example of something that just blew up for me.

On my Ubuntu desktop (~/Desktop) I have links to things like Dropbox (840MB), Downloads (800MB) our internal network shares, my Music (18GB, on another partition and as a separate Seafile Library) and a link to ~/Seafile where a number of other Libraries are located. I also have just under 200MB of in-progress files and directories sitting on my Desktop that I thought would be good to have sync'd into Seafile.

I ran out of diskspace (started with 16GB free) in /home as Seafile tried to follow the links on the desktop, including the link to ~/Seafile which has the index cache.

Seafile shouldn't be following the symlinks as it syncs.

@johanwilfer

This comment has been minimized.

Show comment
Hide comment
@johanwilfer

johanwilfer Jan 17, 2015

Any updates on this?
I would also expect a symlink to remain a symlink and not to be followed.
I installed the linux clients version 4.0.6 and server 4.0.5 and they are syncing the target, not the link.

Any updates on this?
I would also expect a symlink to remain a symlink and not to be followed.
I installed the linux clients version 4.0.6 and server 4.0.5 and they are syncing the target, not the link.

@ibeardslee

This comment has been minimized.

Show comment
Hide comment
@ibeardslee

ibeardslee Jan 19, 2015

We have our own dev looking at adding some functonality around symlinks as options in the client .. see #130

We have our own dev looking at adding some functonality around symlinks as options in the client .. see #130

@MurzNN

This comment has been minimized.

Show comment
Hide comment
@MurzNN

MurzNN May 3, 2015

Is there any progress about symlink support in next seafile version? Will be good to have per-library option how to handle symlink files:

  • copy target file (current behavior)
  • copy as symlink (new feature)

MurzNN commented May 3, 2015

Is there any progress about symlink support in next seafile version? Will be good to have per-library option how to handle symlink files:

  • copy target file (current behavior)
  • copy as symlink (new feature)
@funkyfuture

This comment has been minimized.

Show comment
Hide comment
@funkyfuture

funkyfuture Jul 24, 2015

my two cents: i rather enjoy this as a feature to aggregate folders that are in differrents paths outside a library into one library. this makes it easy to work on same fs-objects in different contexts.

my two cents: i rather enjoy this as a feature to aggregate folders that are in differrents paths outside a library into one library. this makes it easy to work on same fs-objects in different contexts.

@ikcalB

This comment has been minimized.

Show comment
Hide comment
@ikcalB

ikcalB Aug 17, 2015

@funkyfuture then, a syncing problem comes up - when linking to the same file from two or more libraries: which library to sync to or from? (the same reason, why seafile does not allow creation of libraries inside other libraries)

@killing what about the symlink support now? (Pls read above)
I'd suggest to add an option (maybe even on a per library-basis) to:

  • always copy files (this is the current behavior)
  • copy files if the link points outside the library and preserve library-internal symlinks
  • preserve symlinks in any case

ikcalB commented Aug 17, 2015

@funkyfuture then, a syncing problem comes up - when linking to the same file from two or more libraries: which library to sync to or from? (the same reason, why seafile does not allow creation of libraries inside other libraries)

@killing what about the symlink support now? (Pls read above)
I'd suggest to add an option (maybe even on a per library-basis) to:

  • always copy files (this is the current behavior)
  • copy files if the link points outside the library and preserve library-internal symlinks
  • preserve symlinks in any case
@funkyfuture

This comment has been minimized.

Show comment
Hide comment
@funkyfuture

funkyfuture Aug 21, 2015

@ikcalB i don't get your argument. it will sync from and to any library that links the object in question.

@ikcalB i don't get your argument. it will sync from and to any library that links the object in question.

@MurzNN

This comment has been minimized.

Show comment
Hide comment
@MurzNN

MurzNN Aug 21, 2015

Seems that Windows also have symlink support http://www.howtogeek.com/howto/16226/complete-guide-to-symbolic-links-symlinks-on-windows-or-linux/
So we can do support not only in Linux, but on Windows, Mac, Android too.

MurzNN commented Aug 21, 2015

Seems that Windows also have symlink support http://www.howtogeek.com/howto/16226/complete-guide-to-symbolic-links-symlinks-on-windows-or-linux/
So we can do support not only in Linux, but on Windows, Mac, Android too.

@ikcalB

This comment has been minimized.

Show comment
Hide comment
@ikcalB

ikcalB Nov 30, 2015

@funkyfuture I re-read your two cents and am unfortunately not able to get what you mean.

@MurzNN nice, thanks for sharing!
@freeplant As MurzNN pointed out, symlinks are indeed available on all supported platforms. People who sync from linux, should be well aware of symlinks. (Or should at least be informed by the creators of the library)

What do you think about configurable options like:

  • always copy files (this is the current behavior)
  • copy files if the link points outside the library and preserve library-internal symlinks
  • preserve symlinks in any case

ikcalB commented Nov 30, 2015

@funkyfuture I re-read your two cents and am unfortunately not able to get what you mean.

@MurzNN nice, thanks for sharing!
@freeplant As MurzNN pointed out, symlinks are indeed available on all supported platforms. People who sync from linux, should be well aware of symlinks. (Or should at least be informed by the creators of the library)

What do you think about configurable options like:

  • always copy files (this is the current behavior)
  • copy files if the link points outside the library and preserve library-internal symlinks
  • preserve symlinks in any case
@vin-c

This comment has been minimized.

Show comment
Hide comment
@vin-c

vin-c Dec 1, 2015

Hello,
I've again made a test concerning this problem. In my (particular) environment, I use seafile to sync some folders of my home_dir, some of them are wine environments with symlinks to the root_fs (/). This does lead to infinite loops of symlinks as shown below. I need to sync the environments but not the root_fs.

Test case:

  • create a library inside a test_dir folder
  • add a symlink inside that point just one or two level up (the sync process will be in a loop) :
    lrwxrwxrwx test_dir/library_name/link_to_top_dir -> ../../test_dir
  • add a seafile-ignore.txt excluding "link_to_top_dir/"
  • sync to server

the logs are showing :
repo-mgr.c(1211): Failed to stat /(...)/test_dir/library_name/link_to_top_dir/library_name/link_to_top_dir/library_name/link_to_top_dir(...) Too many levels of symbolic links

I don't care about the "right" behaviour against symlinks (follow them or not) but please make sure that at least seafile-ignore.txt work with them !

Sure if users have options for each library or globally, it will help prevent this behaviour.

Regards

Server : 4.4.6
Client : 4.3.4

Will try with client v5

vin-c commented Dec 1, 2015

Hello,
I've again made a test concerning this problem. In my (particular) environment, I use seafile to sync some folders of my home_dir, some of them are wine environments with symlinks to the root_fs (/). This does lead to infinite loops of symlinks as shown below. I need to sync the environments but not the root_fs.

Test case:

  • create a library inside a test_dir folder
  • add a symlink inside that point just one or two level up (the sync process will be in a loop) :
    lrwxrwxrwx test_dir/library_name/link_to_top_dir -> ../../test_dir
  • add a seafile-ignore.txt excluding "link_to_top_dir/"
  • sync to server

the logs are showing :
repo-mgr.c(1211): Failed to stat /(...)/test_dir/library_name/link_to_top_dir/library_name/link_to_top_dir/library_name/link_to_top_dir(...) Too many levels of symbolic links

I don't care about the "right" behaviour against symlinks (follow them or not) but please make sure that at least seafile-ignore.txt work with them !

Sure if users have options for each library or globally, it will help prevent this behaviour.

Regards

Server : 4.4.6
Client : 4.3.4

Will try with client v5

@freeplant

This comment has been minimized.

Show comment
Hide comment
@freeplant

freeplant Dec 1, 2015

Member

What client version do you use? v5.0 should have fixed the ignore problem.

Member

freeplant commented Dec 1, 2015

What client version do you use? v5.0 should have fixed the ignore problem.

@vin-c

This comment has been minimized.

Show comment
Hide comment
@vin-c

vin-c Dec 1, 2015

@freeplant : Updated client to v5.0.0, same behaviour on client side

For the record, I cleared all configuration and .seafile-data, starting from scratch with v5.0.0 and followed the test case explained before.

The client's logs are still showing the error "Too many levels of symbolic links", but now, the server library does not contain the linked folder at all.
Before (client v4.x), the directory structure was created on the server and I could follow the many directories created (library_name/link_to_top_dir/library_name/link_to_top_dir/...)

This is better but I will not try with a root_fs link... as I don't know where the error is thrown and don't want to mess up my system now...

vin-c commented Dec 1, 2015

@freeplant : Updated client to v5.0.0, same behaviour on client side

For the record, I cleared all configuration and .seafile-data, starting from scratch with v5.0.0 and followed the test case explained before.

The client's logs are still showing the error "Too many levels of symbolic links", but now, the server library does not contain the linked folder at all.
Before (client v4.x), the directory structure was created on the server and I could follow the many directories created (library_name/link_to_top_dir/library_name/link_to_top_dir/...)

This is better but I will not try with a root_fs link... as I don't know where the error is thrown and don't want to mess up my system now...

@DreaminginCodeZH

This comment has been minimized.

Show comment
Hide comment
@DreaminginCodeZH

DreaminginCodeZH Dec 25, 2015

Please add a symlink-as-file option, just as how git handles this accross platforms.

See http://stackoverflow.com/a/954575/2420519

git just stores the contents of the link (i.e. the path of the file system object that it links to) in a 'blob' just like it would for a normal file.

Please add a symlink-as-file option, just as how git handles this accross platforms.

See http://stackoverflow.com/a/954575/2420519

git just stores the contents of the link (i.e. the path of the file system object that it links to) in a 'blob' just like it would for a normal file.

@ikcalB

This comment has been minimized.

Show comment
Hide comment
@ikcalB

ikcalB Feb 9, 2016

@killing more than 2 years ago, you said that you were implementing a solution - still I cannot spot anything in the roadmap.

will you implement configurable symlink support, finally!?

  1. decide if the target is inside -> A or outside -> B of that library
  2. and for either case A or B add a seperate option to:
  • recreate that symbolic link, regardless whether the target exists
  • copy the contents (if possible)
  • ignore that symlink
  • leave some kind of explanatory file behind

(admittedly, the last 2 are not optimal, and might need rethinking)

ikcalB commented Feb 9, 2016

@killing more than 2 years ago, you said that you were implementing a solution - still I cannot spot anything in the roadmap.

will you implement configurable symlink support, finally!?

  1. decide if the target is inside -> A or outside -> B of that library
  2. and for either case A or B add a seperate option to:
  • recreate that symbolic link, regardless whether the target exists
  • copy the contents (if possible)
  • ignore that symlink
  • leave some kind of explanatory file behind

(admittedly, the last 2 are not optimal, and might need rethinking)

@blubberdiblub

This comment has been minimized.

Show comment
Hide comment
@blubberdiblub

blubberdiblub Feb 14, 2016

I'm also waiting for non-broken symlink support for quite some time. Obviously, syncing and creating symlinks as actual symlinks (not the files/directories they point to) is the most preferable solution in the majority of cases. Windows can do symlinks, too. They would even be interchangable between operating systems if they are relative and you normalize them (regarding the path separators, for instance) before putting them into a seafile Library.

I'm also waiting for non-broken symlink support for quite some time. Obviously, syncing and creating symlinks as actual symlinks (not the files/directories they point to) is the most preferable solution in the majority of cases. Windows can do symlinks, too. They would even be interchangable between operating systems if they are relative and you normalize them (regarding the path separators, for instance) before putting them into a seafile Library.

@killing

This comment has been minimized.

Show comment
Hide comment
@killing

killing Feb 15, 2016

Member

Sorry we didn't have time to change the code in the past :-(
I did some investigation on Dropbox's behavior. It seems Dropbox follows symlinks by default. The rationale is to support syncing a folder outside of the Dropbox folder. However, since Seafile supports syncing any folder in the system, this use case is not for Seafile.
So I think it's good to at least provide an option to let the client keep the symlink. We'll still keep following symlink the default behavior, for compatibility.

Member

killing commented Feb 15, 2016

Sorry we didn't have time to change the code in the past :-(
I did some investigation on Dropbox's behavior. It seems Dropbox follows symlinks by default. The rationale is to support syncing a folder outside of the Dropbox folder. However, since Seafile supports syncing any folder in the system, this use case is not for Seafile.
So I think it's good to at least provide an option to let the client keep the symlink. We'll still keep following symlink the default behavior, for compatibility.

@CrazyPandar

This comment has been minimized.

Show comment
Hide comment
@CrazyPandar

CrazyPandar Apr 1, 2016

please add keep symlink option in client settings. make it default to false for compatibility.

please add keep symlink option in client settings. make it default to false for compatibility.

@deragon

This comment has been minimized.

Show comment
Hide comment
@deragon

deragon May 28, 2016

I add my voice to those who would like symlinks to be preserved. Though an global option should exist, the user should be able to override it on a per library basis. Options should be:

  • Follow symlinks always (copy data) - Default
  • Preserve symlinks at all cost - Symlinks or nothing; no data if platform does not support it.
  • Preserve symlinks whenever possible. Follow symlinks (copy data) if platform does not support it.

In a more advance feature, if symlinks could be converted to Windows shortcuts and vice-versa, that would be great. Though, it might not be necessary... Apparently, symlinks exist in Windows (https://msdn.microsoft.com/en-us/library/windows/desktop/aa365680(v=vs.85).aspx). I am not a Windows guy, pure Linux here. I cannot tell if these Windows symlinks are any good.

deragon commented May 28, 2016

I add my voice to those who would like symlinks to be preserved. Though an global option should exist, the user should be able to override it on a per library basis. Options should be:

  • Follow symlinks always (copy data) - Default
  • Preserve symlinks at all cost - Symlinks or nothing; no data if platform does not support it.
  • Preserve symlinks whenever possible. Follow symlinks (copy data) if platform does not support it.

In a more advance feature, if symlinks could be converted to Windows shortcuts and vice-versa, that would be great. Though, it might not be necessary... Apparently, symlinks exist in Windows (https://msdn.microsoft.com/en-us/library/windows/desktop/aa365680(v=vs.85).aspx). I am not a Windows guy, pure Linux here. I cannot tell if these Windows symlinks are any good.

@blubberdiblub

This comment has been minimized.

Show comment
Hide comment
@blubberdiblub

blubberdiblub May 28, 2016

@deragon As far as I can tell, Windows symlinks work fine from a pure file system abstraction point of view. However, Windows Explorer was not able to cope with them very well. I'm not sure if this is still the case on Windows 8 or 10, as I haven't tried them there.

@deragon As far as I can tell, Windows symlinks work fine from a pure file system abstraction point of view. However, Windows Explorer was not able to cope with them very well. I'm not sure if this is still the case on Windows 8 or 10, as I haven't tried them there.

@amalaiteko

This comment has been minimized.

Show comment
Hide comment
@amalaiteko

amalaiteko Aug 20, 2016

The following paragraph is for information only as for me not everything was so clear:
Seafile...

  • ... will follow symlinks, even if they point at an address outside of the synced path and sync that content.
  • ... has no options to choose otherwise, but adding each symlink separately into the seafile-ignore.txt file (or via script). One should add the whole path, for avoiding to not sync the original folder or file (e.g. not just "test1" but "~/MyFiles/test2/test1").
  • ... will create duplicates and it will cost additional space on your disk when symlinks are followed and not ignored, no matter, if the content was already snyced in that library (by another symlink or the original path) or not.

I'd really like an option to choose what happens with symlinks, as others have already been pointing out several times already. Following symlinks can still be the default behavior, if I can change it somehow (client level, library level, or even seafile-ignore.txt-level). I like the idea that links are being copied but not followed.

Update: I updated this comment. It now has only information character as I figured out my problem.

amalaiteko commented Aug 20, 2016

The following paragraph is for information only as for me not everything was so clear:
Seafile...

  • ... will follow symlinks, even if they point at an address outside of the synced path and sync that content.
  • ... has no options to choose otherwise, but adding each symlink separately into the seafile-ignore.txt file (or via script). One should add the whole path, for avoiding to not sync the original folder or file (e.g. not just "test1" but "~/MyFiles/test2/test1").
  • ... will create duplicates and it will cost additional space on your disk when symlinks are followed and not ignored, no matter, if the content was already snyced in that library (by another symlink or the original path) or not.

I'd really like an option to choose what happens with symlinks, as others have already been pointing out several times already. Following symlinks can still be the default behavior, if I can change it somehow (client level, library level, or even seafile-ignore.txt-level). I like the idea that links are being copied but not followed.

Update: I updated this comment. It now has only information character as I figured out my problem.

@denisnone

This comment has been minimized.

Show comment
Hide comment
@denisnone

denisnone Oct 18, 2016

I prefer that Seafile would follow first level directory symlinks and not all other deeper symlinks.

This will give us ability to:

  1. make an aggregation library that will gather some unimportant data from all over the disk without cluttering library list in our web/desktop client interface.
  2. preserve all other symlinks without following.

For now it's not possible to sync some complex code programming projects where build systems heavily depend on symlinks.

Linux kernel for example has some links. Not many, but it's sufficient to break source control systems like git. It'll go crazy complaining about source changes all around.

That's why preserving symlinks is absolutely necessary.

I prefer that Seafile would follow first level directory symlinks and not all other deeper symlinks.

This will give us ability to:

  1. make an aggregation library that will gather some unimportant data from all over the disk without cluttering library list in our web/desktop client interface.
  2. preserve all other symlinks without following.

For now it's not possible to sync some complex code programming projects where build systems heavily depend on symlinks.

Linux kernel for example has some links. Not many, but it's sufficient to break source control systems like git. It'll go crazy complaining about source changes all around.

That's why preserving symlinks is absolutely necessary.

@CrazyPandar

This comment has been minimized.

Show comment
Hide comment
@CrazyPandar

CrazyPandar Oct 18, 2016

at least add an option when creating library。we linux developers heavily
need this feature。

On Tue, Oct 18, 2016, 23:20 denisnone notifications@github.com wrote:

I prefer that Seafile would follow first level directory symlinks and not
all other deeper symlinks.

This will give us ability to:

  1. make an aggregation library that will gather some unimportant data from
    all over the disk without cluttering library list in our web/desktop client
    interface.
  2. preserve all other symlinks without following.

For now it's not possible to sync some complex code programming projects
where build systems heavily depend on symlinks.

Linux kernel for example has some links. Not many, but it's sufficient to
break source control systems like git. It'll go crazy complaining about
source changes all around.

That's why preserving symlinks is absolutely necessary.


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#288 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABJevrtk3RnYJQsd-x9ePrAzUOIBN_hHks5q1OPGgaJpZM4AxyAY
.

at least add an option when creating library。we linux developers heavily
need this feature。

On Tue, Oct 18, 2016, 23:20 denisnone notifications@github.com wrote:

I prefer that Seafile would follow first level directory symlinks and not
all other deeper symlinks.

This will give us ability to:

  1. make an aggregation library that will gather some unimportant data from
    all over the disk without cluttering library list in our web/desktop client
    interface.
  2. preserve all other symlinks without following.

For now it's not possible to sync some complex code programming projects
where build systems heavily depend on symlinks.

Linux kernel for example has some links. Not many, but it's sufficient to
break source control systems like git. It'll go crazy complaining about
source changes all around.

That's why preserving symlinks is absolutely necessary.


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#288 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABJevrtk3RnYJQsd-x9ePrAzUOIBN_hHks5q1OPGgaJpZM4AxyAY
.

@blubberdiblub

This comment has been minimized.

Show comment
Hide comment
@blubberdiblub

blubberdiblub Oct 20, 2016

Not only developers are affected. All kinds of people really. Even if you don't create symlinks yourself, many programs still create symlinks on their own in their work or data directories under your $HOME path. Just think of Steam or Wine. If you want to sync those directories between 2 computers, you screw everything up because Seafile follows symlinks unconditionally.

Not only developers are affected. All kinds of people really. Even if you don't create symlinks yourself, many programs still create symlinks on their own in their work or data directories under your $HOME path. Just think of Steam or Wine. If you want to sync those directories between 2 computers, you screw everything up because Seafile follows symlinks unconditionally.

@jcrben

This comment has been minimized.

Show comment
Hide comment
@jcrben

jcrben Nov 18, 2016

This is my number one request (and also has the second most comments).

jcrben commented Nov 18, 2016

This is my number one request (and also has the second most comments).

@xarinatan

This comment has been minimized.

Show comment
Hide comment
@xarinatan

xarinatan Dec 7, 2016

I too would really like to see symlinks synced as symlinks. I tried to make a backup of /var and /etc on a test machine, but that ended up with infinite recursion. I'd love to make Seafile an easy backup solution for desktops and servers, but with current behavior I'll probably have to make archives to a temporary location first, which is a costly solution since it's multiple terabytes in total size. It also doesn't have the benefit of syncing individual changed files anymore since it'd be stored in an intermediate format...

My two cents on people wanting aggregate libraries, is hard links. Hard links have other implications however, so I don't know if everyone would agree with this.

xarinatan commented Dec 7, 2016

I too would really like to see symlinks synced as symlinks. I tried to make a backup of /var and /etc on a test machine, but that ended up with infinite recursion. I'd love to make Seafile an easy backup solution for desktops and servers, but with current behavior I'll probably have to make archives to a temporary location first, which is a costly solution since it's multiple terabytes in total size. It also doesn't have the benefit of syncing individual changed files anymore since it'd be stored in an intermediate format...

My two cents on people wanting aggregate libraries, is hard links. Hard links have other implications however, so I don't know if everyone would agree with this.

@tiramiseb

This comment has been minimized.

Show comment
Hide comment
@tiramiseb

tiramiseb Jan 24, 2017

Same problem than vin-c here : I had a symlink loop inside a library that made seaf-deamon take 100% of the CPU. It took days before I found out this was because of this loop...

Same problem than vin-c here : I had a symlink loop inside a library that made seaf-deamon take 100% of the CPU. It took days before I found out this was because of this loop...

@blubberdiblub

This comment has been minimized.

Show comment
Hide comment
@blubberdiblub

blubberdiblub Jan 26, 2017

Well, I've given up on Seafile now. The issue has been here for years and nothing has been done about it. I found another synchronization solution which syncs symlinks as symlinks .

Well, I've given up on Seafile now. The issue has been here for years and nothing has been done about it. I found another synchronization solution which syncs symlinks as symlinks .

@MurzNN

This comment has been minimized.

Show comment
Hide comment
@MurzNN

MurzNN Jan 26, 2017

@blubberdiblub, can you post link to this solution? At now I must use rsync for syncing symlinks :(

MurzNN commented Jan 26, 2017

@blubberdiblub, can you post link to this solution? At now I must use rsync for syncing symlinks :(

@blubberdiblub

This comment has been minimized.

Show comment
Hide comment
@blubberdiblub

blubberdiblub Jan 26, 2017

I don't believe it's OK to post it here, but I can send you a mail.

I don't believe it's OK to post it here, but I can send you a mail.

@jcrben

This comment has been minimized.

Show comment
Hide comment
@jcrben

jcrben Jan 26, 2017

For those concerned, please put a thumbs on reaction on the first comment so that this shows up higher when sorting based on reactions. Right now it's only got 3 and many more commenters...

jcrben commented Jan 26, 2017

For those concerned, please put a thumbs on reaction on the first comment so that this shows up higher when sorting based on reactions. Right now it's only got 3 and many more commenters...

@ikcalB

This comment has been minimized.

Show comment
Hide comment
@ikcalB

ikcalB Jan 27, 2017

@blubberdiblub i'd want that mail too.

Even if i think, you should mention it here, since: it's a workaround

ikcalB commented Jan 27, 2017

@blubberdiblub i'd want that mail too.

Even if i think, you should mention it here, since: it's a workaround

@blubberdiblub

This comment has been minimized.

Show comment
Hide comment
@blubberdiblub

blubberdiblub Jan 27, 2017

@ikcalB well, you don't have your mail address exposed on your github profile page. Can I use the one in your ethercat-master repository?

I'm not mentioning it here since it's not just a workaround, but a completely different product. Unless the developers tell me it's ok to mention it, I'm going to assume that my comments will be removed if I do.

@ikcalB well, you don't have your mail address exposed on your github profile page. Can I use the one in your ethercat-master repository?

I'm not mentioning it here since it's not just a workaround, but a completely different product. Unless the developers tell me it's ok to mention it, I'm going to assume that my comments will be removed if I do.

@eHanseJoerg

This comment has been minimized.

Show comment
Hide comment
@eHanseJoerg

eHanseJoerg Jan 27, 2017

@blubberdiblub I would be interested in that mail, too. info (at) ehanse.de

@killing , it would be nice to hear a comment on the implementation status of this.

eHanseJoerg commented Jan 27, 2017

@blubberdiblub I would be interested in that mail, too. info (at) ehanse.de

@killing , it would be nice to hear a comment on the implementation status of this.

@CrazyPandar

This comment has been minimized.

Show comment
Hide comment
@CrazyPandar

CrazyPandar Feb 3, 2017

@xifi-kif

This comment has been minimized.

Show comment
Hide comment
@xifi-kif

xifi-kif Feb 10, 2017

I won't give up for now. Still waiting for a Good solution from the developer

I won't give up for now. Still waiting for a Good solution from the developer

@killing

This comment has been minimized.

Show comment
Hide comment
@killing

killing Feb 11, 2017

Member

I think deeper into this feature. And I found that it requires a lot of changes to Seafile to support symbolic links natively, with similar reasons as for Dropbox: http://stackoverflow.com/questions/15254860/how-to-let-dropbox-treat-symbolic-link-as-it-is

Except for changes on the cloud side, it also causes incompatible behaviors on new and older clients. So the short answer is: we will not support symbolic links natively. But we can provide a not-so-optimal option: to sync symbolic links as files, with the destination path as contents.

Member

killing commented Feb 11, 2017

I think deeper into this feature. And I found that it requires a lot of changes to Seafile to support symbolic links natively, with similar reasons as for Dropbox: http://stackoverflow.com/questions/15254860/how-to-let-dropbox-treat-symbolic-link-as-it-is

Except for changes on the cloud side, it also causes incompatible behaviors on new and older clients. So the short answer is: we will not support symbolic links natively. But we can provide a not-so-optimal option: to sync symbolic links as files, with the destination path as contents.

@ikcalB

This comment has been minimized.

Show comment
Hide comment
@ikcalB

ikcalB Feb 11, 2017

@killing IMHO, this is only an excuse!!

only in this thread (not to mention the others, regarding symlinks) many possible solutions have been brought up. Let me summarise the options for you:

Let the current behavior, namely dereferencing files and having them multiple times, be option 1:

1. de-reference symlinks

-> sync to multiple, identical files

In case where symlinks HAVE to be preserved, because everything else is simply unacceptable:

2. just keep the symlinks without any checking

-> relative symlinks like ../../somefile will work, as long as the point inside the library
-> relative symlinks pointing outside the library will most likely break

-> absolute symlinks will very likely break

3. keep symlinks and substitute library path

That means, if I have the top library folder in /path/to/my/lib,
and an absolute symlink is /path/to/my/lib/secondfolder/thirdfolder,
then the symlink is actually a reference within my library to ./secondfolder/thirdfolder.

On the second machine the path where the library is to be synced: /machine2/my/lib
and shall be restored to /machine2/my/lib/secondfolder/thirdfolder.
Quite easy, imho

-> relative symlinks like ../../somefile will work, as long as the point inside the library
-> relative symlinks pointing outside the library will most likely break

-> absolute symlinks pointing inside the library will work
-> absolute symlinks pointing outside the library will very likely break

Proposed Final Solution

  • Implement option 3
  • The first one already exists
  • (Option 2 is optional imho)
  • additionally let the user choose, what shall happen with likely broken symlinks
    (every example above, that will likely break - you cannot check if it actually would!)
    a. keep the symlinks anyway (all above scenarios)
    b. fall back to Option 1 (which would dereference the symlinks, making sure files are synced - even if symlinks are not preserved then)

REMARK: That is all you can do, with reasonable effort. No magic, nothing complicated. The rest is up to the user, to ensure he understands the implications of using symlinks with seafile

EDIT: avoid confusion by splitting additional information:
HINT: 3.b) would allow, to include "library-external" directories for syncing - i.e. creating one accumulative library, instead of having dozen seperate ones.

ikcalB commented Feb 11, 2017

@killing IMHO, this is only an excuse!!

only in this thread (not to mention the others, regarding symlinks) many possible solutions have been brought up. Let me summarise the options for you:

Let the current behavior, namely dereferencing files and having them multiple times, be option 1:

1. de-reference symlinks

-> sync to multiple, identical files

In case where symlinks HAVE to be preserved, because everything else is simply unacceptable:

2. just keep the symlinks without any checking

-> relative symlinks like ../../somefile will work, as long as the point inside the library
-> relative symlinks pointing outside the library will most likely break

-> absolute symlinks will very likely break

3. keep symlinks and substitute library path

That means, if I have the top library folder in /path/to/my/lib,
and an absolute symlink is /path/to/my/lib/secondfolder/thirdfolder,
then the symlink is actually a reference within my library to ./secondfolder/thirdfolder.

On the second machine the path where the library is to be synced: /machine2/my/lib
and shall be restored to /machine2/my/lib/secondfolder/thirdfolder.
Quite easy, imho

-> relative symlinks like ../../somefile will work, as long as the point inside the library
-> relative symlinks pointing outside the library will most likely break

-> absolute symlinks pointing inside the library will work
-> absolute symlinks pointing outside the library will very likely break

Proposed Final Solution

  • Implement option 3
  • The first one already exists
  • (Option 2 is optional imho)
  • additionally let the user choose, what shall happen with likely broken symlinks
    (every example above, that will likely break - you cannot check if it actually would!)
    a. keep the symlinks anyway (all above scenarios)
    b. fall back to Option 1 (which would dereference the symlinks, making sure files are synced - even if symlinks are not preserved then)

REMARK: That is all you can do, with reasonable effort. No magic, nothing complicated. The rest is up to the user, to ensure he understands the implications of using symlinks with seafile

EDIT: avoid confusion by splitting additional information:
HINT: 3.b) would allow, to include "library-external" directories for syncing - i.e. creating one accumulative library, instead of having dozen seperate ones.

@ikcalB

This comment has been minimized.

Show comment
Hide comment
@ikcalB

ikcalB Feb 11, 2017

@blubberdiblub please use the one in ethercat-master. many thanks!

ikcalB commented Feb 11, 2017

@blubberdiblub please use the one in ethercat-master. many thanks!

@eHanseJoerg

This comment has been minimized.

Show comment
Hide comment
@eHanseJoerg

eHanseJoerg Feb 11, 2017

Here is a workaround that is ok for me: all my symlinks have the name "Link to xyz" . I simply add 'Link to *" to my seafile-ignore.txt in the folder, thats it. Not beautiful, bit I am not willing to give up on Seafile. Yet.

Here is a workaround that is ok for me: all my symlinks have the name "Link to xyz" . I simply add 'Link to *" to my seafile-ignore.txt in the folder, thats it. Not beautiful, bit I am not willing to give up on Seafile. Yet.

@amalaiteko

This comment has been minimized.

Show comment
Hide comment
@amalaiteko

amalaiteko Feb 11, 2017

Correct me if I'm wrong, but @killing didn't say that the solutions proposed aren't helpful or need to be rethought or anything. IMHO he says that the IMPLEMENTATION, the programming (of whatever solution you want regarding symbolic links), doesn't seem to be possible without changing core files and/or functions of seafile. So the proposed solution he can provide without having to change too much within seafile is to provide an option that syncs "symbolic links as files, with the destination path as contents". This solution, it seems, is, from a realistic point of view, possible, in contrary to the solutions many other people want or suggested before.

There is no need to give this post (talking about the last post from @killing ) a thumb down. I'd rather give it a thumb up for giving informations to understand why this issue hasn't been already addressed a long time ago (which I would have appreciated... a long time ago, but better late then never).

@ikcalB Not sure how "deep" your knowledge about the seafile source code is. I hope that the seafile devs already thought about several approaches before as this issues has been addressed from a lot of people already. I don't understand either why this is so hard to implement, as many approaches seem to be easy to implement, or at least within reason.

Correct me if I'm wrong, but @killing didn't say that the solutions proposed aren't helpful or need to be rethought or anything. IMHO he says that the IMPLEMENTATION, the programming (of whatever solution you want regarding symbolic links), doesn't seem to be possible without changing core files and/or functions of seafile. So the proposed solution he can provide without having to change too much within seafile is to provide an option that syncs "symbolic links as files, with the destination path as contents". This solution, it seems, is, from a realistic point of view, possible, in contrary to the solutions many other people want or suggested before.

There is no need to give this post (talking about the last post from @killing ) a thumb down. I'd rather give it a thumb up for giving informations to understand why this issue hasn't been already addressed a long time ago (which I would have appreciated... a long time ago, but better late then never).

@ikcalB Not sure how "deep" your knowledge about the seafile source code is. I hope that the seafile devs already thought about several approaches before as this issues has been addressed from a lot of people already. I don't understand either why this is so hard to implement, as many approaches seem to be easy to implement, or at least within reason.

@amalaiteko

This comment has been minimized.

Show comment
Hide comment
@amalaiteko

amalaiteko Feb 11, 2017

An implementation as many people stated above is overdue, imho. Sad that seafile loses users, because they kind of missed to communicate (for a long time, maybe starting now?), why the implementation is such a big thing, but still understandable that people leave.

My workaround: I created a folder and created symlinks to every folder I want to sync. Helpful especially if you want to sync your home folder but can't sync it, because seafile-data folder is stored there. You can try to create a symlink to your home folder and use a seafile-ignore file for preventing to sync the seafile folder and other folders or symlinks. If that doensn't work, create a symlink to every folder you want to sync. For data that you don't need in your cloud but want to back it up: You can use rsync to sync those folders. rsync by default just copies the symlink and does not follow them (as expected imho). rsync as well has strong features like ssh support, you can provide an ignore file, similar to the seafile-ignore file and more.

Of course this uses another tool to bypass the "wrong" symlink behavior from seafile. By just selecting the folders you want to sync you can try to avoid folders that have a lot of symlinks that you don't need anyway in your cloud.

As always this is a personal workaround and it might not fit for you, but maybe you have some ideas about how your own workaround might look like, at least if it doesn’t include changing to another software.

amalaiteko commented Feb 11, 2017

An implementation as many people stated above is overdue, imho. Sad that seafile loses users, because they kind of missed to communicate (for a long time, maybe starting now?), why the implementation is such a big thing, but still understandable that people leave.

My workaround: I created a folder and created symlinks to every folder I want to sync. Helpful especially if you want to sync your home folder but can't sync it, because seafile-data folder is stored there. You can try to create a symlink to your home folder and use a seafile-ignore file for preventing to sync the seafile folder and other folders or symlinks. If that doensn't work, create a symlink to every folder you want to sync. For data that you don't need in your cloud but want to back it up: You can use rsync to sync those folders. rsync by default just copies the symlink and does not follow them (as expected imho). rsync as well has strong features like ssh support, you can provide an ignore file, similar to the seafile-ignore file and more.

Of course this uses another tool to bypass the "wrong" symlink behavior from seafile. By just selecting the folders you want to sync you can try to avoid folders that have a lot of symlinks that you don't need anyway in your cloud.

As always this is a personal workaround and it might not fit for you, but maybe you have some ideas about how your own workaround might look like, at least if it doesn’t include changing to another software.

@xarinatan

This comment has been minimized.

Show comment
Hide comment
@xarinatan

xarinatan Feb 11, 2017

My main reason to want symlink 'support' is to be able to make backups of my servers without having to create extensive seafile-ignore.txts that may also include configs I do want to backup.

Just syncing the symlinks 'carelessly' with the original path inside them is completely fine with me, I'll happily take that any day over the current "hang and use 100% CPU but it's ok it's backwards compatible" solution.

My main reason to want symlink 'support' is to be able to make backups of my servers without having to create extensive seafile-ignore.txts that may also include configs I do want to backup.

Just syncing the symlinks 'carelessly' with the original path inside them is completely fine with me, I'll happily take that any day over the current "hang and use 100% CPU but it's ok it's backwards compatible" solution.

@ikcalB

This comment has been minimized.

Show comment
Hide comment
@ikcalB

ikcalB Feb 11, 2017

@amalaiteko I'm afraid, I have not looked into the seafile sources. Yet I feel able to argue that way, having worked with inotfy, sync algorithms and related stuff.

As I understand it, the seafile team does not see an easy all-in-one-perfect solution - which is not achieveable imho. One cannot consider any corner case in backing up & restoring.

Additional to the fact that only a minority of people actually depend on that feature (devs are probably only a fraction of all seafile users), I sense the (non)response as unwillingness to deal with the topic itself, rather than implementing a reasonable solution (which means compromise, as I tried to explain)

ikcalB commented Feb 11, 2017

@amalaiteko I'm afraid, I have not looked into the seafile sources. Yet I feel able to argue that way, having worked with inotfy, sync algorithms and related stuff.

As I understand it, the seafile team does not see an easy all-in-one-perfect solution - which is not achieveable imho. One cannot consider any corner case in backing up & restoring.

Additional to the fact that only a minority of people actually depend on that feature (devs are probably only a fraction of all seafile users), I sense the (non)response as unwillingness to deal with the topic itself, rather than implementing a reasonable solution (which means compromise, as I tried to explain)

@ikcalB

This comment has been minimized.

Show comment
Hide comment
@ikcalB

ikcalB Feb 11, 2017

@amalaiteko when using rsync for backup, seafile is pointless. Really.

@eHanseJoerg Renaming symlinks might be viable to you - I would never ever let any software force me into that. Regardless of the fact, that countless software packages depend on symlinks and of course, their correct (unrenamed) filename

ikcalB commented Feb 11, 2017

@amalaiteko when using rsync for backup, seafile is pointless. Really.

@eHanseJoerg Renaming symlinks might be viable to you - I would never ever let any software force me into that. Regardless of the fact, that countless software packages depend on symlinks and of course, their correct (unrenamed) filename

@jcrben

This comment has been minimized.

Show comment
Hide comment

jcrben commented Feb 11, 2017

@killing you might look at how other syncing systems handle symlinks - e.g. https://docs.syncthing.net/users/faq.html?highlight=symbolic#what-things-are-synced

@amalaiteko

This comment has been minimized.

Show comment
Hide comment
@amalaiteko

amalaiteko Feb 11, 2017

@ikcalB I agree with you to everything but the rsync <-> seafile statement. I use seafile not only as a backup solution, but also as a cloud service. I use rsync only for files that I don't use (often) or for files that are not so important (it was some work to set up filters to distinguish between files that are important to me and files that aren't). For my files and folders that are important to me that contain symlinks I kinda sync them carelessly as they hardly ever reach 1MB in size (or I restructured my documents so it won't sync the same files twice).

I agree with you that this solution surely is not beautiful. It works for me. It would be a big enhancement though, if I could rely on seafile for everything.

@ikcalB I agree with you to everything but the rsync <-> seafile statement. I use seafile not only as a backup solution, but also as a cloud service. I use rsync only for files that I don't use (often) or for files that are not so important (it was some work to set up filters to distinguish between files that are important to me and files that aren't). For my files and folders that are important to me that contain symlinks I kinda sync them carelessly as they hardly ever reach 1MB in size (or I restructured my documents so it won't sync the same files twice).

I agree with you that this solution surely is not beautiful. It works for me. It would be a big enhancement though, if I could rely on seafile for everything.

@ikcalB

This comment has been minimized.

Show comment
Hide comment
@ikcalB

ikcalB Feb 11, 2017

@amalaiteko I did not inted to criticise your setup - glad it works for you!

Not everyone is able, to accept the "degradation" of symlinks to duplicate files - i.e. symlinks are often used to choose between alternatives, the same script may be symlinked to different names,...

What's more: If I feel the need to back something up, I want this to happen automatically - which perfectly works with seafile. And because this works so well, I backup everythin that's worth it with that software. Just, that seafile currently isn't able to handle symlinks properly is a dealbreaker for me, and some others

ikcalB commented Feb 11, 2017

@amalaiteko I did not inted to criticise your setup - glad it works for you!

Not everyone is able, to accept the "degradation" of symlinks to duplicate files - i.e. symlinks are often used to choose between alternatives, the same script may be symlinked to different names,...

What's more: If I feel the need to back something up, I want this to happen automatically - which perfectly works with seafile. And because this works so well, I backup everythin that's worth it with that software. Just, that seafile currently isn't able to handle symlinks properly is a dealbreaker for me, and some others

@killing

This comment has been minimized.

Show comment
Hide comment
@killing

killing Feb 12, 2017

Member

Thanks for all your feedbacks. We understand the algorithms very well. But the main difficulty of adding support for symlinks is that the old clients don't recognize symlinks in Seafile's internal metadata structure. Since Seafile clients will create metadata objects and upload them to the server, adding a metadata type that old clients don't recognize is very hard. That would cause a lot of instability. The difficulty is in implementation and architecture.

Member

killing commented Feb 12, 2017

Thanks for all your feedbacks. We understand the algorithms very well. But the main difficulty of adding support for symlinks is that the old clients don't recognize symlinks in Seafile's internal metadata structure. Since Seafile clients will create metadata objects and upload them to the server, adding a metadata type that old clients don't recognize is very hard. That would cause a lot of instability. The difficulty is in implementation and architecture.

@jcrben

This comment has been minimized.

Show comment
Hide comment
@jcrben

jcrben Feb 12, 2017

Allow for people to break the backwards compatibility in a new version, perhaps? Figuring out how to migrate the customers old data sounds tricky, but can't people opt to wipe their history clean and import the same as someone who is starting fresh if they want?

jcrben commented Feb 12, 2017

Allow for people to break the backwards compatibility in a new version, perhaps? Figuring out how to migrate the customers old data sounds tricky, but can't people opt to wipe their history clean and import the same as someone who is starting fresh if they want?

@amalaiteko

This comment has been minimized.

Show comment
Hide comment
@amalaiteko

amalaiteko Feb 12, 2017

I'd even set up my seafile server and client architecture from scratch again to make this happen. For those who can't afford this, they can just stick to their setup, but without the symlink support. I like the idea.

I'd even set up my seafile server and client architecture from scratch again to make this happen. For those who can't afford this, they can just stick to their setup, but without the symlink support. I like the idea.

@ikcalB

This comment has been minimized.

Show comment
Hide comment
@ikcalB

ikcalB Feb 12, 2017

@killing Thanks for clarifying!! I did not know, that is the problem you are facing.

I'd like to continue on @jcrben 's idea: This feature is not of interest for the majority of users. As soon as it is tested & working, what about:

  • new seafile-server installations: activate by default
  • upgraded seafile-server installations: do NOT activate this by default. activating this means, that all clients have to be updated to a special version.

When having this feature on the server side:

  • Then you could either automatically replace all standard files with symlinks on the server, so one does not lose his history.
  • If implementing this is too much effort, I'd happily accept, to dump my history and resync to a new library!!
  • like @amalaiteko said: If only a fresh install of the seafile-server is a viable solution, I'd happily accept that too!!!

HINT: making this option configurable, seems like a good idea anyhow, imho. So ppl can opt out, if they do not want to confuse their users (windows-media-only backups for non-devs come to my mind)

ikcalB commented Feb 12, 2017

@killing Thanks for clarifying!! I did not know, that is the problem you are facing.

I'd like to continue on @jcrben 's idea: This feature is not of interest for the majority of users. As soon as it is tested & working, what about:

  • new seafile-server installations: activate by default
  • upgraded seafile-server installations: do NOT activate this by default. activating this means, that all clients have to be updated to a special version.

When having this feature on the server side:

  • Then you could either automatically replace all standard files with symlinks on the server, so one does not lose his history.
  • If implementing this is too much effort, I'd happily accept, to dump my history and resync to a new library!!
  • like @amalaiteko said: If only a fresh install of the seafile-server is a viable solution, I'd happily accept that too!!!

HINT: making this option configurable, seems like a good idea anyhow, imho. So ppl can opt out, if they do not want to confuse their users (windows-media-only backups for non-devs come to my mind)

@tiramiseb

This comment has been minimized.

Show comment
Hide comment
@tiramiseb

tiramiseb Feb 12, 2017

A workaround that would please me is to replace symlinks with files which contain the symlink target as text, or even completely ignore them ; it would be a selectable option, of course. It would need modifications only on the client side and would not break backward compatibility.

A workaround that would please me is to replace symlinks with files which contain the symlink target as text, or even completely ignore them ; it would be a selectable option, of course. It would need modifications only on the client side and would not break backward compatibility.

@ikcalB

This comment has been minimized.

Show comment
Hide comment
@ikcalB

ikcalB Feb 12, 2017

@tiramiseb only as the last, very last resort ...!!

ikcalB commented Feb 12, 2017

@tiramiseb only as the last, very last resort ...!!

@DreaminginCodeZH

This comment has been minimized.

Show comment
Hide comment
@DreaminginCodeZH

DreaminginCodeZH Feb 12, 2017

I'd like to (have an option to) handle symbolic links the same way as in git, i.e. similar to regular files.

See also:
http://stackoverflow.com/questions/954560/how-does-git-handle-symbolic-links

DreaminginCodeZH commented Feb 12, 2017

I'd like to (have an option to) handle symbolic links the same way as in git, i.e. similar to regular files.

See also:
http://stackoverflow.com/questions/954560/how-does-git-handle-symbolic-links

@ikcalB

This comment has been minimized.

Show comment
Hide comment
@ikcalB

ikcalB Feb 12, 2017

@DreaminginCodeZH which is "option 2." as I called it a few posts above.

if seafile decided to implement symlink handling in any fashion, simply substituting the library path (my "option 3.") is not much of additional effort - where possible - yet plenty of convenience!

ikcalB commented Feb 12, 2017

@DreaminginCodeZH which is "option 2." as I called it a few posts above.

if seafile decided to implement symlink handling in any fashion, simply substituting the library path (my "option 3.") is not much of additional effort - where possible - yet plenty of convenience!

@MurzNN

This comment has been minimized.

Show comment
Hide comment
@MurzNN

MurzNN Feb 12, 2017

Except for changes on the cloud side, it also causes incompatible behaviors on new and older clients. So the short answer is: we will not support symbolic links natively. But we can provide a not-so-optimal option: to sync symbolic links as files, with the destination path as contents.

@killing, as Seafile based on Git for store data, we can use Git version of symlinks in the cloud side: http://stackoverflow.com/a/15465497 - store symlink as regular file, and mark it type as symlink in server side storage.

As I write earlier #288 (comment) - all OS have native symlink support, even Windows, so there are no problem with creating and managing it on client side.

The problem is only on client side with older clients and it's libraries - we must not broke/change it sync process. Best solution is add option "Follow symlinks" to library like --copy-links option for rsync, and turn it on by default on all old libraries.

So users that want to use symlinks will can turn off this option and work normally with symlinks.

P.S. Many linux users actively use symlinks in their works, and broken symlink support in Seafile will be big "surprize" for linux users :(

MurzNN commented Feb 12, 2017

Except for changes on the cloud side, it also causes incompatible behaviors on new and older clients. So the short answer is: we will not support symbolic links natively. But we can provide a not-so-optimal option: to sync symbolic links as files, with the destination path as contents.

@killing, as Seafile based on Git for store data, we can use Git version of symlinks in the cloud side: http://stackoverflow.com/a/15465497 - store symlink as regular file, and mark it type as symlink in server side storage.

As I write earlier #288 (comment) - all OS have native symlink support, even Windows, so there are no problem with creating and managing it on client side.

The problem is only on client side with older clients and it's libraries - we must not broke/change it sync process. Best solution is add option "Follow symlinks" to library like --copy-links option for rsync, and turn it on by default on all old libraries.

So users that want to use symlinks will can turn off this option and work normally with symlinks.

P.S. Many linux users actively use symlinks in their works, and broken symlink support in Seafile will be big "surprize" for linux users :(

@pboley

This comment has been minimized.

Show comment
Hide comment
@pboley

pboley Apr 17, 2017

Just set up Seafile today and discovered this unexpected issue for myself... I know it's not very constructive, but I would like to add my voice to those very strongly wishing for proper symbolic link support in Seafile. I understand the issues with backwards compatibility for clients, but isn't it reasonable that certain server versions (e.g. >=6) will only work with certain (newer) client versions?
This unfortunately is a glaring shortcoming in an otherwise exemplary personal cloud solution.

pboley commented Apr 17, 2017

Just set up Seafile today and discovered this unexpected issue for myself... I know it's not very constructive, but I would like to add my voice to those very strongly wishing for proper symbolic link support in Seafile. I understand the issues with backwards compatibility for clients, but isn't it reasonable that certain server versions (e.g. >=6) will only work with certain (newer) client versions?
This unfortunately is a glaring shortcoming in an otherwise exemplary personal cloud solution.

@KarlZeilhofer

This comment has been minimized.

Show comment
Hide comment
@KarlZeilhofer

KarlZeilhofer Jun 12, 2017

we will use these lines in our seafile-ignore.txt

*/dontsync*
*/dontsync*/

*/symlink*
*/symlink*/

so every folder or file which starts with symlink or dontsync will be ignored by the seafile client.

we will use these lines in our seafile-ignore.txt

*/dontsync*
*/dontsync*/

*/symlink*
*/symlink*/

so every folder or file which starts with symlink or dontsync will be ignored by the seafile client.

@lbr88

This comment has been minimized.

Show comment
Hide comment
@lbr88

lbr88 Nov 1, 2017

I need this too. Incompatibility with older clients is to be expected when upgrading major version so what is the real issue here?

lbr88 commented Nov 1, 2017

I need this too. Incompatibility with older clients is to be expected when upgrading major version so what is the real issue here?

@xarinatan

This comment has been minimized.

Show comment
Hide comment
@xarinatan

xarinatan Nov 4, 2017

IMHO it's kind of crazy we've been debating symlinks for so long. Surely not treating symlinks as symlinks with a program that touts synchronisation as its primary feature counts as broken? And hanging and using 100% CPU until the enduser kills the program is absolutely insane. Recursive symlinks are part of literally every windows, mac and linux installation, and this broken behavior will cause more people issues than it will benefit others.

My solution for the problem would be simply implementing symlinks as symlinks, breaking backwards compatibility with a major version if needed (although I don't see why, older clients will just be incompatible with symlinks and cause the old, already broken behavior), and letting people who require symlinks to work as data aggregation use hard links, which the client will have to detect as normal files as that's what they essentially are to the filesystem.
In my opinion, hard links are what one's supposed to use in this scenario, not letting a bug sit in the code that breaks workflow for the majority of people but allowing certain people to use it as backup 'feature' even though there's other, better ways to do that. https://imgs.xkcd.com/comics/workflow.png

Just my two cents. I'd still love but still cannot use Seafile as the backup client I was hoping it would be. I consider this is a defect in Seafile that I'm waiting for a fix on.

IMHO it's kind of crazy we've been debating symlinks for so long. Surely not treating symlinks as symlinks with a program that touts synchronisation as its primary feature counts as broken? And hanging and using 100% CPU until the enduser kills the program is absolutely insane. Recursive symlinks are part of literally every windows, mac and linux installation, and this broken behavior will cause more people issues than it will benefit others.

My solution for the problem would be simply implementing symlinks as symlinks, breaking backwards compatibility with a major version if needed (although I don't see why, older clients will just be incompatible with symlinks and cause the old, already broken behavior), and letting people who require symlinks to work as data aggregation use hard links, which the client will have to detect as normal files as that's what they essentially are to the filesystem.
In my opinion, hard links are what one's supposed to use in this scenario, not letting a bug sit in the code that breaks workflow for the majority of people but allowing certain people to use it as backup 'feature' even though there's other, better ways to do that. https://imgs.xkcd.com/comics/workflow.png

Just my two cents. I'd still love but still cannot use Seafile as the backup client I was hoping it would be. I consider this is a defect in Seafile that I'm waiting for a fix on.

@ikcalB

This comment has been minimized.

Show comment
Hide comment
@ikcalB

ikcalB Nov 13, 2017

@xarinatan
xD +10 for the hilarious comic!

ikcalB commented Nov 13, 2017

@xarinatan
xD +10 for the hilarious comic!

@dreua

This comment has been minimized.

Show comment
Hide comment
@dreua

dreua Mar 10, 2018

@blubberdiblub Could you tell me your alternative via E-Mail: seafile_pls_make_symlinks@posteo.de (I just created this alias to prevent spam)

@ALL: Have you upvoted the first comment?

@killing : So you are saying you are not going to change anything that needs to change Seafile's internal metadata structure for ever? I guess it's hard, but I think it's worth it! Can we help in any way to make this happen anytime soon?

Simple client-only implementation proposal (opt-in)

Instead of a Symlink, just send a Text file named "seafile__symlink_not_supported__name_of_symlink" with the destination (and if needed other metadata and more comment) as contents to the server.
If the client receives such a file and is capable of symlinks it can recreate the symlink (maybe with an alert to activate the symlink feature in the settings first), if it doesn't the user is informed about possible problems by the file's name and content.

dreua commented Mar 10, 2018

@blubberdiblub Could you tell me your alternative via E-Mail: seafile_pls_make_symlinks@posteo.de (I just created this alias to prevent spam)

@ALL: Have you upvoted the first comment?

@killing : So you are saying you are not going to change anything that needs to change Seafile's internal metadata structure for ever? I guess it's hard, but I think it's worth it! Can we help in any way to make this happen anytime soon?

Simple client-only implementation proposal (opt-in)

Instead of a Symlink, just send a Text file named "seafile__symlink_not_supported__name_of_symlink" with the destination (and if needed other metadata and more comment) as contents to the server.
If the client receives such a file and is capable of symlinks it can recreate the symlink (maybe with an alert to activate the symlink feature in the settings first), if it doesn't the user is informed about possible problems by the file's name and content.

@amalaiteko

This comment has been minimized.

Show comment
Hide comment
@amalaiteko

amalaiteko Mar 11, 2018

Since nothing changed for quite a while I finally gave up on seafile and dropped it. I'll probably go with owncloud/nextcloud for stuff that I need to have access to from different devices and different locations. Everything else will be synced with rsync. This is not quite as nice as seafile would be with the mentioned option, but imho nothing happened for so long with so much requests that I finally came to this decision.

I will still have this issue subscribed in the hope, that it will change one day. Then I can reconsider to use it again or not.

Since nothing changed for quite a while I finally gave up on seafile and dropped it. I'll probably go with owncloud/nextcloud for stuff that I need to have access to from different devices and different locations. Everything else will be synced with rsync. This is not quite as nice as seafile would be with the mentioned option, but imho nothing happened for so long with so much requests that I finally came to this decision.

I will still have this issue subscribed in the hope, that it will change one day. Then I can reconsider to use it again or not.

@IllyaMoskvin

This comment has been minimized.

Show comment
Hide comment
@IllyaMoskvin

IllyaMoskvin Jul 9, 2018

I have some good news! I wrote some scripts to automate symlink management:

https://github.com/IllyaMoskvin/seafile-symlink

There's one for PowerShell, and one for bash. They should be functionally identical. They will find symlinks, add them to seafile-ignore.txt, store their data either in syncable placeholder files or in a text database (configurable), and restore them therefrom.

Obviously not as good as the real thing, but I hope this helps in the meantime.


One thing I noticed while writing this tool is that the Seafile docs might be wrong about how to ignore symbolic links to directories:

If the pattern ends with a slash, it would only match a folder. In other words, "foo/" will match a folder "foo" and paths underneath it, but will not match a regular file or a symbolic link "foo".

If a pattern doesn't end with a slash or a wildcard, it would not match a folder. For example, "foo" can only match regular file "foo" or a symbolic link; while "foo/" and "foo*" match a folder and paths under it.

It appears that Seafile follows symbolic links, seeing them as normal files and directories. So if you want to ignore a symbolic link to a directory, you do need to end it with a forward-slash.

At least – that appears to be the case on Windows. I suspect that due to the whole "everything is a file" thing on *nix systems, you may need to add two seafile-ignore.txt entries for directory symlinks: one with a forward-slash for Windows, and one without, for *nix.

I can confirm that the tool seems to work 100% as intended on Windows, but I haven't had a chance to see if it blocks symlink syncing on other systems. I'm operating on the assumption that the Seafile Client is functionally identical for all systems in regards to ignore rules. I'm a bit burned out from writing it, so if anyone feels like testing that on Linux or macOS, please do, and let me know of any bugs you find.

IllyaMoskvin commented Jul 9, 2018

I have some good news! I wrote some scripts to automate symlink management:

https://github.com/IllyaMoskvin/seafile-symlink

There's one for PowerShell, and one for bash. They should be functionally identical. They will find symlinks, add them to seafile-ignore.txt, store their data either in syncable placeholder files or in a text database (configurable), and restore them therefrom.

Obviously not as good as the real thing, but I hope this helps in the meantime.


One thing I noticed while writing this tool is that the Seafile docs might be wrong about how to ignore symbolic links to directories:

If the pattern ends with a slash, it would only match a folder. In other words, "foo/" will match a folder "foo" and paths underneath it, but will not match a regular file or a symbolic link "foo".

If a pattern doesn't end with a slash or a wildcard, it would not match a folder. For example, "foo" can only match regular file "foo" or a symbolic link; while "foo/" and "foo*" match a folder and paths under it.

It appears that Seafile follows symbolic links, seeing them as normal files and directories. So if you want to ignore a symbolic link to a directory, you do need to end it with a forward-slash.

At least – that appears to be the case on Windows. I suspect that due to the whole "everything is a file" thing on *nix systems, you may need to add two seafile-ignore.txt entries for directory symlinks: one with a forward-slash for Windows, and one without, for *nix.

I can confirm that the tool seems to work 100% as intended on Windows, but I haven't had a chance to see if it blocks symlink syncing on other systems. I'm operating on the assumption that the Seafile Client is functionally identical for all systems in regards to ignore rules. I'm a bit burned out from writing it, so if anyone feels like testing that on Linux or macOS, please do, and let me know of any bugs you find.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment