sftp: add symlink support - fixes #5011

[NikitaCOEUR]

What is the purpose of this change?

The primary goal of this change is to enhance rclone's SFTP backend by adding support for preserving symlink structures during file transfers. This is particularly useful for users who need to maintain the integrity of symbolic links when syncing files between systems.

Add a new flag, --sftp-links, to the SFTP backend of rclone. The --sftp-links flag allows users to copy symbolic links (symlinks) as symlinks instead of following them.
The added functionality preserves the symlink structure on the destination, provided both the source and destination remotes support symlinks

Important

Currently supported only between two SFTP remotes

The validation process ensures that symlinks are copied correctly by checking the size and hash of the target file, except when the target is a directory.

Was the change discussed in an issue or in the forum before?

This issue was opened in 2021 and still requested : #5011

Checklist

• I have read the contribution guidelines.

Tip

Need Help to add unit tests, i never do that before ...

• I have added tests for all changes in this PR if appropriate.

• I have added documentation for the changes if appropriate.
• All commit messages are in house style.
• I'm done, this Pull Request is ready for review :-)

[ncw]

What do you think of changing this to work the way the local backend does with the --link flag?

-l, --links                           Translate symlinks to/from regular files with a '.rclonelink' extension

If this was compatible with the local backend then we'd be able to copy symlinks from local <-> sftp as well as sftp <-> sftp which would make this a lot more generally useful

[NikitaCOEUR]

Thank you for your feedback! I'd be happy to try using the --links flag instead of --sftp-links. However, I'm not entirely sure how to achieve this because I added the option directly in the SFTP backend, which automatically created the --sftp-links flag. Could you please clarify how I can utilize --links in this context?

[ncw]

I didn't mean use --links, I meant make --sftp-links work in the same way, making files with the contents of the link with suffix .rclonelink

[NikitaCOEUR]

Ok I'll work on it.

[NikitaCOEUR]

Hello,

I wanted to provide an update on my progress with the SFTP backend and symlink implementation for this PR. I've encountered a few challenges and would appreciate your feedback on the following points:

Comparison of Symlink Sizes [Use target size instead of Symlink Size]

Instead of comparing the size of the symlink itself at the source and destination, I prefer comparing the size of the file pointed to by the symlink. This is because some SFTP servers correct symlinks with relative paths to absolute paths (e.g., ../directory/file becomes /full/path/directory/file).
When verification occurs, the size comparison between the source and destination may differ since the symlink with the absolute path is larger than the one with the relative path. Rclone currently flags this as an issue.
Does this approach seem reasonable, or do you have other suggestions?

Handling Symlinks Pointing to Directories [Ensure target is a directory on source and destination]

The same issue arises with symlinks pointing to directories. It is challenging to verify if the target directories are the same, as comparing sizes or hashing directories is impractical.
My current thought is to simplify the verification by ensuring that both source and destination symlinks point to directories, without further verification of their contents. What are your thoughts on this approach?

Timestamp Management for Symlinks [Ignore Timestamp for symlink on sftp backend]

I've encountered some surprises regarding timestamp management for symlinks. Most SFTP servers don't seem to support modifying the access and modification times of symlinks directly.
Although Chtimes is available, it operates on the file pointed to by the symlink, not the symlink itself.
An equivalent to Lutimes would be needed but is not widely implemented (or possibly not supported at all with the SFTP protocol).
Should we ignore timestamp modifications for symlinks, or should we consider adding a flag like SetModTime=False when TranslateSymlinks is enabled in the SFTP backend?

I’ve opened an issue on the lutimes topic in the SFTP library to gather more input: Issue #594.

---

I wanted to keep you informed of my progress. I’m working on this in my spare time, which may explain some delays in communication.

Thank you for your feedback.

[puellanivis]

An equivalent to Lutimes would be needed but is not widely implemented (or possibly not supported at all with the SFTP protocol).

I’ve responded to your request at the SFTP package itself, but I want to add here for quick reference: your suspicion here is correct. There is sadly no way within the protocol to operate on symlinks directly, except for SSH_FXP_LSTAT and SSH_FXP_READLINK.

All the workarounds I could find “escape-hatch” out of SFTP into the greater SSH protocol to execute an arbitrary command.

[ncw]

Thanks for working on this @NikitaCOEUR

I wanted to provide an update on my progress with the SFTP backend and symlink implementation for this PR. I've encountered a few challenges and would appreciate your feedback on the following points:

Comparison of Symlink Sizes [Use target size instead of Symlink Size]

Instead of comparing the size of the symlink itself at the source and destination, I prefer comparing the size of the file pointed to by the symlink. This is because some SFTP servers correct symlinks with relative paths to absolute paths (e.g., ../directory/file becomes /full/path/directory/file).

That might cause a problem with continual syncing if we are syncing symlinks to the sftp server.

When verification occurs, the size comparison between the source and destination may differ since the symlink with the absolute path is larger than the one with the relative path. Rclone currently flags this as an issue. Does this approach seem reasonable, or do you have other suggestions?

If we want to emulate the handling of the .rclonelink files in the local backend, then the size will need to be the size of the symlink itself as rclone will want to download the contents of the symlink rather than the file pointed to.

I guess we could if necessary mark the size of the symlinks as unknown (-1). The rest of rclone knows what to do in this case. This isn't ideal as rclone then won't see if the symlink size has changed but it would probably work. Maybe this should be an option if it isn't needed on all sftp servers.

Handling Symlinks Pointing to Directories [Ensure target is a directory on source and destination]

The same issue arises with symlinks pointing to directories. It is challenging to verify if the target directories are the same, as comparing sizes or hashing directories is impractical. My current thought is to simplify the verification by ensuring that both source and destination symlinks point to directories, without further verification of their contents. What are your thoughts on this approach?

I think this should just be the symlink size and the code shouldn't care if it is pointing to a file or directory.

Timestamp Management for Symlinks [Ignore Timestamp for symlink on sftp backend]

I've encountered some surprises regarding timestamp management for symlinks. Most SFTP servers don't seem to support modifying the access and modification times of symlinks directly. Although Chtimes is available, it operates on the file pointed to by the symlink, not the symlink itself. An equivalent to Lutimes would be needed but is not widely implemented (or possibly not supported at all with the SFTP protocol). Should we ignore timestamp modifications for symlinks, or should we consider adding a flag like SetModTime=False when TranslateSymlinks is enabled in the SFTP backend?

Hmm, we should certainly avoid setting the timestamp of the destination when we meant to set the timestamp of the symlink.

If we just ignore the SetModTime for the symlinks then potentially you'll have rclone trying to modify the time on each sync.

Returning fs.ErrorCantSetModTime in this case will probably cause rclone to do something sensible!

I’ve opened an issue on the lutimes topic in the SFTP library to gather more input: Issue #594.

I wanted to keep you informed of my progress. I’m working on this in my spare time, which may explain some delays in communication.

No worries, work at whatever pace you need. I work in an entirely interrupt driven way so if you write something on the PR I'll look at it, be that in 1 day or week or month!

[NikitaCOEUR]

Hi,

Returning fs.ErrorCantSetModTime in this case will probably cause rclone to do something sensible!

This causes the recreation of each symlink every time. When we create a symlink, the ModTime is set to the same as the Created Time. When rclone is launched for the first time and the symlink was created by a preceding rclone run, the symlink on the destination and source appear similar, but the ModTime differs. Raising this error triggers the systematic recreation of the symlink. I believe we need to find an alternative because systematically recreating the symlink doesn't make sense to me.

I think we should completely ignore the ModTime check for symlink objects. But I’m not sure how to go about it yet.

[Neustradamus]

Any progress on it?

[pepa65]

Please just implement it by recreating the symlinks first. People choosing to run with this flag would foremost want the symlinks, regardless of whatever they might have pointed at in the source filesystem, and regardless of potential inefficiencies.

I think recreating all symlinks is not a huge cost, and would greatly increase the value of rclone as a whole to me.

[NikitaCOEUR]

Hi everyone,

I took some time to revisit this and tried to implement the approach that ncw suggested for the local backend. I can confirm that it works, but I’ve run into some important limitations along the way.

That said, the scenario that I think will interest most people—synchronizing a local backend with an OpenSSH SFTP server, or synchronizing between two OpenSSH SFTP backends while keeping the same directory structure—should work. This is probably the use case that matters the most in practice.

It’s important to note that the SFTP protocol itself does not allow this feature to be implemented perfectly. There are inherent constraints (like path normalization, chroot environments, and lack of lutimes for symlinks) that can cause edge cases to fail.

I also want to apologize for the time I spent on this. The problems I encountered initially were quite demotivating, and I had even worked around them by adding a script to replicate symlinks between the servers I wanted to sync.

However, the recent comments made it clear that this feature isn’t only desired by me, so I hope this implementation can serve the purpose and be useful for others as well.

Thanks for taking the time to review this!

[pepa65]

I checked out your repo and build the feat/add-sftp-links branch, and it accepts the flag --sftp-links, but for each symlink it reports: "Can't follow symlink without -L/--copy-links". So I tried both --sftp-links and -l, which gives no errors, but does seems to copy the whole file instead of the symlink. Then I tried --sftp-links and -L, also no errors, but also no symlinks. What am I doing wrong?

[NikitaCOEUR]

What exactly was the command you executed?
From local backend to sftp backend ?
Did you use the flag --sftp-links or directly --links ?

[pepa65]

Started with: rclone copy --sftp-links /var/www pp:/data/www -- should I be using --links??

[NikitaCOEUR]

--sftp-links enables TranslateSymlinks only for the SFTP backend, just like --local-links does for the local backend.
To enable TranslateSymlinks for all backends that support it, you need to specify each corresponding flag (--local-links + --sftp-links) or simply use --links, which is the global flag that enables --<backend>-links.

I think that’s the problem you encountered, because source symlinks are not transmitted as .rclonelink files to the SFTP backend.

You should verify the symlinks on the destination side, as I noticed you're transferring them from /var/www to /data/www. Some SFTP servers may rewrite or normalize symlinks depending on their implementation. I ran into a few unexpected behaviors during my tests.

[pepa65]

Hmmm. Local and remote are both ext4 filesystems, and the remote SFTP-server is openssh-sftp-server 9.6p1. What I would like to happen is that the local symlinks will translate into remote symlinks. How can I make that happen, just use --links?

[NikitaCOEUR]

I think you can just use --links yes

[pepa65]

Okay, it seems that copying to an existing remote copy doesn't overwrite existing files... When writing to a new remote directory, it works!

But this does have me concerned, that old/obsolete content does not get overwritten. What if I in the future replace a file with a symlink, will the remote keep having the file instead of the symlink?

[ncw]

@NikitaCOEUR wrote:

I also want to apologize for the time I spent on this. The problems I encountered initially were quite demotivating, and I had even worked around them by adding a script to replicate symlinks between the servers I wanted to sync.

However, the recent comments made it clear that this feature isn’t only desired by me, so I hope this implementation can serve the purpose and be useful for others as well.

Thanks for persevering with this, and I'm sorry for the demotivation.

I will give this a review now - thank you :-)

[ncw]

This is looking very good - thank you :-)

I put one minor comment inline and there are some lint errors for you to fix up too.

And some questions:

Do the integration tests still pass? If you run test_all -backends sftp? You'll need docker for that to work. (See the docs https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#backend-testing )

I guess rclone serve sftp doesn't support symlinks either so I think the integration test will fail there.

Great work :-)

[ncw]

This needs to be called out in the docs like the relevant section in the local backend

Note that --sftp-links just enables this feature for the sftp backend. --links and -l enable the feature for all supported backends (including local) and the VFS.

[NikitaCOEUR]

Indeed, after quickly re-testing to reproduce the issue reported by pepa65, I managed to trigger the same error using rclone serve.

I’m currently working on a detector that evaluates whether the SFTP backend supports symlinks, so that an error can be raised earlier if it doesn’t. The goal is to warn users when the backend cannot handle symlinks and that --links cannot be used with it.

I also noticed that when trying to replace a regular file with a symlink, the symlink is created, but it gets removed during the cleanup phase (because of the .rclonelink handling — this happens even between two local backends). Running the sync a second time creates the symlink correctly.

Okay, it seems that copying to an existing remote copy doesn't overwrite existing files... When writing to a new remote directory, it works!
@pepa65 What exactly is your command? Overwrite “works” for me.

Finally, I saw that symlinks are always flagged for recreation because their ModTime cannot be compared (due to lutime lacks), and hash handling isn’t natively supported with the SFTP backend.
I’m considering “forcing” hash computation — not on the SFTP server side, but on the rclone client side — since the hash can be derived from the string representing the target path. This would only apply to TranslatedLink objects.

Should I add a new commit to this PR, or would it be better to handle this in a separate one?
Also, regarding your earlier comment — should that be a new commit or a force-push update too?

Note: I’m currently trying to run test_all --backends sftp, but I’m seeing a lot of
NOTICE: Failed to open remote "TestSFTPRclone:": NewFs: couldn't connect SSH: dial tcp 127.0.0.1:28621: connect: connection refused
and occasionally messages like Finished OK in 504.613562ms (try 2/5).

I’m not sure if this is expected behavior or if there’s an actual issue.

[ncw]

A detector sounds like a good idea. You can check for that error in the integration tests and skip them.

I also noticed that when trying to replace a regular file with a symlink, the symlink is created, but it gets removed during the cleanup phase (because of the .rclonelink handling — this happens even between two local backends). Running the sync a second time creates the symlink correctly.

Effectively there are two files with the same name which confuses the sync.

Finally, I saw that symlinks are always flagged for recreation because their ModTime cannot be compared (due to lutime lacks), and hash handling isn’t natively supported with the SFTP backend.

We need to fix that. Just calculating the hash should do it though shouldn't it?

Should I add a new commit to this PR, or would it be better to handle this in a separate one?

Yes add to this commit. Let's get everything we need for the feature in this PR.

Also, regarding your earlier comment — should that be a new commit or a force-push update too?

Don't mind - however you like to work. It will all get squashed into one commit when it gets merged.

Note: I’m currently trying to run test_all --backends sftp, but I’m seeing a lot of
NOTICE: Failed to open remote "TestSFTPRclone:": NewFs: couldn't connect SSH: dial tcp 127.0.0.1:28621: connect: connection refused
and occasionally messages like Finished OK in 504.613562ms (try 2/5).

I’m not sure if this is expected behavior or if there’s an actual issue.

Sounds like it is working!

It takes a while then opens an HTML page with the results.

The can't connect might just be startup things or maybe you have a local firewall?

test_all is usually run on Linux so gets the most testing there. It works on macOS I believe but is unlikely to work on Windows.

[NikitaCOEUR]

Hi @ncw,

I've implemented all the changes:

Changes Made

1. ✅ Lint errors - Fixed (0 issues)
2. ✅ Documentation - Added the note about --sftp-links vs --links in the option help text (backend/sftp/sftp.go:285-287)
3. ✅ Symlink capability detection - Added SymlinkValidator interface that:
   - Tests if the backend can actually create symlinks before sync operations
   - Provides clear early error messages when symlinks are enabled but not supported
   - Implemented for both local (Windows privilege check) and SFTP backends
4. ✅ Hash-based comparison for symlinks - Modified CheckHashes() in fs/operations to:
   - Compute hashes for .rclonelink files even when filesystems don't declare common hash support
   - Prevents systematic recreation of symlinks on each sync
5. ✅ SetModTime handling for symlinks - Added SetModTimeCapability interface:
   - SFTP backend declares when SetModTime is not supported (e.g., for symlinks due to lack of lutimes)
   - Equality check skips modtime verification when not supported
   - Prevents false "not equal" detections

Integration Tests

I'm having difficulties getting consistent results with test_all -backends sftp.

• A lot of connection errors ("connection refused")
• These issues appear even on master branch, making it hard to validate if my changes introduce problems

I’m working on Windows through a WSL environment, and I’ve never encountered issues like the ones I’m seeing here. The tests seem to start a container or a service, but those are stopped or restarted between tests — though that doesn’t always seem to happen. When they aren’t running, the tests consistently return a “connection refused” error.
I tried to fix it but couldn’t find a solution. Could you check on your side whether everything looks fine, or if some integration tests might need to be adjusted?
Thanks.

[pepa65]

My job with symlinks went fine.

[ncw]

@NikitaCOEUR

Thanks for the changes - will check those over in a bit

I’m working on Windows through a WSL environment, and I’ve never encountered issues like the ones I’m seeing here. The tests seem to start a container or a service, but those are stopped or restarted between tests — though that doesn’t always seem to happen. When they aren’t running, the tests consistently return a “connection refused” error.
I tried to fix it but couldn’t find a solution. Could you check on your side whether everything looks fine, or if some integration tests might need to be adjusted?

The integration tests work fine on Linux - you can see them running here https://integration.rclone.org

I suspect this is a problem with WSL, perhaps with docker images on WSL. I don't have time to fix the tests on WSL though. Maybe @roucc can test this for you?