Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SFTP Upload does not work with synology diskstation #1474

Closed
mineyannik opened this issue Jun 11, 2017 · 20 comments

Comments

Projects
None yet
6 participants
@mineyannik
Copy link

commented Jun 11, 2017

What is your rclone version (eg output from rclone -V)
1.36

Which OS you are using and how many bits (eg Windows 7, 64 bit)
Ubuntu 64 bit

Which cloud storage system are you using? (eg Google Drive)
SFTP

The command you were trying to run (eg rclone copy /tmp remote:tmp)
mineyannik@s4:~/backup$ /usr/sbin/rclone copy minecraft.zip remote:SemperCraft/Backup/S4/1 -vv

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

2017/06/11 11:09:11 DEBUG : rclone: Version "v1.36" starting with parameters ["/usr/sbin/rclone" "copy" "minecraft.zip" "remote:SemperCraft/Backup/S4/1" "-vv"]
2017/06/11 11:09:12 INFO  : sftp://sempercraft@**:22/SemperCraft/Backup/S4/1: Modify window is 1s
2017/06/11 11:09:12 ERROR : Attempt 1/3 failed with 1 errors and: mkdir dirExists failed: dirExists stat failed: EOF
2017/06/11 11:09:12 ERROR : Attempt 2/3 failed with 1 errors and: mkdir dirExists failed: dirExists stat failed: failed to send packet header: EOF
2017/06/11 11:09:12 ERROR : Attempt 3/3 failed with 1 errors and: mkdir dirExists failed: dirExists stat failed: failed to send packet header: EOF
2017/06/11 11:09:12 Failed to copy: mkdir dirExists failed: dirExists stat failed: failed to send packet header: EOF
@ncw

This comment has been minimized.

Copy link
Owner

commented Jun 11, 2017

Which SFTP server are you using? It looks like it is having a problem with something rclone is doing.

Can you also try with the latest beta from https://beta.rclone.org please?

@mineyannik

This comment has been minimized.

Copy link
Author

commented Jun 11, 2017

It is the default sftp server of my synology diskstation.

@mineyannik

This comment has been minimized.

Copy link
Author

commented Jun 11, 2017

It's the same error with the latest beta.

2017/06/11 12:30:38 DEBUG : rclone: Version "v1.36-170-g2ca477c5β" starting with parameters ["/usr/sbin/rclone" "copy" "minecraft.zip" "remote:SemperCraft/Backup/S4/1" "-vv"]
2017/06/11 12:30:40 INFO  : sftp://sempercraft@**:22/SemperCraft/Backup/S4/1: Modify window is 1s
2017/06/11 12:30:40 ERROR : Attempt 1/3 failed with 1 errors and: mkdir dirExists failed: dirExists stat failed: EOF
2017/06/11 12:30:40 ERROR : Attempt 2/3 failed with 1 errors and: mkdir dirExists failed: dirExists stat failed: failed to send packet header: EOF
2017/06/11 12:30:40 ERROR : Attempt 3/3 failed with 1 errors and: mkdir dirExists failed: dirExists stat failed: failed to send packet header: EOF
2017/06/11 12:30:40 Failed to copy: mkdir dirExists failed: dirExists stat failed: failed to send packet header: EOF
@piotrekoo

This comment has been minimized.

Copy link
Contributor

commented Jul 5, 2017

I'm having the same problem not only when uploading, but also when downloading via SFTP from Synology. Transfering using WinSCP is flawless though.
Synology DSM runs internal-sftp of OpenSSH.

@ncw ncw changed the title SFTP Upload does not work SFTP Upload does not work with synology diskstation Aug 7, 2017

@ncw ncw added this to the Known Problem milestone Aug 7, 2017

@Ossssip

This comment has been minimized.

Copy link

commented Nov 13, 2017

Hi, I am not a user of rclone, but I had the same problem with a Synology NAS sftp and duplicacy, an open source backup tool. I have found out that duplicacy works just fine if absolute path is provided (with double slash between the hostname and the directory, like sftp://user@server:port//directory).

@ncw

This comment has been minimized.

Copy link
Owner

commented Nov 15, 2017

@Ossssip - good tip. Can @mineyannik and @piotrekoo give that a go?

So in rclone speak you would try transferring with remote:/full/path/to/file rather than remote:file

@mineyannik

This comment has been minimized.

Copy link
Author

commented Nov 15, 2017

@blevins321

This comment has been minimized.

Copy link

commented Apr 15, 2018

I was having this problem with the current regular release and stumbled upon this thread. It wouldn't happen every time, but after a dozen or so successful transfers, one would trigger this error and cause all the other running threads to stop transferring.

Extremely simple fix: Just put a / at the beginning of the path. Not the entire root path, just a slash at the beginning.

sftp:Plex/TV Shows becomes sftp:/Plex/TV Shows and all is well.

@ncw

This comment has been minimized.

Copy link
Owner

commented Apr 15, 2018

@blevins321 thanks for the analysis.

That kind of implies a bug in rclone.

Can you try to get a reproducible way of getting the error? If you can do that I can probably fix it!

What SFTP server are you using?

@piotrekoo

This comment has been minimized.

Copy link
Contributor

commented Apr 18, 2018

I can reproduce it on my Synology running an OpenSSH server.
Putting a / at the beginning of the path fixes file transfer but breaks checksum calculation. I'll check this evening if it's by any chance related to the config file.

@ncw

This comment has been minimized.

Copy link
Owner

commented Apr 18, 2018

For SFTP the paths rclone use should be different depending on whether you put a / at the front

sftp:file/path - tries to find the file in the user's home directory
sftp:/file/path - tries to find the file relative to the root of the disk

It should really say this in the docs!

The home directory of the synology sftp user may not make much sense though hence the need for the leading /.

It should probably say this in the docs too!

However this shouldn't break the checksum calculation @piotrekoo - I'd be very interested to see a log made with -vv of the checksum calculation breaking.

@ncw ncw added the doc fix label Apr 18, 2018

@piotrekoo

This comment has been minimized.

Copy link
Contributor

commented Apr 18, 2018

The directory tree in Synology is not the same for SSH and SFTP.
SFTP lists only Shared Folders in / and doesn't start in user's home directory. When SSHing to Synology those folders can be found in /volume[1,2,..]/. That's why the checksum calculation breaks I suppose.
The log: issue1474-1.log.gz

ncw added a commit that referenced this issue Apr 19, 2018

sftp: fail soft with a debug on hash failure #1474
If md5sum/sha1sum fails we debug what it outputed on stderr and return
an empty hash indicating we didn't have a hash, rather than
hash.ErrUnsupported indicating that we don't support this hash type.

This fixes lots of ERROR messages for sftp and synology NAS which,
while it supports md5sum the SFTP paths and the SSH paths are
different so md5sum doesn't work.

We also stop disabling md5sum/sha1sum on errors since typically Hashes
is only checked at the start of a sync run and isn't expected to
change dynamically.
@ncw

This comment has been minimized.

Copy link
Owner

commented Apr 19, 2018

@piotrekoo thanks for the log. Can you try this beta? It will soft fail on hash failure and also print as a debug the output from md5sum. I'd be interested to see what it says. That should stop the ERROR logs.

https://beta.rclone.org/v1.40-095-ga8e52cdd-fix-1474-sftp-hash/ (uploaded in 15-30 mins)

I've also put a bit of extra text about using a leading / for synology in the docs.

ncw added a commit that referenced this issue Apr 19, 2018

sftp: fail soft with a debug on hash failure #1474
If md5sum/sha1sum fails we debug what it outputed on stderr and return
an empty hash indicating we didn't have a hash, rather than
hash.ErrUnsupported indicating that we don't support this hash type.

This fixes lots of ERROR messages for sftp and synology NAS which,
while it supports md5sum the SFTP paths and the SSH paths are
different so md5sum doesn't work.

We also stop disabling md5sum/sha1sum on errors since typically Hashes
is only checked at the start of a sync run and isn't expected to
change dynamically.
@ncw

This comment has been minimized.

Copy link
Owner

commented Apr 19, 2018

Oops that build failed - try this one:

https://beta.rclone.org/v1.40-095-g350b3c03-fix-1474-sftp-hash/ (uploaded in 15-30 mins)

ncw added a commit that referenced this issue Apr 21, 2018

sftp: fail soft with a debug on hash failure #1474
If md5sum/sha1sum fails we debug what it outputed on stderr and return
an empty hash indicating we didn't have a hash, rather than
hash.ErrUnsupported indicating that we don't support this hash type.

This fixes lots of ERROR messages for sftp and synology NAS which,
while it supports md5sum the SFTP paths and the SSH paths are
different so md5sum doesn't work.

We also stop disabling md5sum/sha1sum on errors since typically Hashes
is only checked at the start of a sync run and isn't expected to
change dynamically.
@ncw

This comment has been minimized.

Copy link
Owner

commented Apr 21, 2018

I've merged this to master in

https://beta.rclone.org/v1.40-111-g870c58f7/ (uploaded in 15-30 mins)

which will be in the latest beta and 1.41

@ncw ncw closed this Apr 21, 2018

@ncw ncw modified the milestones: Known Problem, v1.41 Apr 21, 2018

@piotrekoo

This comment has been minimized.

Copy link
Contributor

commented Apr 25, 2018

@ncw Tested, but forgot to send the log. Sorry. issue1474-2.log.gz

I think the only way to make hash calc working for synology boxes is to introduce an option allowing user to specify a ssh path
I'll try coding it this week. My go skills are rather non-existent but the code doesn't look complicated

@ncw

This comment has been minimized.

Copy link
Owner

commented Apr 25, 2018

@piotrekoo thanks for the log. The error does look sensible :-)

I think the only way to make hash calc working for synology boxes is to introduce an option allowing user to specify a ssh path

That is a good idea. So a new config variable for the whole path for the ssh commands.

I'll try coding it this week. My go skills are rather non-existent but the code doesn't look complicated

If you've ever programmed in C or a descendant language (Java, C# etc) go is really pretty easy!

piotrekoo added a commit to piotrekoo/rclone that referenced this issue Apr 26, 2018

sftp: Add --ssh-path-override flag ncw#1474
The flag allows calculation of checksums on systems using
different paths for SSH and SFTP, like synology NAS boxes.

piotrekoo added a commit to piotrekoo/rclone that referenced this issue Apr 26, 2018

sftp: Add --ssh-path-override flag ncw#1474
The flag allows calculation of checksums on systems using
different paths for SSH and SFTP, like synology NAS boxes.

ncw added a commit that referenced this issue Apr 30, 2018

sftp: Add --ssh-path-override flag #1474
The flag allows calculation of checksums on systems using
different paths for SSH and SFTP, like synology NAS boxes.
@ncw

This comment has been minimized.

Copy link
Owner

commented Apr 30, 2018

The --ssh-path-override will be here

https://beta.rclone.org/v1.41-006-g31162496/ (uploaded in 15-30 mins)

and in the latest beta from now on and in the latest beta.

@curious-jim

This comment has been minimized.

Copy link

commented Nov 4, 2018

was this option removed in new version? I was using 1.42 for a while and it worked, but today I needed to setup another backup and I downloaded the latest 1.44 and I got:

Fatal error: unknown flag: --ssh-path-override
Error: unknown flag: --ssh-path-override

downgrade to 1.42 fixed the issue

@ncw

This comment has been minimized.

Copy link
Owner

commented Nov 4, 2018

The parameter is this

  --sftp-path-override string   Override path used by SSH connection.

It got renamed in the backend config unification - sorry I missed that out the changelog. However you can now store it in the config file too (config with the advanced options) which you should find useful as path_override.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.