sftp request with range set to "<file size>-" fails with CURLE_BAD_DOWNLOAD_RESUME #359

Closed
tstack opened this Issue Jul 30, 2015 · 4 comments

Projects

None yet

2 participants

@tstack
tstack commented Jul 30, 2015

An SFTP request raises an error if you try to do a request with the range set to the file size instead of returning no content. However, the correct behavior is seen if the file is empty. Here is an example when requesting an empty file:

curl -v -r "0-" sftp://stack@localhost/Users/stack/empty > /dev/null

  • Trying 127.0.0.1...
    % Total % Received % Xferd Average Speed Time Time Time Current
    Dload Upload Total Spent Left Speed
    0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to localhost (127.0.0.1) port 22 (#0)
  • SSH MD5 fingerprint: 89dcc2863281ded6467103a7a3d69ba1
  • SSH host check: 0, key: AAAAB3NzaC1yc2EAAAABIwAAAQEAs19QFrigpKqUboN8oDhfhqIfZZoqB0Kk9fene3zqivp7bdYwRNV4y6BcEt5vEL42zw0Mo2goMn18luPxSEWg+e7VdBaRL/qrYK6PYevCzvKkqr83zorUq1zeOJNSY+dURCZbR6wIH32Ksf1iiok5EiUXTzdYDnyohGbk1QXGrAVrMqDfrAI0e3L1Y3O55ePHicz38cZtAdj9Zkl4rOn9Ok/4gbLQ4FUtQnqzEAbHNXlWgeDmM1URdc+b6i7F2wbaeHCkbsRLikjWIx5OH8MrAtplRwV6nNd2iXWE5CIOxUXPt8Tt084HeggIdQrkUePTRhtO0IZY35wdkOEZELEYrQ==
  • SSH authentication methods available: publickey,keyboard-interactive
  • Using SSH public key file '(nil)'
  • Using SSH private key file '/Users/stack/.ssh/id_rsa'
  • Initialized SSH public key authentication
  • Authentication complete
    { [0 bytes data]
    0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
    0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
  • Connection #0 to host localhost left intact

So, the request completes successfully with zero bytes. But, downloading a two-byte file fails:

curl -v -r "2-" sftp://stack@localhost/Users/stack/twobytes.txt > /dev/null

  • Trying 127.0.0.1...
    % Total % Received % Xferd Average Speed Time Time Time Current
    Dload Upload Total Spent Left Speed
    0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to localhost (127.0.0.1) port 22 (#0)
  • SSH MD5 fingerprint: 89dcc2863281ded6467103a7a3d69ba1
  • SSH host check: 0, key: AAAAB3NzaC1yc2EAAAABIwAAAQEAs19QFrigpKqUboN8oDhfhqIfZZoqB0Kk9fene3zqivp7bdYwRNV4y6BcEt5vEL42zw0Mo2goMn18luPxSEWg+e7VdBaRL/qrYK6PYevCzvKkqr83zorUq1zeOJNSY+dURCZbR6wIH32Ksf1iiok5EiUXTzdYDnyohGbk1QXGrAVrMqDfrAI0e3L1Y3O55ePHicz38cZtAdj9Zkl4rOn9Ok/4gbLQ4FUtQnqzEAbHNXlWgeDmM1URdc+b6i7F2wbaeHCkbsRLikjWIx5OH8MrAtplRwV6nNd2iXWE5CIOxUXPt8Tt084HeggIdQrkUePTRhtO0IZY35wdkOEZELEYrQ==
  • SSH authentication methods available: publickey,keyboard-interactive
  • Using SSH public key file '(nil)'
  • Using SSH private key file '/Users/stack/.ssh/id_rsa'
  • Initialized SSH public key authentication
  • Authentication complete
  • Offset (2) was beyond file size (2)
    0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
  • Connection #0 to host localhost left intact

curl: (36) Offset (2) was beyond file size (2)

I'm not actually sure if this is curl (7.43.0) complaining, the SSH server (OpenSSH_6.2p2, OSSLShim 0.9.8r 8 Dec 2011 on OS X 10.10.4), or libssh2 (1.4.3).

I tried doing similar requests with a couple HTTP sites and they behave inconsistently, unfortunately. For example, github just sends back the whole file again if you set the range to the file size, but dropbox does the "right" thing and sends back zero-length content.

The use-case is trying to tail a log file. I've ended up just downloading the last byte of the file since that seems to work everywhere.

@bagder
Member
bagder commented Aug 1, 2015

The fact that HTTP range requests are so shaky, it isn't that easy to say for sure what an SFTP resume should do when asked to resume outside of a file size... Just not returning anything and an OK return code doesn't seem entirely desirable either.

@bagder bagder self-assigned this Aug 1, 2015
@tstack
tstack commented Aug 5, 2015

I wouldn't say it's outside the file size, though, it's exactly at the end. This is why I mention the request from '0-' for a zero size file. I do think returning an error for a range of "file_size + 1" is correct, requesting the edge oughta work.

I think the variation in HTTP range requests is more due to the number of HTTP server implementations and it's likely that the edge (literally!) cases are not always going to be handled well.

@bagder
Member
bagder commented Aug 6, 2015

Oh right!! Looks like an off-by-one error in the file size check. Does this patch make anything different?

diff --git a/lib/ssh.c b/lib/ssh.c
index 7b0e57c..94195a7 100644
--- a/lib/ssh.c
+++ b/lib/ssh.c
@@ -2142,11 +2142,11 @@ static CURLcode ssh_statemach_act(struct connectdata *conn, bool *block)
           }
           if(from < 0) {
             /* from is relative to end of file */
             from += size;
           }
-          if(from >= size) {
+          if(from > size) {
             failf(data, "Offset (%"
                   CURL_FORMAT_CURL_OFF_T ") was beyond file size (%"
                   CURL_FORMAT_CURL_OFF_T ")", from, attrs.filesize);
             return CURLE_BAD_DOWNLOAD_RESUME;
           }
@tstack
tstack commented Aug 10, 2015

That patch makes it work. With range "2-" it returns success with no content and "3-" returns an error. Thanks!

@bagder bagder added a commit that closed this issue Aug 10, 2015
@bagder bagder SFTP: fix range request off-by-one in size check
Reported-by: Tim Stack

Closes #359
ade6682
@bagder bagder closed this in ade6682 Aug 10, 2015
@jgsogo jgsogo added a commit to jgsogo/curl that referenced this issue Oct 19, 2015
@bagder @jgsogo bagder + jgsogo SFTP: fix range request off-by-one in size check
Reported-by: Tim Stack

Closes #359
c58a116
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment