sftp request with range set to "<file size>-" fails with CURLE_BAD_DOWNLOAD_RESUME

[tstack]

An SFTP request raises an error if you try to do a request with the range set to the file size instead of returning no content. However, the correct behavior is seen if the file is empty. Here is an example when requesting an empty file:

curl -v -r "0-" sftp://stack@localhost/Users/stack/empty > /dev/null

• Trying 127.0.0.1...
  % Total % Received % Xferd Average Speed Time Time Time Current
  Dload Upload Total Spent Left Speed
  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to localhost (127.0.0.1) port 22 (#0)
• SSH MD5 fingerprint: 89dcc2863281ded6467103a7a3d69ba1
• SSH host check: 0, key: AAAAB3NzaC1yc2EAAAABIwAAAQEAs19QFrigpKqUboN8oDhfhqIfZZoqB0Kk9fene3zqivp7bdYwRNV4y6BcEt5vEL42zw0Mo2goMn18luPxSEWg+e7VdBaRL/qrYK6PYevCzvKkqr83zorUq1zeOJNSY+dURCZbR6wIH32Ksf1iiok5EiUXTzdYDnyohGbk1QXGrAVrMqDfrAI0e3L1Y3O55ePHicz38cZtAdj9Zkl4rOn9Ok/4gbLQ4FUtQnqzEAbHNXlWgeDmM1URdc+b6i7F2wbaeHCkbsRLikjWIx5OH8MrAtplRwV6nNd2iXWE5CIOxUXPt8Tt084HeggIdQrkUePTRhtO0IZY35wdkOEZELEYrQ==
• SSH authentication methods available: publickey,keyboard-interactive
• Using SSH public key file '(nil)'
• Using SSH private key file '/Users/stack/.ssh/id_rsa'
• Initialized SSH public key authentication
• Authentication complete
  { [0 bytes data]
  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
• Connection #0 to host localhost left intact

So, the request completes successfully with zero bytes. But, downloading a two-byte file fails:

curl -v -r "2-" sftp://stack@localhost/Users/stack/twobytes.txt > /dev/null

• Trying 127.0.0.1...
  % Total % Received % Xferd Average Speed Time Time Time Current
  Dload Upload Total Spent Left Speed
  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to localhost (127.0.0.1) port 22 (#0)
• SSH MD5 fingerprint: 89dcc2863281ded6467103a7a3d69ba1
• SSH host check: 0, key: AAAAB3NzaC1yc2EAAAABIwAAAQEAs19QFrigpKqUboN8oDhfhqIfZZoqB0Kk9fene3zqivp7bdYwRNV4y6BcEt5vEL42zw0Mo2goMn18luPxSEWg+e7VdBaRL/qrYK6PYevCzvKkqr83zorUq1zeOJNSY+dURCZbR6wIH32Ksf1iiok5EiUXTzdYDnyohGbk1QXGrAVrMqDfrAI0e3L1Y3O55ePHicz38cZtAdj9Zkl4rOn9Ok/4gbLQ4FUtQnqzEAbHNXlWgeDmM1URdc+b6i7F2wbaeHCkbsRLikjWIx5OH8MrAtplRwV6nNd2iXWE5CIOxUXPt8Tt084HeggIdQrkUePTRhtO0IZY35wdkOEZELEYrQ==
• SSH authentication methods available: publickey,keyboard-interactive
• Using SSH public key file '(nil)'
• Using SSH private key file '/Users/stack/.ssh/id_rsa'
• Initialized SSH public key authentication
• Authentication complete
• Offset (2) was beyond file size (2)
  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
• Connection #0 to host localhost left intact

curl: (36) Offset (2) was beyond file size (2)

I'm not actually sure if this is curl (7.43.0) complaining, the SSH server (OpenSSH_6.2p2, OSSLShim 0.9.8r 8 Dec 2011 on OS X 10.10.4), or libssh2 (1.4.3).

I tried doing similar requests with a couple HTTP sites and they behave inconsistently, unfortunately. For example, github just sends back the whole file again if you set the range to the file size, but dropbox does the "right" thing and sends back zero-length content.

The use-case is trying to tail a log file. I've ended up just downloading the last byte of the file since that seems to work everywhere.

[bagder]

The fact that HTTP range requests are so shaky, it isn't that easy to say for sure what an SFTP resume should do when asked to resume outside of a file size... Just not returning anything and an OK return code doesn't seem entirely desirable either.

[tstack]

I wouldn't say it's outside the file size, though, it's exactly at the end. This is why I mention the request from '0-' for a zero size file. I do think returning an error for a range of "file_size + 1" is correct, requesting the edge oughta work.

I think the variation in HTTP range requests is more due to the number of HTTP server implementations and it's likely that the edge (literally!) cases are not always going to be handled well.

[bagder]

Oh right!! Looks like an off-by-one error in the file size check. Does this patch make anything different?

diff --git a/lib/ssh.c b/lib/ssh.c
index 7b0e57c..94195a7 100644
--- a/lib/ssh.c
+++ b/lib/ssh.c
@@ -2142,11 +2142,11 @@ static CURLcode ssh_statemach_act(struct connectdata *conn, bool *block)
           }
           if(from < 0) {
             /* from is relative to end of file */
             from += size;
           }
-          if(from >= size) {
+          if(from > size) {
             failf(data, "Offset (%"
                   CURL_FORMAT_CURL_OFF_T ") was beyond file size (%"
                   CURL_FORMAT_CURL_OFF_T ")", from, attrs.filesize);
             return CURLE_BAD_DOWNLOAD_RESUME;
           }

[tstack]

That patch makes it work. With range "2-" it returns success with no content and "3-" returns an error. Thanks!