Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using --etag-compare with --output truncate existing output #5181

Closed
kwon-young opened this issue Apr 4, 2020 · 2 comments
Closed

Using --etag-compare with --output truncate existing output #5181

kwon-young opened this issue Apr 4, 2020 · 2 comments
Labels

Comments

@kwon-young
Copy link
Contributor

@kwon-young kwon-young commented Apr 4, 2020

I did this

$ curl --etag-save test.etag -o test.ext -v 'http://myurl.com/test.ext'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                                                                                                                                                              
                                 Dload  Upload   Total   Spent    Left  Speed                                                                                                                                                                                
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying ip:80...                                                                                                                                                  
* Connected to myurl.com (ip) port 80 (#0)                                                                                                                                                                                                     
> GET /test.ext HTTP/1.1                                                                                                                    
> Host: myurl.com                                                                                                                                                                                                                                    
> User-Agent: curl/7.69.1                                                                                                                                                                                                                                    
> Accept: */*                                                                                                                                                                                                                                                
>                                                                                                                                                                                                                                                            
* Mark bundle as not supporting multiuse                                                                                                                                                                                                                     
< HTTP/1.1 200 OK                                                                                                                                                                                                                                            < Date: Sat, 04 Apr 2020 00:21:28 GMT                                                                                                                                                                                                                        
< Content-Type: application/ext                                                                                                                                                                                                                             
< Content-Length: 3840872                                                                                                                                                                                                                                    
< Connection: keep-alive                                                                                                                                                                                                                                     
< Set-Cookie: __cfduid=d55f833dcabfc0685a31a6437ba68ad091585959688; expires=Mon, 04-May-20 00:21:28 GMT; path=/; domain=.myurl.com; HttpOnly; SameSite=Lax                                                                                                   
< Last-Modified: Tue, 25 Feb 2014 02:26:16 GMT                                                                                                                                                                                                               
< ETag: "530bff48-3a9b68"                                                                                                                                                                                                                                    
< Cache-Control: max-age=14400                                                                                                                                                                                                                               
< CF-Cache-Status: HIT                                                                                                                                                                                                                                       
< Age: 6791                                                                                                                                                                                                                                                  
< Accept-Ranges: bytes                                                                                                                                                                                                                                       
< Server: cloudflare                                                                                                                                                                                                                                         
< CF-RAY: 57e6de961b60ee4c-CDG                                                                                                                                                                                                                               
<                                                                                                                                                                                                                                                            
{ [964 bytes data]                                                                                                                                                                                                                                           
100 3750k  100 3750k    0     0  5138k      0 --:--:-- --:--:-- --:--:-- 5131k                                                                                                                                                                               
* Connection #0 to host myurl.com left intact

This download test.ext as expected.

If we try to update test.ext but only if the file changed on the server using --etag-compare and the server respond that the file has not been updated, test.ext is truncated:

$ curl --etag-compare test.etag -o test.ext -v 'http://myurl.com/test.ext'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                                                                                                                                                              
                                 Dload  Upload   Total   Spent    Left  Speed                                                                                                                                                                                
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying ip:80...                                                                                                                                                  
* Connected to myurl.com (ip) port 80 (#0)                                                                                                                                                                                                     
> GET /test.ext HTTP/1.1                                                                                                                    
> Host: myurl.com                                                                                                                                                                                                                                    
> User-Agent: curl/7.69.1                                                                                                                                                                                                                                    
> Accept: */*                                                                                                                                                                                                                                                
> If-None-Match: "530bff48-3a9b68"                                                                                                                                                                                                                           
>                                                                                                                                                                                                                                                            
* Mark bundle as not supporting multiuse                                                                                                                                                                                                                     
< HTTP/1.1 304 Not Modified                                                                                                                                                                                                                                  
< Date: Sat, 04 Apr 2020 00:23:30 GMT                                                                                                                                                                                                                        < Connection: keep-alive                                                                                                                                                                                                                                     
< Set-Cookie: __cfduid=d582f0a38586003e65053440888ac938f1585959810; expires=Mon, 04-May-20 00:23:30 GMT; path=/; domain=.myurl.com; HttpOnly; SameSite=Lax                                                                                                   
< Last-Modified: Tue, 25 Feb 2014 02:26:16 GMT                                                                                                                                                                                                               
< ETag: "530bff48-3a9b68"                                                                                                                                                                                                                                    
< Cache-Control: max-age=14400                                                                                                                                                                                                                               
< CF-Cache-Status: HIT                                                                                                                                                                                                                                       
< Age: 6913                                                                                                                                                                                                                                                  
< Server: cloudflare                                                                                                                                                                                                                                         
< CF-RAY: 57e6e18f2f0da84f-CDG                                                                                                                                                                                                                               
<                                                                                                                                                                                                                                                              0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0                                                                                                                                                                               
* Connection #0 to host myurl.com left intact
$ file test.ext
test.ext: empty

curl/libcurl version

curl 7.69.1 (x86_64-pc-linux-gnu) libcurl/7.69.1 OpenSSL/1.1.1e zlib/1.2.11 libidn2/2.3.0 libpsl/0.21.0 (+libidn2/2.2.0) libssh2/1.9.0 nghttp2/1.40.0
Release-Date: 2020-03-11
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets

operating system

Linux edwin 5.5.13-1-MANJARO #1 SMP PREEMPT Wed Mar 25 17:14:28 UTC 2020 x86_64 GNU/Linux

Discussion

I would like to help resolve this issue, but I am having a hard writing a test case of this.

I would need the ability to specify existing files like test.etag and test.ext prior running the curl command.
Then assert that test.ext is still the same after running the curl command.

Is there some documentation on the language used for the test cases?

@bagder
Copy link
Member

@bagder bagder commented Apr 4, 2020

I would need the ability to specify existing files like test.etag and test.ext prior running the curl command.

You can create files for the test in <client><file> (where file can also have 1,2,3,4 suffixes I believe).

Then assert that test.ext is still the same after running the curl command.

<verify><file>, also with suffixes if you need more than one

Is there some documentation on the language used for the test cases?

Yes tests/FILEFORMAT but I will also recommend grepping existing test cases in case the language in the docs isn't clear!

@bagder bagder added the cmdline tool label Apr 4, 2020
bagder added a commit that referenced this issue Apr 4, 2020
Using the --etag-compare option, we can ask the server to either send
the file if it was updated or respond with a 304 response code to
signify that the file did not change.
This commit recognize this response code and avoid writing the output
file, which avoid truncating a previously downloaded file.

This fix issue #5181.
@kwon-young
Copy link
Contributor Author

@kwon-young kwon-young commented Apr 4, 2020

I saw that you wrote the test case for this.

Thank you for your help.

@kwon-young kwon-young closed this Apr 5, 2020
@kwon-young kwon-young reopened this Apr 5, 2020
@bagder bagder closed this in 54ecc11 Apr 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.