Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use both --etag-compare and --etag-save with the same etag file name #5179

Closed
kwon-young opened this issue Apr 3, 2020 · 0 comments
Closed

Comments

@kwon-young
Copy link
Contributor

kwon-young commented Apr 3, 2020

I did this

On the first run where test.etag and test.ext does not exist:

$ curl --etag-compare test.etag --etag-save test.etag -o test.ext -v 'http://myurl.com/test.ext'

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying ip:80...
* Connected to myurl.com (ip) port 80 (#0)
> GET /test.ext HTTP/1.1
> Host: myurl.com
> User-Agent: curl/7.69.1
> Accept: */*
> If-None-Match: ""
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Fri, 03 Apr 2020 17:43:36 GMT
< Content-Type: application/ext
< Content-Length: 3840872
< Connection: keep-alive
< Set-Cookie: __cfduid=d8d26cdd5d1359a2545e22facdeb5c54f1585935816; expires=Sun, 03-May-20 17:43:36 GMT; path=/; domain=.myurl.com; HttpOnly; SameSite=Lax
< Last-Modified: Tue, 25 Feb 2014 02:26:16 GMT
< ETag: "530bff48-3a9b68"
< Cache-Control: max-age=14400
< CF-Cache-Status: REVALIDATED
< Accept-Ranges: bytes
< Server: cloudflare
< CF-RAY: 57e497c43ee9ee8d-CDG
< 
{ [967 bytes data]
100 3750k  100 3750k    0     0  2976k      0  0:00:01  0:00:01 --:--:-- 2976k
* Connection #0 to host myurl.com left intact

On the second run, test.ext is downloaded again.

$ curl --etag-compare test.etag --etag-save test.etag -o test.ext -v 'http://myurl.com/test.ext'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying ip:80...
* Connected to myurl.com (ip) port 80 (#0)
> GET /test.ext HTTP/1.1
> Host: myurl.com
> User-Agent: curl/7.69.1
> Accept: */*
> If-None-Match: ""
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Fri, 03 Apr 2020 17:46:33 GMT
< Content-Type: application/ext
< Content-Length: 3840872
< Connection: keep-alive
< Set-Cookie: __cfduid=d2c4673cee3162b5136f49cde885137ca1585935993; expires=Sun, 03-May-20 17:46:33 GMT; path=/; domain=.myurl.com; HttpOnly; SameSite=Lax
< Last-Modified: Tue, 25 Feb 2014 02:26:16 GMT
< ETag: "530bff48-3a9b68"
< Cache-Control: max-age=14400
< CF-Cache-Status: HIT
< Age: 177
< Accept-Ranges: bytes
< Server: cloudflare
< CF-RAY: 57e49c182b07a8c1-CDG
< 
{ [965 bytes data]
100 3750k  100 3750k    0     0  3015k      0  0:00:01  0:00:01 --:--:-- 3015k
* Connection #0 to host myurl.com left intact

I expected the following

The second curl command should not download text.ext again.

curl/libcurl version

curl 7.69.1 (x86_64-pc-linux-gnu) libcurl/7.69.1 OpenSSL/1.1.1e zlib/1.2.11 libidn2/2.3.0 libpsl/0.21.0 (+libidn2/2.2.0) libssh2/1.9.0 nghttp2/1.40.0
Release-Date: 2020-03-11
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets

operating system

Linux edwin 5.5.13-1-MANJARO #1 SMP PREEMPT Wed Mar 25 17:14:28 UTC 2020 x86_64 GNU/Linux

Discussion

After going through the pull-request #4678 that implemented using both --etag-compare and --etag-save, I understood that the --etag-save is processed first and open for writing test.etag (see

curl/src/tool_operate.c

Lines 908 to 976 in 57476a9

/* --etag-save */
etag_save = &per->etag_save;
etag_save->stream = stdout;
if(config->etag_save_file) {
/* open file for output: */
if(strcmp(config->etag_save_file, "-")) {
FILE *newfile = fopen(config->etag_save_file, "wb");
if(!newfile) {
warnf(
config->global,
"Failed to open %s\n", config->etag_save_file);
result = CURLE_WRITE_ERROR;
break;
}
else {
etag_save->filename = config->etag_save_file;
etag_save->s_isreg = TRUE;
etag_save->fopened = TRUE;
etag_save->stream = newfile;
}
}
else {
/* always use binary mode for protocol header output */
set_binmode(etag_save->stream);
}
}
/* --etag-compare */
if(config->etag_compare_file) {
char *etag_from_file = NULL;
char *header = NULL;
/* open file for reading: */
FILE *file = fopen(config->etag_compare_file, FOPEN_READTEXT);
if(!file) {
errorf(config->global,
"Failed to open %s\n", config->etag_compare_file);
result = CURLE_READ_ERROR;
break;
}
if((PARAM_OK == file2string(&etag_from_file, file)) &&
etag_from_file) {
header = aprintf("If-None-Match: \"%s\"", etag_from_file);
Curl_safefree(etag_from_file);
}
else
header = aprintf("If-None-Match: \"\"");
if(!header) {
if(file)
fclose(file);
errorf(config->global,
"Failed to allocate memory for custom etag header\n");
result = CURLE_OUT_OF_MEMORY;
break;
}
/* add Etag from file to list of custom headers */
add2list(&config->headers, header);
Curl_safefree(header);
if(file) {
fclose(file);
}
}
).
This results into creating test.etag if it does not exists or truncating test.etag if it exists.
Truncating test.etag then make the --etag-compare option useless.

I believe this behavior is a bug because it is different to the behavior described here: https://daniel.haxx.se/blog/2019/12/06/curl-speaks-etag/ where this example is given:

curl --etag-compare etag.txt --etag-save etag.txt https://example.com -o saved-file

I would like to propose a pull-request where --etag-compare is processed before --etag-save which will results in the expected behavior.

I suppose curl should not fail if --etag-compare test.etag is specified but test.etag does not exist and --etag-save is specified.

Thank you for this awesome tool!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

No branches or pull requests

2 participants