Use both --etag-compare and --etag-save with the same etag file name

[kwon-young]

I did this

On the first run where test.etag and test.ext does not exist:

$ curl --etag-compare test.etag --etag-save test.etag -o test.ext -v 'http://myurl.com/test.ext'

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying ip:80...
* Connected to myurl.com (ip) port 80 (#0)
> GET /test.ext HTTP/1.1
> Host: myurl.com
> User-Agent: curl/7.69.1
> Accept: */*
> If-None-Match: ""
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Fri, 03 Apr 2020 17:43:36 GMT
< Content-Type: application/ext
< Content-Length: 3840872
< Connection: keep-alive
< Set-Cookie: __cfduid=d8d26cdd5d1359a2545e22facdeb5c54f1585935816; expires=Sun, 03-May-20 17:43:36 GMT; path=/; domain=.myurl.com; HttpOnly; SameSite=Lax
< Last-Modified: Tue, 25 Feb 2014 02:26:16 GMT
< ETag: "530bff48-3a9b68"
< Cache-Control: max-age=14400
< CF-Cache-Status: REVALIDATED
< Accept-Ranges: bytes
< Server: cloudflare
< CF-RAY: 57e497c43ee9ee8d-CDG
< 
{ [967 bytes data]
100 3750k  100 3750k    0     0  2976k      0  0:00:01  0:00:01 --:--:-- 2976k
* Connection #0 to host myurl.com left intact

On the second run, test.ext is downloaded again.

$ curl --etag-compare test.etag --etag-save test.etag -o test.ext -v 'http://myurl.com/test.ext'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying ip:80...
* Connected to myurl.com (ip) port 80 (#0)
> GET /test.ext HTTP/1.1
> Host: myurl.com
> User-Agent: curl/7.69.1
> Accept: */*
> If-None-Match: ""
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Fri, 03 Apr 2020 17:46:33 GMT
< Content-Type: application/ext
< Content-Length: 3840872
< Connection: keep-alive
< Set-Cookie: __cfduid=d2c4673cee3162b5136f49cde885137ca1585935993; expires=Sun, 03-May-20 17:46:33 GMT; path=/; domain=.myurl.com; HttpOnly; SameSite=Lax
< Last-Modified: Tue, 25 Feb 2014 02:26:16 GMT
< ETag: "530bff48-3a9b68"
< Cache-Control: max-age=14400
< CF-Cache-Status: HIT
< Age: 177
< Accept-Ranges: bytes
< Server: cloudflare
< CF-RAY: 57e49c182b07a8c1-CDG
< 
{ [965 bytes data]
100 3750k  100 3750k    0     0  3015k      0  0:00:01  0:00:01 --:--:-- 3015k
* Connection #0 to host myurl.com left intact

I expected the following

The second curl command should not download text.ext again.

curl/libcurl version

curl 7.69.1 (x86_64-pc-linux-gnu) libcurl/7.69.1 OpenSSL/1.1.1e zlib/1.2.11 libidn2/2.3.0 libpsl/0.21.0 (+libidn2/2.2.0) libssh2/1.9.0 nghttp2/1.40.0
Release-Date: 2020-03-11
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets

operating system

Linux edwin 5.5.13-1-MANJARO #1 SMP PREEMPT Wed Mar 25 17:14:28 UTC 2020 x86_64 GNU/Linux

Discussion

After going through the pull-request #4678 that implemented using both --etag-compare and --etag-save, I understood that the --etag-save is processed first and open for writing test.etag (see

curl/src/tool_operate.c

Lines 908 to 976 in 57476a9

	/* --etag-save */
	etag_save = &per->etag_save;
	etag_save->stream = stdout;
	if(config->etag_save_file) {
	/* open file for output: */
	if(strcmp(config->etag_save_file, "-")) {
	FILE *newfile = fopen(config->etag_save_file, "wb");
	if(!newfile) {
	warnf(
	config->global,
	"Failed to open %s\n", config->etag_save_file);
	result = CURLE_WRITE_ERROR;
	break;
	}
	else {
	etag_save->filename = config->etag_save_file;
	etag_save->s_isreg = TRUE;
	etag_save->fopened = TRUE;
	etag_save->stream = newfile;
	}
	}
	else {
	/* always use binary mode for protocol header output */
	set_binmode(etag_save->stream);
	}
	}
	/* --etag-compare */
	if(config->etag_compare_file) {
	char *etag_from_file = NULL;
	char *header = NULL;
	/* open file for reading: */
	FILE *file = fopen(config->etag_compare_file, FOPEN_READTEXT);
	if(!file) {
	errorf(config->global,
	"Failed to open %s\n", config->etag_compare_file);
	result = CURLE_READ_ERROR;
	break;
	}
	if((PARAM_OK == file2string(&etag_from_file, file)) &&
	etag_from_file) {
	header = aprintf("If-None-Match: \"%s\"", etag_from_file);
	Curl_safefree(etag_from_file);
	}
	else
	header = aprintf("If-None-Match: \"\"");
	if(!header) {
	if(file)
	fclose(file);
	errorf(config->global,
	"Failed to allocate memory for custom etag header\n");
	result = CURLE_OUT_OF_MEMORY;
	break;
	}
	/* add Etag from file to list of custom headers */
	add2list(&config->headers, header);
	Curl_safefree(header);
	if(file) {
	fclose(file);
	}
	}

).
This results into creating test.etag if it does not exists or truncating test.etag if it exists.
Truncating test.etag then make the --etag-compare option useless.

I believe this behavior is a bug because it is different to the behavior described here: https://daniel.haxx.se/blog/2019/12/06/curl-speaks-etag/ where this example is given:

curl --etag-compare etag.txt --etag-save etag.txt https://example.com -o saved-file

I would like to propose a pull-request where --etag-compare is processed before --etag-save which will results in the expected behavior.

I suppose curl should not fail if --etag-compare test.etag is specified but test.etag does not exist and --etag-save is specified.

Thank you for this awesome tool!