http: add support to read and store the referrer header

[vszakats]

This patch implements:

• CURLINFO_REFERER libcurl option
• --write-out '%{referer}' command-line option
• --xattr to fill user.xdg.referrer.url extended attribute with the referrer. This extended attribute is supported by wget and the referrer value is also stored by at least Safari (albeit in a platform-specific format.)

This can also be useful to retain a meaningful download URL where the initial URL is being redirected to a temporary one, e.g. with GitHub releases:

$ curl --xattr --location --referer ';auto' --remote-name \
  https://github.com/curl/curl/releases/download/curl-7_75_0/curl-7.75.0.tar.xz
$ xattr -l curl-7.75.0.tar.xz
[...]
user.xdg.referrer.url: https://github.com/curl/curl/releases/download/curl-7_75_0/curl-7.75.0.tar.xz

Implemented:

• rename "referrer" to "referer" to match existing names
• test examples
• delete "URL"
• possibly add a regression test

Any comments/suggestions are welcome.

[vszakats]

Any opinions if using "URL" in this context is correct or should we just use 'referer'? [YES] After all the referrer field can contain any string, not just a URL.

Next question, if stripcredentials() should in this case be applied to the referer string before saving it to the user.xdg.referrer.url file attribute (or should this be done by the ;auto logic for data->set.http_auto_referer?). [NO]

[bagder]

Any opinions if using "URL" in this context is correct or should we just use 'referer'?

I vote for just referer.

Next question, if stripcredentials() should in this case be applied to the referer string before saving it to the user.xdg.referrer.url file attribute (or should this be done by the ;auto logic for data->set.http_auto_referer?).

Hm. That's a good point. I think we should make sure any credentials get stripped off already in the internal logic.

[vszakats]

Still some failures here and there, but couldn't tell conclusively that they have something to do with the changes in this PR:
1206 (2x), 1503, 579, 1188, some torture tests.

[jay]

i've noticed seemingly random failures in the torture tests recently i'm not sure what that's about i'll take a look

[bagder]

I've seen test 2100 fail in torture tests recently but when I've tried to reproduce locally I have not managed ... 😞

[jay]

I found one problem in ASAN, exit on leak disrupts closing the memdebug file so it reports wrong results. Reported as google/sanitizers#1374. Could be solved in curl with an atexit to fclose the file. or with MEMDEBUG_LOG_SYNC but IMO we should close it explicitly. I'm not sure if valgrind does the same type of abort when a leak is detected.

Second problem (an actual leak) has something to do with the proxy, all the arbitrary failed torture tests are proxy I noticed. 264 335 552 try -t on any of those. If I had to guess it's 46620b9 since that is a close proxy one and also I randomly picked the CI of some earlier commits and don't see anything similar. 264 I can reproduce consistently, and so far I've traced it back to this call returning OOM, which causes something to be missed at cleanup:

curl/lib/url.c

Lines 2411 to 2412 in 94719e7

	result = Curl_setstropt(&data->state.aptr.proxyuser,
	proxyuser);

** MEMORY FAILURE
Leak detected: memory still allocated: 5 bytes
At 603000022a28, there's 5 bytes.
 allocated by escape.c:159
LIMIT setopt.c:66 strdup reached memlimit

 ==4694==ERROR: LeakSanitizer: detected memory leaks
 Direct leak of 21 byte(s) in 1 object(s) allocated from:
     #0 0x7fbcf6087602 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x98602)
     #1 0x7fbcf590374a in curl_dbg_malloc /home/owner/curl/lib/memdebug.c:126
     #2 0x7fbcf58798d3 in Curl_urldecode /home/owner/curl/lib/escape.c:159
     #3 0x7fbcf59fd1c6 in curl_url_get /home/owner/curl/lib/urlapi.c:1198
     #4 0x7fbcf59e418c in parse_proxy /home/owner/curl/lib/url.c:2417
     #5 0x7fbcf59e5795 in create_conn_helper_init_proxy /home/owner/curl/lib/url.c:2612
     #6 0x7fbcf59eecf6 in create_conn /home/owner/curl/lib/url.c:3577
     #7 0x7fbcf59f41b3 in Curl_connect /home/owner/curl/lib/url.c:4063
     #8 0x7fbcf5928b83 in multi_runsingle /home/owner/curl/lib/multi.c:1672
     #9 0x7fbcf592d467 in curl_multi_perform /home/owner/curl/lib/multi.c:2414
     #10 0x7fbcf5874a12 in easy_transfer /home/owner/curl/lib/easy.c:607
     #11 0x7fbcf587516d in easy_perform /home/owner/curl/lib/easy.c:698
     #12 0x7fbcf587528c in curl_easy_perform /home/owner/curl/lib/easy.c:718
     #13 0x451447 in serial_transfers /home/owner/curl/src/tool_operate.c:2348
     #14 0x45267b in run_all_transfers /home/owner/curl/src/tool_operate.c:2523
     #15 0x452eef in operate /home/owner/curl/src/tool_operate.c:2640
     #16 0x436870 in main /home/owner/curl/src/tool_main.c:279
     #17 0x7fbcf40c983f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2083f)
 SUMMARY: AddressSanitizer: 21 byte(s) leaked in 1 allocation(s).

edit: ignore the line numbers I sprinkled in a lot of printfs..

[vszakats]

@jay: Thank you for investigating!

Any thoughts if we should make this PR pending till these get resolved, or do you think this looks fine enough to merge now?

[bagder]

The 264 problem should hopefully be fixed with #6614...

[vszakats]

One permanent-looking CI issue is that the Cirrus task for FreeBSD 12.1 fails at pkginstall:

pkg update -f
Updating FreeBSD repository catalogue...
Fetching meta.txz: . done
Fetching packagesite.txz: .......... done
Processing entries: 
Newer FreeBSD version for package py37-mlt:
To ignore this error set IGNORE_OSVERSION=yes
- package: 1202000
- running kernel: 1201000
Ignore the mismatch and continue? [Y/n]: pkg: repository FreeBSD contains packages for wrong OS version: FreeBSD:12:amd64
Processing entries... done
Unable to update repository FreeBSD
Error updating repositories!

Opened a PR that deletes CI builds with this image, following the suggestion of FreeBSD forum thread.

[jay]

I'd stick to referer spelling (one r) throughout so it's less confusing

[vszakats]

My rule of thumb was to use the rr flavour in human sentences and the other one in code. (this also kind-of matches existing use). In this particular case we can work it around by going The Referer: header, if there was any. (Added in 7.76.0). This style can be found in CURLOPT_AUTOREFERER and CURLOPT_REFERER-related docs.

Does that sound OK?

[jay]

Both spellings seem to be used, see --referer using referrer for example. Personally I find it confusing but there seems to be some precedent for it.

[vszakats]

Yes, it isn't used fully consistently as it is. I've pushed an update for this specific line here, which I'm hoping makes it look better.

[vszakats]

Much better CI results so far. There is one test consistently failing for c-ares builds:

test 1188...[--write-out with %{onerror} and %{urlnum} to stderr]

 1188: stderr FAILED:
--- log/check-expected	2021-02-18 12:53:57.308870137 +0000
+++ log/check-generated	2021-02-18 12:53:57.308870137 +0000
@@ -1 +1 @@
-0 says 6 Could not resolve host: non-existing-host.haxx.se[LF]
+0 says 6 Could not resolve: non-existing-host.haxx.se (Domain name not found)[LF]
== Contents of files in the log/ dir after test 1188

[bagder]

Yeah, my fix for 1188 was not good enough. I landed #6626 since then should fix that too.

[vszakats]

Remaining failures:

• msys1_mingw_debug_openssl: test595 (FTP)
• msys2_mingw32_debug_schannel: timed out possibly?

It seems unlikely these have something to do with this patch, and they were happening quite randomly in the past few days for all CI passes. msys1_mingw_debug_openssl tended to fail on different, single tests in this period.

[jay]

It seems unlikely these have something to do with this patch, and they were happening quite randomly in the past few days for all CI passes. msys1_mingw_debug_openssl tended to fail on different, single tests in this period.

random ci weirdness. i restarted them and they passed