CISCO IronPort breaks If-Modified-Since check, causing refetch every time

[bdrewery]

The check added in #425 does not work behind CISCO IronPort proxies.

Running curl -I http://pkg0.nyi.freebsd.org/freebsd:10:x86:64/latest/digests.txz behind the proxy and outside the proxy reveals:

No proxy

Last-Modified: Thu, 06 Mar 2014 23:32:49 GMT

Proxy

Last-Modified: Thu, 06 Mar 2014 23:32:49 UTC

The standard (http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html) does state, "All HTTP date/time stamps MUST be represented in Greenwich Mean Time (GMT), without exception. For the purposes of HTTP, GMT is exactly equal to UTC (Coordinated Universal Time)."

libfetch(3) ignored any Last-Modified not having GMT:

r  = strptime(p, "%a, %d %b %Y %H:%M:%S GMT", &tm);

We could fix libfetch(3) to also accept UTC.

I would prefer a fix that works on all releases though. Perhaps bundling libfetch, or using another mechanism to check for changes.

It is also too bad that libfetch(3) does not support ETAG / If-None-Match, as that does work properly behind the proxy:

# curl -I http://pkg0.nyi.freebsd.org/freebsd:10:x86:64/latest/digests.txz|grep ETag
ETag: "531905a1-10c530"
# curl --header 'If-None-Match: "531905a1-10c530"' http://pkg0.nyi.freebsd.org/freebsd:10:x86:64/latest/digests.txz -o digests.txz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
# curl --header 'If-None-Match: "CHANGED531905a1-10c530"' http://pkg0.nyi.freebsd.org/freebsd:10:x86:64/latest/digests.txz -o digests.txz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1073k  100 1073k    0     0   682k      0  0:00:01  0:00:01 --:--:--  681k

[bdrewery]

I am submitting a libfetch(3) fix for this for 10.1,9.3 at least.

[Jamie-Landeg-Jones]

I think that's the best solution, even though it's a fudge around
someone elses brokenness...

sigh... In a perfect world...

What's the old saying regarding specs/standards? "Be conservative
in what you send, but liberal in what you accept".. or something!

[bdrewery]

The Cisco IornPort also impacted wget https://savannah.gnu.org/bugs/?40746 (debian's discussion at https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=730662)

[bdrewery]

Fixed in FreeBSD libfetch head for 10.1/9.3.

[bdrewery]

Closing as there is not much we can do without embedding libfetch.