installer/pkg/config/libvirt: Caching for libvirt pulls

[wking]

Checking our RHCOS source against ETag / If-None-Match and Last-Modified / If-Modified-Since, it seems to support In-None-Match well, but only supports If-Modified-Since for exact matches:

$ URL=http://aos-ostree.rhev-ci-vms.eng.rdu2.redhat.com/rhcos/images/cloud/latest/rhcos-qemu.qcow2.gz
$ curl -I "${URL}"
HTTP/1.1 200 OK
Server: nginx/1.8.0
Date: Wed, 19 Sep 2018 04:32:19 GMT
Content-Type: application/octet-stream
Content-Length: 684934062
Last-Modified: Tue, 18 Sep 2018 20:05:24 GMT
Connection: keep-alive
ETag: "5ba15a84-28d343ae"
Accept-Ranges: bytes
$ curl -sIH 'If-None-Match: "5ba15a84-28d343ae"' "${URL}" | head -n1
HTTP/1.1 304 Not Modified
$ curl -sIH 'If-Modified-Since: Tue, 18 Sep 2018 20:05:24 GMT' "${URL}" | head -n1
HTTP/1.1 304 Not Modified
$ curl -sIH 'If-Modified-Since: Tue, 18 Sep 2015 20:05:24 GMT' "${URL}" | head -n1
HTTP/1.1 200 OK
$ curl -sIH 'If-Modified-Since: Tue, 18 Sep 2018 20:05:25 GMT' "${URL}" | grep 'HTTP\|Last-Modified'
HTTP/1.1 200 OK
Last-Modified: Tue, 18 Sep 2018 20:05:24 GMT

That last entry should have 304ed, although the spec has:

When used for cache updates, a cache will typically use the value of the cached message's Last-Modified field to generate the field value of If-Modified-Since. This behavior is most interoperable for cases where clocks are poorly synchronized or when the server has chosen to only honor exact timestamp matches (due to a problem with Last-Modified dates that appear to go "back in time" when the origin server's clock is corrected or a representation is restored from an archived backup)...

So the server is violating the SHOULD by not 304ing dates greater than Last-Modified, but it's not violating a MUST-level requirement. The server requirements around If-None-Match are MUST-level, so using it should be more portable. The RFC also seems to prefer clients use If(-None)-Match.

I'm using gregjones/httpcache for the caching, since that implemenation seems reasonably popular and the repo's been around for a few years. That library uses the belt-and-suspenders approach of setting both If-None-Match (to the cached ETag) and If-Modified-Since (to the cached Last-Modified), so we should be fine.

UserCacheDir requires Go 1.11.

No unit tests, but you can excercise the logic with:

$ go1.11 build ./installer/cmd/tectonic
$ cat libvirt.yaml 
admin:
  email: a@b.c
  password: verysecure
  sshKey: "ssh-rsa AAAA..."
baseDomain: installer.testing
name: wking
platform: libvirt
libvirt:
  uri: qemu+tcp://192.168.122.1/system
  network:
    name: tectonic
    ifName: tt0
    dnsServer: 8.8.8.8
    ipRange: 192.168.124.0/24
  image: http://aos-ostree.rhev-ci-vms.eng.rdu2.redhat.com/rhcos/images/cloud/latest/rhcos-qemu.qcow2.gz
pullSecretPath: /home/trking/src/openshift/installer/pull-secret.json
master:
  nodePools:
    - master
worker:
  nodePools:
    - worker
nodePools:
  - count: 1
    name: master
  - count: 2
    name: worker
$ rm -rf wking && ./tectonic init --config=libvirt.yaml && cat wking/config.yaml  # slow....
...
  image: file:///tmp/openshift-install-092942808
...
$ rm -rf wking && ./tectonic init --config=libvirt.yaml && cat wking/config.yaml  # fast :)
...
  image: file:///tmp/openshift-install-567329896
...
$ sha1sum /tmp/openshift-install-*
d4554ab97c91f1d807fafab0c81b5492a98219a8  /tmp/openshift-install-092942808
d4554ab97c91f1d807fafab0c81b5492a98219a8  /tmp/openshift-install-567329896
$ tree -a ~/.cache/openshift-install/
/home/trking/.cache/openshift-install/
└── libvirt
    └── 914a1ca2be4f5adecaea8ceda514b02e

1 directory, 1 file

[wking]

The errors are because we aren't using Go 1.11 yet, so we get complaints about os.UserCacheDir being undefined. Can we bump our Go (like we bumped to 1.10 in openshift/release#1132)? Should I create a dummy shim for earlier Gos? How dumb can my shim be?

[wking]

I also haven't bothered vendoring httpcache until folks have time to look this over and review the general direction. I don't think our CI will green out without the vendor bump, but just in case, I'll drop a...

/hold

[crawford]

Is this supposed to be in here?

[wking]

Is this supposed to be in here?

Yeah. We used to just copy the input config over. Now that I'm saving the updates config (because it might contain a new libvirt image URI), I need to clear PullSecretPath (after reading the content into PullSecret) to avoid hitting this when we reload the config for the install step.

[wking]

Rebased around #273 with 04d2575 -> 5f705cf.

[wking]

Updated to unzip .gz URIs for you.

[crawford]

/lgtm

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: crawford, wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [crawford,wking]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[crawford]

/hold cancel