Separate volume or EmptyDir for dind

[minrk]

Now that we are back to having long-lived nodes, our dind image caches are filling up. Our image-cleaning script seems to cause various problems, and is disabled right now (I just re-enabled it for dind-only in #753). If we used EmptyDir for the dind volume instead of /var/lib/dind, then deleting the dind pod would immediately empty;

Advantages of using EmptyDir volume instead:

• it's very easy/fast to delete the build cache (delete the dind pod)

Disadvantages:

• updates to the dind deployment always clear the dind image cache, causing an immediate and significant slowdown across the cluster for builds after any upgrade that touches the dind configuration.

[minrk]

An alternative here is to put dind storage on a different volume. If we could do this, then I think the separate image cleaning would play just fine together. I don't know how to get each dind pod to have its own volume, though, since each pod would need its own state, and the current number of dind pods is variable (one per node, since it's a daemonset)

[minrk]

I think we can use local ssds to put dind on its own disk.

This will require:

1. allocate a new node pool, with: gcloud beta container node-pools create ... --local-ssd-count 1
2. drain the old node pool
3. update config/prod.yaml with dind.hostLibDir: /mnt/ssd0/dind

That should alleviate all of our disk-full woes, because dind and host quota/gc behavior will be using different disks, and thus shouldn't conflict with each other.

I'd like to schedule this to deploy tomorrow, since alternating full-disk and image-cleaner issues have been an ongoing low-level headache.