add kube client disk cache

[billiford]

The Kubernetes dynamic client relies on API discovery to know the API group and version of a given Kubernetes Kind. Until now, we've been declaring a new in memory cache client for all of our needs. This is fine for things like Apply as the discovery will only happen once before applying all manifests, and any manifest that gets applied will reference this in memory discovery cache. However, this becomes a problem when Spinnaker is requesting information for manifests or trying to list resources (for the infrastructure page) frequently. We see a lot of requests against the cluster take place - and very frequently. These requests are small, but I wanted our go-client to utilize the same efficiency as kubectl does with its disk cache without having to shell out.

I added logs to the client-go package at https://github.com/kubernetes/client-go/blob/release-1.19/rest/request.go#L954 to log all of these discovery requests to get an idea of the overhead we're looking at.

Here's an example:

[GIN] 2020/10/12 - 15:48:56 | 200 |   59.555262ms |       10.8.5.18 | GET      "/applications/cd-go-clouddriver/loadBalancers"

This is a request to get all load balancers, which are kinds services and ingresses in Kubernetes. Here are all the underlying requests that our kube go client is making to the server:

2020/10/12 15:48:56 Req: ~ /apis
2020/10/12 15:48:56 Req: ~ /apis/metrics.k8s.io/v1beta1
2020/10/12 15:48:56 Req: ~ /apis/apps/v1beta2
2020/10/12 15:48:56 Req: ~ /api/v1
2020/10/12 15:48:56 Req: ~ /apis/authorization.k8s.io/v1
2020/10/12 15:48:56 Req: ~ /apis/apiregistration.k8s.io/v1
2020/10/12 15:48:56 Req: ~ /apis/apps/v1
2020/10/12 15:48:56 Req: ~ /apis/authentication.k8s.io/v1
2020/10/12 15:48:56 Req: ~ /apis/apiregistration.k8s.io/v1beta1
2020/10/12 15:48:56 Req: ~ /apis/authentication.k8s.io/v1beta1
2020/10/12 15:48:56 Req: ~ /apis/scheduling.k8s.io/v1
2020/10/12 15:48:56 Req: ~ /apis/scheduling.k8s.io/v1beta1
2020/10/12 15:48:56 Req: ~ /apis/apps/v1beta1
2020/10/12 15:48:56 Req: ~ /apis/metrics.k8s.io/v1beta1
2020/10/12 15:48:56 Req: ~ /apis/authorization.k8s.io/v1beta1
2020/10/12 15:48:56 Req: ~ /apis/coordination.k8s.io/v1
2020/10/12 15:48:56 Req: ~ /apis/autoscaling/v1
2020/10/12 15:48:56 Req: ~ /apis/autoscaling/v2beta1
2020/10/12 15:48:56 Req: ~ /apis/node.k8s.io/v1beta1
2020/10/12 15:48:56 Req: ~ /apis/batch/v1beta1
2020/10/12 15:48:56 Req: ~ /apis/nodemanagement.gke.io/v1alpha1
2020/10/12 15:48:56 Req: ~ /apis/monitoring.coreos.com/v1
2020/10/12 15:48:56 Req: ~ /apis/coordination.k8s.io/v1beta1
2020/10/12 15:48:56 Req: ~ /apis/autoscaling.k8s.io/v1
2020/10/12 15:48:56 Req: ~ /apis/batch/v1
2020/10/12 15:48:56 Req: ~ /apis/networking.k8s.io/v1beta1
2020/10/12 15:48:56 Req: ~ /apis/certificates.k8s.io/v1beta1
2020/10/12 15:48:56 Req: ~ /apis/apiextensions.k8s.io/v1beta1
2020/10/12 15:48:56 Req: ~ /apis/admissionregistration.k8s.io/v1beta1
2020/10/12 15:48:56 Req: ~ /apis/networking.gke.io/v1beta2
2020/10/12 15:48:56 Req: ~ /apis/spinnaker.io/v1alpha2
2020/10/12 15:48:56 Req: ~ /apis/policy/v1beta1
2020/10/12 15:48:56 Req: ~ /apis/autoscaling.k8s.io/v1beta2
2020/10/12 15:48:56 Req: ~ /apis/networking.k8s.io/v1
2020/10/12 15:48:56 Req: ~ /apis/extensions/v1beta1
2020/10/12 15:48:56 Req: ~ /apis/networking.gke.io/v1beta1
2020/10/12 15:48:56 Req: ~ /apis/scalingpolicy.kope.io/v1alpha1
2020/10/12 15:48:56 Req: ~ /apis/spinnaker.armory.io/v1alpha2
2020/10/12 15:48:56 Req: ~ /apis/cloud.google.com/v1beta1
2020/10/12 15:48:56 Req: ~ /apis/storage.k8s.io/v1
2020/10/12 15:48:56 Req: ~ /apis/rbac.authorization.k8s.io/v1
2020/10/12 15:48:56 Req: ~ /apis/storage.k8s.io/v1beta1
2020/10/12 15:48:56 Req: ~ /apis/internal.autoscaling.k8s.io/v1alpha1
2020/10/12 15:48:56 Req: ~ /apis/rbac.authorization.k8s.io/v1beta1
2020/10/12 15:48:56 Req: ~ /api/v1/services
2020/10/12 15:48:56 Req: ~ /apis/extensions/v1beta1/ingresses

All but the last two requests are from the API discovery! Let's compare that to when we use a disk cache.

2020/10/12 16:14:50 Req: ~ /api/v1/services
2020/10/12 16:14:50 Req: ~ /apis/extensions/v1beta1/ingresses

Since the API discovery cache is already stored on disk, we don't need to make the several requests to figure out which endpoint to call to get our services and ingresses.

A few notes:

• We're using the same ttl for the cache (10 minutes) as kubectl
• The default cache directory is /var/kube/cache
• We can use either an emptyDir volume or persistent volume to store this cache. I think we should go with a persistent volume so that we can scale clouddriver and all instances will reference the same on disk cache.
• Checking the on disk cache in OSS Clouddriver I doubt this directory will take up more than a few gigs, even for more than 1000 clusters.
• Instead of using kubernetes cached disk, I copied it over and commented out a few lines that have given us trouble in the past.

[dmrogers7]

Looks good