Add option to toggle "skip_kubelet_verification" flag for k8s workload attestor

[LaithLite]

The Spire-Agent sub-chart currently has the hard-coded line skip_kubelet_verification = true for its k8s workload attestor.

[faisal-memon]

what kind of additional config do we need to verify kubelet? Ive never run with that setting.

[kfox1111]

I think the plugin config stuff we've been talking about could handle this override? will likely need other config as well to work such as including the ca that signed the kubelets.

[azdagron]

I personally feel like skipping kubelet verification is the wrong default. As a project, we should aim for secure-by-default. If someone needs to disable that (because they are running in minikube) then they should be the special case.

[kfox1111]

@azdagron I agree, except that there has been an open issue about it in kubeadm for ages.It is the norm, not the exception. :/
kubernetes/kubeadm#1223

Honestly, I'd like to see the spire team work with the kubelet team and add spire support to kubelet, so that kubelet server/client certs are from spire itself. This would go a very long way to solve a very old Kubernetes problem.

[faisal-memon]

I personally feel like skipping kubelet verification is the wrong default. As a project, we should aim for secure-by-default. If someone needs to disable that (because they are running in minikube) then they should be the special case.

Valid point. But still the first interaction most people will have with this project is through minikube or docker desktop. Id like it to work out of the gate on those platforms to ensure a good user experience. And then have guides for more production scale deployments.

[kfox1111]

I think it might be good to add some warnings in the generated notes if the value is true though? Lets the user know there may be a problem. They can then choose to ignore it if they think it is not a problem, or they will know it needs fixing?

[azdagron]

Is there a "so you want to run this in production? here's what to consider" checklist anywhere? I'm with @kfox1111, the louder we can be about this kind of stuff, the better.

[kfox1111]

So far, we've been putting all the recommended settings in examples/production/values.yaml that the user can directly include. This is one of the first that we can't really do a setting but need the user to consider some things.

So, would the checklist being in the generated NOTES work better, as it can be generated to include/exclude things based on the users actual specified values, or should it be outside of the chart where it may get a bit more visibility?

[kfox1111]

I could take a stab at a NOTES patch, but I think it probably should depend on #166. Would greatly simplify the checks if it could verify global.spire.profile[x].production was specified.