Liveness probes for kube-apiserver pod are failing with --anonymous-auth=false in place

[Evalle]

What keywords did you search in kubeadm issues before filing this one?

apiserver, anonymous-auth

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

kubeadm version

kubeadm version: &version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.4", GitCommit:"9befc2b8928a9426501d3bf62f72849d5cbcd5a3", GitTreeState:"clean", BuildDate:"2017-11-20T05:17:43Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version):

kubectl version
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.4", GitCommit:"9befc2b8928a9426501d3bf62f72849d5cbcd5a3", GitTreeState:"clean", BuildDate:"2017-11-20T05:28:34Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.4", GitCommit:"9befc2b8928a9426501d3bf62f72849d5cbcd5a3", GitTreeState:"clean", BuildDate:"2017-11-20T05:17:43Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

• OS:

[root@evgeny-k8s-master02:~] cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

• Kernel:

Linux 3.10.0-514.6.2.el7.x86_64 #1 SMP Thu Feb 23 03:04:39 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

• Others:

What happened?

When I activated the option --anonymous-auth=false in kube-apiserver.yaml kubelet started to kill the apiserver pod over and over again because the liveness probes were unsuccessful. I can workaround this issue with insucure-port and insecure-bind-address options and make liveness probes ask on this insecure address and port but as of Kubernetes 1.10, the insecure flags will be deprecated: kubernetes/kubernetes#59018
Currently, there is no other way to allow unauthenticated health checks (requests on kube-apiserver's /healthz endpoint) other than allowing anonymous requests (which we do not want). Related issue: kubernetes/kubernetes#43784. Is there something I'm missing?

What you expected to happen?

apiserver works fine with --anonymous-auth=falseoption.

How to reproduce it (as minimally and precisely as possible)?

Just add --anonymous-auth=false option to kube-apiserver.yaml

[yagonobre]

@timothysc can I work on that?

[Evalle]

@yagonobre I don't know about @timothysc but I would be happy if someone will take a look at this issue :). I'm happy to help you as much as I can.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[neolit123]

@yagonobre @Evalle have you experienced this in newer k8s/kubeadm versions?
we might have fixed this already, but i haven't checked.

[Evalle]

@neolit123 I've tested v.1.10.x a couple of minutes ago - the same issue.

[neolit123]

@Evalle we have a small team and therefore have bandwidth issues testing 1.10 near the release of 1.12.
does v1.11 or the v1.12 beta/RC work?

[neolit123]

/remove-lifecycle stale

[xmik]

I think this won't be resolved until the idea from this comment is implemented.

[timothysc]

/assign @yagonobre

[jnummelin]

This is still happening with 1.13.2. IMO kubeadm should be clever enough to use different probe if anonymous-auth is set to false.

And without setting anonymous-auth=false it's impossible to get clean papers from CIS etc. security tests. So I feel this is kinda important to fix.

[neolit123]

And without setting anonymous-auth=false it's impossible to get clean papers from CIS etc. security tests. So I feel this is kinda important to fix.

some of the k8s maintainers will probably disagree with such tests; i'm tempted to do the same.

but most importantly this is not a kubeadm problem, we just expose the apiserver flags.
see this comment and the discussion bellow for the wider problem:
kubernetes/kubernetes#51076 (comment)

i'm going to close this issue and kindly ask someone to open a new one in kubernetes/kubernetes and reference both this ticket and 51076. also tag with /sig api-machinery, /sig auth, /kind bug.

thank you.

[adamacosta]

I know this is long closed, but the CIS benchmark specifically says anonymous-auth=false isn't scored because it isn't needed if you have RBAC on. Nonetheless, if absolutely needed, you can still get the API server to come up healthy by changing the probes. RKE2 does this, using kubectl get raw on the /livez and /readyz endpoints, using the API server's own client cert to authenticate. Unfortunately, the httpGet probe doesn't have any way to provide certificates, so they have to use exec to do this.

kubeadm could support this by adding to the ControlPlaneComponent kind to allow passing elements aside from args, env vars, and mounts to the generated pod specs, specifically allowing custom probes. Right now, the only way to do this using kubeadm is to skip the phase that generates the API server spec and pre-generate it some other way with probes that work.

[neolit123]

I know this is long closed, but the CIS benchmark specifically says anonymous-auth=false isn't scored because it isn't needed if you have RBAC on. Nonetheless, if absolutely needed, you can still get the API server to come up healthy by changing the probes. RKE2 does this, using kubectl get raw on the /livez and /readyz endpoints, using the API server's own client cert to authenticate. Unfortunately, the httpGet probe doesn't have any way to provide certificates, so they have to use exec to do this.

2 days ago (6th of Dec) there was a discussion in sig-auth's meeting to turn annon-auth to false by default.
kubeadm cannot do that, but also the discussion indicated it is a very breaking change...instead the authz layer should be amended.

https://docs.google.com/document/d/1woLGRoONE3EBVx-wTb4pvp4CI7tmLZ6lS26VTbosLKM/edit#heading=h.ucn6b5acqauf

CIS benchmark is not approved by k8s maintainers, by far.
TL;DR you could make kubeadm work with turning annon auth false, but this is unsupported and you are on your own.
i've seen people use tokens for http probes, exec probes, certs instead of BS tokens, and so on.

kubeadm could support this by adding to the ControlPlaneComponent kind to allow passing elements aside from args, env vars, and mounts to the generated pod specs, specifically allowing custom probes. Right now, the only way to do this using kubeadm is to skip the phase that generates the API server spec and pre-generate it some other way with probes that work.

kubeadm already supports extraargs; extra envs are comming in v1beta4.
there are also patches, so you can customize the static pod manifests completely.

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/control-plane-flags/

[adamacosta]

I suppose patch is the real answer, the all-purpose fallback way to do whatever you want that the available configuration doesn't explicitly support, similar to Helm's post-renderers.

Defaulting to anonymous-auth=false is arguably one of the more annoying things about running RKE2 with the CIS-profile enabled. My experience has largely been with defense customers operating in either AWS GovCloud or C2S. Beyond the need for exec-based probes, this also has the consequence that a load balancer for the API server can only do TCP health checks as there is no way AWS provides to add a client cert to a load balancer.

Ultimately, I think the sanest way to do this would be put the /healthz endpoint on its own port like the other control plane components do. Anyone still concerned about the possibility of information leaking from the response could use firewall rules to only allow ingress on that port from localhost and the load balancer.