Recommended liveness check for kube-apiserver

[justinsb]

For 1.6, what is the healthz endpoint / recommended liveness check for a secure setup (RBAC, kubeadm discovery not enabled, insecure port disabled)?

curl https://127.0.0.1/healthz is returning a 401 for me.

Edit: What I originally called "kubeadm discovery not enabled" is now more commonly / correctly called "setting --anonymous-auth=false on apiserver".

[liggitt]

You can include credentials in the liveness check or allow unauthenticated requests (until a healthz-only option is available)

[justinsb]

And I came so close! Probably going to enable the insecure port, for the time being at least, in kops.

Is there an effort to implement an unprotected healthz endpoint? Can I help?

[liggitt]

@justinsb are you setting --anonymous-auth=false or leaving the authorization mode as AlwaysAllow?

[justinsb]

We always set anonymous-auth=false. We currently have two authorization modes - AlwaysAllow & RBAC; no hybrids supported yet.

[liggitt]

We always set anonymous-auth=false

is this a holdover from 1.5.0? From 1.5.1 on, the defaults are reasonable (anonymous auth is disabled by default in 1.5.x, and only enabled in 1.6 when using authorizers that require explicit ACL grants to anonymous users)

[justinsb]

It's a safety gate, yes. We never want anonymous authn to the API, even if authz should catch it it's too easy to make a mistake.

[liggitt]

it's certainly up to the deployer, but I expect that to cause difficulties as auth discovery gets built out (kubectl login asks where it should go to log in, etc).

[adamhaney]

kops v1.6.0 references this issue

W0528 11:00:47.597850 9970 apiserver.go:90] Enabling apiserver insecure port, for healthchecks (issue #43784)

But it's not clear what if any action I should take as a part of my configuration. Is that line kops telling me that they're using a workaround or that I need to change my configuration?

[liggitt]

There are currently two ways to allow an unauthenticated health check:

1. enable the unsecured port, which lets anyone who can reach it be a superuser
2. enable anonymous requests to the TLS port and depend on authorization to limit which endpoints anonymous requests are allowed to reach

kops disables anonymous requests to the TLS port, so it must enable the unsecured port if it wants to make anonymous health checks

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle rotten
/remove-lifecycle stale

[dghubble]

I sympathize with the kops point. Neither of these options seem particularly safe. Enabling insecure port isn't viable and anon seems to have its own subtleties and risks for mistakes. https://gist.github.com/erictune/9dc7ae4b22505b9a8c20ad9cd03a45cc

[justinsb]

/remove-lifecycle rotten

[liggitt]

anon seems to have its own subtleties and risks for mistakes. https://gist.github.com/erictune/9dc7ae4b22505b9a8c20ad9cd03a45cc

that was from the 1.5 timeframe, when that option was added. by 1.6, all authorizers were updated to require explicitly granting permissions to anonymous requests. the cautions about granting permissions to * and having those include anonymous users no longer apply.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[cheftako]

No. I tested on a 1.17 build. I just checked the manifest for the kube-paiserver and the anonymous-auth flag was not present. I checked the logs and it looks like the flag defaults to "--anonymous-auth=true" I did have to set a "-k" on the curl, so it would ignore the certificate issue.

[justinsb]

anonymous-auth=false is the key bit of this issue - I'll update the original issue description to clarify that what I called kubeadm discovery == anonymous-auth.

[msschl]

This is still an issue on v1.17.3. Are there any plans e.g. a separate port for health checks?

[SergeyMuha]

As a workaround I did
TOKEN=$(kubectl get secret $SECRET_NAME -o jsonpath='{.data.token}' | base64 --decode)
and then added in /etc/kubernetes/manifests/kube-apiserver.yaml

    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: xxxx
        path: /healthz
        port: 6443
        scheme: HTTPS
        httpHeaders:
          - name: Authorization
            value: Bearer TOKEN
      initialDelaySeconds: 15
      timeoutSeconds: 15

So that kubelet are able to check

[liuyuan10]

A qq if anyone can help to answer. Even with default setup (--anonymous-auth=true), I'm seeing 403 code:

curl -k https://127.0.0.1:6444/healthz -I
HTTP/2 403
cache-control: no-cache, private
content-type: application/json
x-content-type-options: nosniff
content-length: 241
date: Thu, 24 Sep 2020 00:14:15 GMT

But actually the payload is valid:

curl -k https://127.0.0.1:6444/healthz
ok

I can see public-info-viewer binded to system:unauthenticated. But do we expect it always return 403? matching payloads are less ideal for a healthcheck

This k8s v1.17

[liuyuan10]

oh, nvm. I found curl -I is using http head which is forbiden by k8s. For anyone that gets confused as me, this is what you want:

curl -k https://127.0.0.1:6444/version  -I -X GET

[erezo9]

As a workaround I did
TOKEN=$(kubectl get secret $SECRET_NAME -o jsonpath='{.data.token}' | base64 --decode)
and then added in /etc/kubernetes/manifests/kube-apiserver.yaml

    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: xxxx
        path: /healthz
        port: 6443
        scheme: HTTPS
        httpHeaders:
          - name: Authorization
            value: Bearer TOKEN
      initialDelaySeconds: 15
      timeoutSeconds: 15

So that kubelet are able to check

Hi, Is there anyway to add this to kubeadm-config? so that every upgrade you dont need to change the kube-api yaml?

[marshallford]

I'm running into the "accessible health checks" issue while using RKE2, a distribution that has a goal/feature of passing the CIS k8s benchmark. The CIS control for anonymous auth doesn't take into consideration the usage of the AlwaysAllow authorizer (as mentioned in related GH issues).

[leoryu]

As a workaround I did
TOKEN=$(kubectl get secret $SECRET_NAME -o jsonpath='{.data.token}' | base64 --decode)
and then added in /etc/kubernetes/manifests/kube-apiserver.yaml

    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: xxxx
        path: /healthz
        port: 6443
        scheme: HTTPS
        httpHeaders:
          - name: Authorization
            value: Bearer TOKEN
      initialDelaySeconds: 15
      timeoutSeconds: 15

So that kubelet are able to check

Hi, Is there anyway to add this to kubeadm-config? so that every upgrade you dont need to change the kube-api yaml?

Any ideas about adding to kubeadm-config?

[nugasescuserbun]

As a workaround I did
TOKEN=$(kubectl get secret $SECRET_NAME -o jsonpath='{.data.token}' | base64 --decode)
and then added in /etc/kubernetes/manifests/kube-apiserver.yaml

What happens when the token expires though? Do you refresh the token out of band?

[fert-f]

I have an ugly workaround for kube-apiserver health check that is based on tcpSocket probes. Here is a script that rewrites the manifest (yq as dependency, but can be replaced):

sudo yq -i 'with(.spec.containers[];
    with(.startupProbe;
      del(.httpGet)|
      .tcpSocket.port = 6443)|
    with(.readinessProbe;
      del(.httpGet)|
      .tcpSocket.port = 6443)|
    with(.livenessProbe;
      del(.httpGet)|
      .tcpSocket.port = 6443))' \
      -i /etc/kubernetes/manifests/kube-apiserver.yaml

[sathieu]

This issue is tagged api-machinery, but maybe it should be tagged to be handled on the kubeadm side instead?

I won't ping @kubernetes/sig-cluster-lifecycle (to avoid spamming), but I hope someone will see this...

[Pishniy]

Hi! Issue still exist in 1.28.2. When set in /etc/kubernetes/manifests/kube-apiserver.yaml --anonymous-auth=false api-server probes fails with 401 code.

My workaround was:

1. Create ServiceAccount in NS kube-system

2. Create secret for ServiseAccount in NS kube-system

3. kubectl get secret SecretName -o jsonpath='{.data.token}' -n kube-system | base64 --decode

4. And then add this token into each probe into /etc/kubernetes/manifests/kube-apiserver.yaml :

readinessProbe:
      failureThreshold: 3
      httpGet:
        host: 192.168.122.126
        path: /readyz
        port: 6443
        scheme: HTTPS
        httpHeaders:
         - name: Authorization
           value: Bearer <TOKEN>

Only after this api-server up and probes finished with success.

Maybe will be useful for someone.