New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Recommended liveness check for kube-apiserver #43784

Open
justinsb opened this Issue Mar 29, 2017 · 19 comments

Comments

Projects
None yet
8 participants
@justinsb
Member

justinsb commented Mar 29, 2017

For 1.6, what is the healthz endpoint / recommended liveness check for a secure setup (RBAC, kubeadm discovery not enabled, insecure port disabled)?

curl https://127.0.0.1/healthz is returning a 401 for me.

justinsb added a commit to justinsb/kops that referenced this issue Mar 29, 2017

Disable liveness check for apiserver, when running secure
Temporary workaround for
kubernetes/kubernetes#43784 until we can find
a better solution.

justinsb added a commit to justinsb/kops that referenced this issue Mar 29, 2017

Disable liveness check for apiserver, when running secure
Temporary workaround for
kubernetes/kubernetes#43784 until we can find
a better solution.
@liggitt

This comment has been minimized.

Member

liggitt commented Mar 29, 2017

You can include credentials in the liveness check or allow unauthenticated requests (until a healthz-only option is available)

@justinsb

This comment has been minimized.

Member

justinsb commented Mar 29, 2017

And I came so close! Probably going to enable the insecure port, for the time being at least, in kops.

Is there an effort to implement an unprotected healthz endpoint? Can I help?

justinsb added a commit to justinsb/kops that referenced this issue Mar 29, 2017

Enable insecure port after all
Temporary workaround for
kubernetes/kubernetes#43784 until we can find
a better solution.
@liggitt

This comment has been minimized.

Member

liggitt commented Mar 31, 2017

@justinsb are you setting --anonymous-auth=false or leaving the authorization mode as AlwaysAllow?

@justinsb

This comment has been minimized.

Member

justinsb commented Apr 3, 2017

We always set anonymous-auth=false. We currently have two authorization modes - AlwaysAllow & RBAC; no hybrids supported yet.

@liggitt

This comment has been minimized.

Member

liggitt commented Apr 3, 2017

We always set anonymous-auth=false

is this a holdover from 1.5.0? From 1.5.1 on, the defaults are reasonable (anonymous auth is disabled by default in 1.5.x, and only enabled in 1.6 when using authorizers that require explicit ACL grants to anonymous users)

@justinsb

This comment has been minimized.

Member

justinsb commented Apr 3, 2017

It's a safety gate, yes. We never want anonymous authn to the API, even if authz should catch it it's too easy to make a mistake.

@liggitt

This comment has been minimized.

Member

liggitt commented Apr 3, 2017

it's certainly up to the deployer, but I expect that to cause difficulties as auth discovery gets built out (kubectl login asks where it should go to log in, etc).

@adamhaney

This comment has been minimized.

adamhaney commented May 28, 2017

kops v1.6.0 references this issue

W0528 11:00:47.597850 9970 apiserver.go:90] Enabling apiserver insecure port, for healthchecks (issue #43784)

But it's not clear what if any action I should take as a part of my configuration. Is that line kops telling me that they're using a workaround or that I need to change my configuration?

@liggitt

This comment has been minimized.

Member

liggitt commented May 28, 2017

There are currently two ways to allow an unauthenticated health check:

  1. enable the unsecured port, which lets anyone who can reach it be a superuser
  2. enable anonymous requests to the TLS port and depend on authorization to limit which endpoints anonymous requests are allowed to reach

kops disables anonymous requests to the TLS port, so it must enable the unsecured port if it wants to make anonymous health checks

@fejta-bot

This comment has been minimized.

fejta-bot commented Dec 25, 2017

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

@fejta-bot

This comment has been minimized.

fejta-bot commented Jan 24, 2018

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle rotten
/remove-lifecycle stale

@dghubble

This comment has been minimized.

Contributor

dghubble commented Jan 24, 2018

I sympathize with the kops point. Neither of these options seem particularly safe. Enabling insecure port isn't viable and anon seems to have its own subtleties and risks for mistakes. https://gist.github.com/erictune/9dc7ae4b22505b9a8c20ad9cd03a45cc

@justinsb

This comment has been minimized.

Member

justinsb commented Feb 2, 2018

/remove-lifecycle rotten

@liggitt

This comment has been minimized.

Member

liggitt commented Feb 10, 2018

anon seems to have its own subtleties and risks for mistakes. https://gist.github.com/erictune/9dc7ae4b22505b9a8c20ad9cd03a45cc

that was from the 1.5 timeframe, when that option was added. by 1.6, all authorizers were updated to require explicitly granting permissions to anonymous requests. the cautions about granting permissions to * and having those include anonymous users no longer apply.

rfranzke added a commit to gardener/gardener that referenced this issue Mar 2, 2018

Do not longer expose insecure kube-apiserver HTTP server
As of Kubernetes 1.10, the insecure flags will be deprecated: kubernetes/kubernetes#59018
Currently, there is no other way to allow unauthenticated health checks (requests on kube-apiserver's /healthz endpoint) other than allowing anonymous requests (which we do not want). Related issue: kubernetes/kubernetes#43784
We are now going to use the basic authentication credentials which the kubelet will use to reach the /healthz endpoint.

bendrucker added a commit to TakeScoop/typhoon that referenced this issue Mar 12, 2018

replace nlb will classic elb
Can't use NLB because:

* During bootstrap a single instance is in service
* NLB cannot direct traffic to the originating instance
* It preserves IPs

Can't use ALB because:

* Health check is HTTP/HTTPS
* apiserver will reject health checks w/ 401 (unauthorized)
* kubernetes/kubernetes#43784
@fejta-bot

This comment has been minimized.

fejta-bot commented May 11, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@justinsb

This comment has been minimized.

Member

justinsb commented May 25, 2018

/remove-lifecycle stale

@george-angel

This comment has been minimized.

george-angel commented Aug 22, 2018

Linking #51076 here as it seems to be discussing the same thing.

We are currently facing this dilemma, we are replicating AWS setup on GCP where the external LB only supports http healthchecks:

Currently, only Network Load Balancing requires legacy HttpHealthChecks. All other load balancers can use the newer style.

https://cloud.google.com/load-balancing/docs/health-checks

Our api-server is using self-signed certificates that we rotate daily, so we have to use a TCP LB.

The cleanest solution we can think of is running a systemd service on masters that is running http://k8s.gcr.io/exechealthz:1.0 that does curl localhost:8080/healthz and exposes a http port for the healthcheck to hit. This is obviously not great and we would rather have apiserver expose a non-TLS port to serve /healthz.

@fejta-bot

This comment has been minimized.

fejta-bot commented Nov 20, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@george-angel

This comment has been minimized.

george-angel commented Nov 20, 2018

/remove-lifecycle stale

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment