Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

This adds an etcd health check endpoint to kube-apiserver #49412

Merged
merged 1 commit into from Aug 3, 2017

Conversation

bjhaid
Copy link
Contributor

@bjhaid bjhaid commented Jul 21, 2017

addressing #48215.

What this PR does / why we need it:
This ensures kube-apiserver /healthz endpoint fails whenever connectivity cannot be established to etcd, also ensures the etcd preflight checks works with unix sockets

Which issue this PR fixes: fixes #48215

Special notes for your reviewer:
This PR does not use the etcd client directly as the client object is wrapped behind the storage interface and not exposed directly for use, so I decided to reuse what's being done in the preflight. So this will only check fail for connectivity and not etcd auth related problems. I did not write tests for the endpoint because I couldn't find examples that I could follow for writing tests for healthz related endpoints, I'll be willing to write those tests if someone can point me at a relevant one.

Release note:

Add etcd connectivity endpoint to healthz

@deads2k please help review, thanks!

@k8s-ci-robot
Copy link
Contributor

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://github.com/kubernetes/kubernetes/wiki/CLA-FAQ to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.


Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@k8s-ci-robot k8s-ci-robot added the cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. label Jul 21, 2017
@k8s-ci-robot
Copy link
Contributor

Hi @bjhaid. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@k8s-ci-robot k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jul 21, 2017
@k8s-github-robot k8s-github-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. release-note Denotes a PR that will be considered when it comes time to generate release notes. labels Jul 21, 2017
@bjhaid
Copy link
Contributor Author

bjhaid commented Jul 21, 2017

/assign @deads2k

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Jul 21, 2017
@@ -20,6 +20,9 @@ import (
"fmt"

"github.com/spf13/pflag"
"net/http"
"k8s.io/apiserver/pkg/server/healthz"
"k8s.io/kubernetes/cmd/kube-apiserver/app/preflight"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This dependency won't be allowed. You'll need to move the etcd check code to the k8s.io/apiserver staging section somewhere.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the reverse allowed, that is adding a k8s.io/apiserver/... dependency to k8s.io/kubernetes/cmd/kube-apiserver/app/preflight? So I don't end up duplicating code

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the reverse allowed, that is adding a k8s.io/apiserver/... dependency to k8s.io/kubernetes/cmd/kube-apiserver/app/preflight

yes the reverse is allowed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can I just move preflight and it's test in to staging and fix the references?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can I just move preflight and it's test in to staging and fix the references?

Yes. This seems more like apiserver/pkg/storage/etcd3/healthz

@deads2k
Copy link
Contributor

deads2k commented Jul 24, 2017

@k8s-bot ok to test

@k8s-ci-robot k8s-ci-robot removed the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jul 24, 2017
@deads2k
Copy link
Contributor

deads2k commented Jul 24, 2017

This is looking good.

@bjhaid bjhaid force-pushed the etcd_healthz_endpoint branch 2 times, most recently from 89ee978 to 9e62dc9 Compare July 25, 2017 22:26
@deads2k
Copy link
Contributor

deads2k commented Jul 26, 2017

looks like something is going wrong.

@bjhaid bjhaid changed the title This add and etcd health check endpoint to kube-apiserver This add an etcd health check endpoint to kube-apiserver Jul 26, 2017
@bjhaid
Copy link
Contributor Author

bjhaid commented Jul 26, 2017

@deads2k

looks like something is going wrong.

I spoke to folks in #sig-testing and they told me to run make coverage and one of the many things I saw is the not allowed dependency, if I move things around maybe that will help

@bjhaid bjhaid changed the title This add an etcd health check endpoint to kube-apiserver This adds an etcd health check endpoint to kube-apiserver Jul 29, 2017
@bjhaid bjhaid force-pushed the etcd_healthz_endpoint branch 4 times, most recently from 0630735 to bbed16e Compare July 30, 2017 03:36
@bjhaid
Copy link
Contributor Author

bjhaid commented Jul 30, 2017

/test pull-kubernetes-bazel

2 similar comments
@bjhaid
Copy link
Contributor Author

bjhaid commented Jul 31, 2017

/test pull-kubernetes-bazel

@bjhaid
Copy link
Contributor Author

bjhaid commented Jul 31, 2017

/test pull-kubernetes-bazel

@bjhaid
Copy link
Contributor Author

bjhaid commented Jul 31, 2017

/test Submit Queue

@bjhaid
Copy link
Contributor Author

bjhaid commented Jul 31, 2017

@deads2k needs a lgtm..

@kubernetes kubernetes deleted a comment from bjhaid Aug 1, 2017
@@ -20,6 +20,9 @@ import (
"fmt"

"github.com/spf13/pflag"
"k8s.io/apiserver/pkg/server/healthz"
"k8s.io/apiserver/pkg/storage/etcd3/preflight"
"net/http"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: this goes in first section of imports since its a stdlib

@@ -20,6 +20,9 @@ import (
"fmt"

"github.com/spf13/pflag"
"k8s.io/apiserver/pkg/server/healthz"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: want an empty line before these. we normally put a gap between dependencies and in-project deps

@deads2k
Copy link
Contributor

deads2k commented Aug 1, 2017

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 1, 2017
@k8s-github-robot k8s-github-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 1, 2017
@k8s-github-robot k8s-github-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 1, 2017
@bjhaid
Copy link
Contributor Author

bjhaid commented Aug 1, 2017

/test pull-kubernetes-unit

@bjhaid
Copy link
Contributor Author

bjhaid commented Aug 1, 2017

/test pull-kubernetes-verify

@bjhaid
Copy link
Contributor Author

bjhaid commented Aug 2, 2017

/test pull-kubernetes-e2e-kops-aws

@bjhaid
Copy link
Contributor Author

bjhaid commented Aug 2, 2017

/test pull-kubernetes-e2e-gce-etcd3

@k8s-ci-robot
Copy link
Contributor

@bjhaid: you can't request testing unless you are a kubernetes member.

In response to this:

/test pull-kubernetes-e2e-kops-aws

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot
Copy link
Contributor

@bjhaid: you can't request testing unless you are a kubernetes member.

In response to this:

/test pull-kubernetes-e2e-gce-etcd3

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@bjhaid
Copy link
Contributor Author

bjhaid commented Aug 2, 2017

/test pull-kubernetes-e2e-kops-aws

@k8s-ci-robot
Copy link
Contributor

@bjhaid: you can't request testing unless you are a kubernetes member.

In response to this:

/test pull-kubernetes-e2e-kops-aws

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@fejta
Copy link
Contributor

fejta commented Aug 2, 2017

@deads2k we retired the @k8s-bot ok to test command, please start using /ok-to-test (we no longer recognize the ok to test message)

/ok-to-test

@deads2k
Copy link
Contributor

deads2k commented Aug 2, 2017

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 2, 2017
@k8s-github-robot
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bjhaid, deads2k

Associated issue: 48215

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

@k8s-github-robot
Copy link

Automatic merge from submit-queue (batch tested with PRs 49989, 49806, 49649, 49412, 49512)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Exercise etcd connectivity in the /healthz/ping endpoint
7 participants