Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set minAvailable:1 to unblock node upgrades #3784

Merged
merged 1 commit into from
Feb 24, 2021

Conversation

imjasonh
Copy link
Member

Fixes #3654

cc @nikhil-thomas

/kind bug

Submitter Checklist

These are the criteria that every PR should meet, please check them off as you
review them:

  • [n] Includes tests (if functionality changed/added)
  • [n] Includes docs (if user facing)
  • [y] Commit messages follow commit message best practices
  • [y] Release notes block has been filled in or deleted (only if no user facing changes)

See the contribution guide for more details.

Double check this list of stuff that's easy to miss:

Reviewer Notes

If API changes are included, additive changes must be approved by at least two OWNERS and backwards incompatible changes must be approved by more than 50% of the OWNERS, and they must first be added in a backwards compatible way.

Release Notes

Modify webhook PodDisruptionBudget minAvailable to 1, so node upgrades aren't blocked

@tekton-robot tekton-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/bug Categorizes issue or PR as related to a bug. labels Feb 23, 2021
@tekton-robot tekton-robot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Feb 23, 2021
Copy link
Member

@vdemeester vdemeester left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/meow

@tekton-robot
Copy link
Collaborator

@vdemeester: cat image

In response to this:

/meow

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@tekton-robot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vdemeester

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@tekton-robot tekton-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 23, 2021
@bobcatfish
Copy link
Collaborator

/lgtm

@tekton-robot tekton-robot added the lgtm Indicates that a PR is ready to be merged. label Feb 23, 2021
@nikhil-thomas
Copy link
Member

@imjasonh may be im missing how pdb works. 🤓 I am wondering whether this will be enough in our case as we have set the webhook replica count to 1 (

replicas: 1
)

so my concern is that if we set minAvailable: 1 in pdb where the replica count is 1 in the deployment spec, wouldn't it still block an eviction/drain.

@nikhil-thomas
Copy link
Member

nikhil-thomas commented Feb 24, 2021

i experimented with minAvailable: 1

what i did

  1. create pdb with minAvailable: 1 = 1
  2. use `kubectl drain command
$ kubectl drain ip-10-0.us-east-2.compute.internal
evicting pod openshift-pipelines/tekton-pipelines-webhook-84cf757556-js76b
error when evicting pod "tekton-pipelines-webhook-84cf757556-js76b" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
  1. then i set the minAvailable: 0 in pdb. this avolided the drain block
$ kubectl drain ip-10-0.us-east-2.compute.internal --ignore-daemonsets                                                                                       130 ↵
node/ip-10-0-177-72.us-east-2.compute.internal cordoned
... 
evicting pod tekton-pipelines/tekton-pipelines-controller-68f85bfbd8-jwrz9
evicting pod tekton-pipelines/tekton-pipelines-webhook-76c87bfb9c-dbmxj

i don't fully understand, how autoscaller, pdb and replica count in a deployment mainfest work together.
Im just trying to make sure this patch will solve the problem for us. :)

observed same result on

  • Kubernetes (Server Version: v1.18.12-gke.1210)
  • OpenShift (Server Version: 4.6.18, Kubernetes Version: v1.19.0+f173eb4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. kind/bug Categorizes issue or PR as related to a bug. lgtm Indicates that a PR is ready to be merged. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

PodDisruptionBudget causing inability to gracefully drain a node with tekton-pipelines-webhook pod
5 participants