Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug 1761609: Remove SDN ds liveness check #385

Merged

Conversation

rcarrillocruz
Copy link
Contributor

Fixes BZ 1761609

@openshift-ci-robot
Copy link
Contributor

@rcarrillocruz: This pull request references Bugzilla bug 1761609, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

Bug 1761609: Remove SDN ds liveness check

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot openshift-ci-robot added bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Nov 6, 2019
@@ -159,11 +159,6 @@ spec:
preStop:
exec:
command: ["rm","-f","/etc/cni/net.d/80-openshift-network.conf", "/host/opt/cni/bin/openshift-sdn"]
# this comes from the kube-proxy code
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@squeed do you think this should be removed altogether or we should just tweak around the probe parameters at scale to account for slower response times?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem (IIRC) is that the kube-proxy liveness server doesn't get started until after the sdn "node" stuff is fully started up, so in large clusters where startup takes longer, the kubelet may time out on the liveness check before it's even running. So the fix, if we want to keep the liveness check, is to rearrange sdn startup. But I think there was some feeling that the liveness check isn't really super helpful anyway...

@squeed
Copy link
Contributor

squeed commented Nov 8, 2019

Right.
At this point, we've not seen the liveness check detect any sort of liveness failure.. Especially troublesome is that if we take a long time to fill our informer caches (because of slow apiserver or big cluster), the liveness check will shoot us in the head. I know that I had assumed the liveness would hold off until the pod entered Ready state. Nope.

/approve
/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Nov 8, 2019
@openshift-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rcarrillocruz, squeed

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 8, 2019
@openshift-merge-robot openshift-merge-robot merged commit 56c5a7a into openshift:master Nov 8, 2019
@openshift-ci-robot
Copy link
Contributor

@rcarrillocruz: All pull requests linked via external trackers have merged. Bugzilla bug 1761609 has been moved to the MODIFIED state.

In response to this:

Bug 1761609: Remove SDN ds liveness check

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. lgtm Indicates that a PR is ready to be merged. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants