Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bug 1816302: UPSTREAM: 88505: Honor status.podIP over status.podIPs, node.spec.podCIDR over node.spec.podCIDRs #24750

Merged
merged 1 commit into from Mar 25, 2020
Merged

Bug 1816302: UPSTREAM: 88505: Honor status.podIP over status.podIPs, node.spec.podCIDR over node.spec.podCIDRs #24750

merged 1 commit into from Mar 25, 2020

Conversation

danwinship
Copy link
Contributor

pod.status.podIPs was introduced in kube 1.16 with the requirement that if podIP and podIPs are both set, they must match. This means that when a kube 1.14 / OCP 4.2 kubelet tries to change the podIP of a pod on a kube 1.16 / OCP 4.3 apiserver, it will fail because it didn't also try to change podIPs:

Mar 23 18:04:42 ip-10-0-163-3 hyperkube[1938]: W0323 18:04:42.769863    1938
status_manager.go:519] Failed to update status for pod "kube-apiserver-operator-
5d7c58bbb4-27mww_openshift-kube-apiserver-operator(93729130-6d20-11ea-
a89d-0a193a986132)": failed to patch status "..." for pod
"openshift-kube-apiserver-operator"/"kube-apiserver-operator-5d7c58bbb4-27mww":
conversion Error: v1.PodIP(10.129.0.65) != v1.PodIPs[0](10.129.0.41)

eg, specifically, if during a 4.2 to 4.3 upgrade, the apiserver pods are updated to 4.3, and then another pod gets restarted on a node that is still running a 4.2 kubelet, and upon being restarted it has a different IP then it did before, then it will get stuck.

This seems like something we should have hit sooner and I'm not sure why we didn't. (Possibly we used to end up always reassigning the pod its old IP again in this case, but some recent sdn/multus fix changed that?)

Upstream discussion: kubernetes/kubernetes#88505

This will need to be backported to 4.4 and 4.3.

@openshift-ci-robot
Copy link

@danwinship: This pull request references Bugzilla bug 1816302, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.5.0) matches configured target release for branch (4.5.0)
  • bug is in the state NEW, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

Bug 1816302: UPSTREAM: 88505: Honor status.podIP over status.podIPs, node.spec.podCIDR over node.spec.podCIDRs

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot openshift-ci-robot added the bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. label Mar 23, 2020
@openshift-ci-robot openshift-ci-robot added the vendor-update Touching vendor dir or related files label Mar 23, 2020
@danwinship
Copy link
Contributor Author

/retest

@deads2k
Copy link
Contributor

deads2k commented Mar 24, 2020

/lgtm

but I prefer bumping all of kube together for this. We have two CVEs to grab too.

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Mar 24, 2020
@openshift-ci-robot
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: danwinship, deads2k

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 24, 2020
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

12 similar comments
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-merge-robot openshift-merge-robot merged commit 927b43c into openshift:master Mar 25, 2020
@openshift-ci-robot
Copy link

@danwinship: All pull requests linked via external trackers have merged. Bugzilla bug 1816302 has been moved to the MODIFIED state.

In response to this:

Bug 1816302: UPSTREAM: 88505: Honor status.podIP over status.podIPs, node.spec.podCIDR over node.spec.podCIDRs

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@sdodson
Copy link
Member

sdodson commented Mar 25, 2020

/cherrypick release-4.4

@openshift-cherrypick-robot openshift-cherrypick-robot mentioned this pull request Mar 25, 2020
Closed
@openshift-cherrypick-robot
Copy link

@sdodson: new pull request created: #24761

In response to this:

/cherrypick release-4.4

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

This was referenced Mar 25, 2020
Merged
Merged
@danwinship danwinship deleted the pod-ip-pod-ips branch March 30, 2020 15:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@deads2k deads2k Awaiting requested review from deads2k

@soltysh soltysh Awaiting requested review from soltysh

Assignees

@deads2k deads2k

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. lgtm Indicates that a PR is ready to be merged. vendor-update Touching vendor dir or related files
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
7 participants
@danwinship @openshift-ci-robot @deads2k @openshift-bot @sdodson @openshift-cherrypick-robot @openshift-merge-robot