[release-4.11] OCPBUGS-3490: OVN-Kubernetes: Prefer oldest nodes #1641

kyrtapz · 2022-11-28T11:13:05Z

This cherry picks the improvements to ovn-k startup from #1579

Sometimes the number of masters changes, like when in the etcd test: etcd [apigroup:config.openshift.io] is able to vertically scale up and down with a single node This leads to problems like: I0909 11:16:02.221234 1 ovn_kubernetes.go:938] Waiting to complete OVN bootstrap: found (4) master nodes out of (3) expected: timing out in 235 seconds ovsdb-server only ever wants an odd number of members to ensure consensus in RAFT clusters. If we have 4 members and one of them is dead (like when the 4th one gets deleted) the RAFT cluster gets a bit unhappy. The CNO currently renders the ovnkube master pods with the IP addresses of all master nodes, regardless of how many control plane nodes were actually requested at install time. That's not cool. Don't do that. Instead, take the oldest master nodes (sorted by creation time) as the RAFT cluster members. Tell any NB/SB containers that aren't in the list to do nothing for a really long time (to prevent CrashloopBackoff due to early exits from the container script) and not join the cluster. If this really is a master replacement, then the cluster will shift over to the new master when the original one is finally removed. Signed-off-by: Dan Williams <dcbw@redhat.com> (cherry picked from commit c0c317e)

(cherry picked from commit 9d22f87)

When the postStart hooks fail kubelet kills the DB containers with a 30s grace period. If the DBs started at different times (because they're on different nodes, have different kubelets, etc) they may not have enough runtime overlap to establish the RAFT cluster before one or more of them get killed by kubelet. First, make the postStart scripts wait longer by retrying the stuff they do more times until the cluster is established. Second, wrap the IPsec enable/disable in a retry loop too and make it exit with an error if it fails instead of ignoring the problem. Third, add an IPsec check to the SB postStart to wait a bit more time for the SB cluster to establish, if needed. (cherry picked from commit d994351)

openshift-ci · 2022-11-28T11:13:21Z

@kyrtapz: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

In response to this:

[release-4.11] OVN-Kubernetes: Prefer oldest nodes

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2022-11-28T11:13:45Z

@kyrtapz: GitHub didn't allow me to request PR reviews from the following users: kyrtapz.

Note that only openshift members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

This cherry picks the improvements to ovn-k startup from #1579

/cc
/hold

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

kyrtapz · 2022-11-28T14:20:20Z

/retest

openshift-ci · 2022-11-29T12:05:13Z

@kyrtapz: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

In response to this:

[release-4.11] OCPBUGS-3490 OVN-Kubernetes: Prefer oldest nodes

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2022-11-29T12:05:28Z

@kyrtapz: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

In response to this:

[release-4.11] OCPBUGS-3490 OVN-Kubernetes: Prefer oldest nodes

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2022-11-29T12:05:39Z

@kyrtapz: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

In response to this:

[release-4.11] OCPBUGS-3490: OVN-Kubernetes: Prefer oldest nodes

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2022-11-29T12:05:39Z

@kyrtapz: This pull request references Jira Issue OCPBUGS-3490, which is invalid:

expected the bug to target the "4.11.z" version, but no target version was set
expected Jira Issue OCPBUGS-3490 to depend on a bug targeting a version in 4.12.0 and in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), but no dependents were found

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

This cherry picks the improvements to ovn-k startup from #1579

/cc @martinkennelly @dcbw

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

kyrtapz · 2022-11-29T12:07:20Z

/jira refresh

openshift-ci-robot · 2022-11-29T12:07:27Z

@kyrtapz: This pull request references Jira Issue OCPBUGS-3490, which is valid. The bug has been moved to the POST state.

6 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.11.z) matches configured target version for branch (4.11.z)
bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)
dependent bug Jira Issue OCPBUGS-4237 is in the state Closed (Done), which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE))
dependent Jira Issue OCPBUGS-4237 targets the "4.12.0" version, which is one of the valid target versions: 4.12.0
bug has dependents

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

kyrtapz · 2022-11-29T12:09:24Z

/remove-hold

kyrtapz · 2022-11-29T12:09:36Z

/retest

martinkennelly · 2022-11-29T14:54:08Z

/retest
Job e2e-hypershift is failing with "failed to get version for release image". Looks like registry issue.

kyrtapz · 2022-12-06T11:25:54Z

/retest

martinkennelly · 2022-12-06T13:56:29Z

/retest
:(

kyrtapz · 2022-12-08T10:53:14Z

/retest

kyrtapz · 2022-12-08T12:50:25Z

HyperShift failure doesn't look related, OVN-K Master is up and running

dcbw · 2022-12-08T19:02:00Z

/approve
/lgtm

openshift-ci · 2022-12-08T19:03:40Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dcbw, kyrtapz

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [dcbw]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

jcaamano · 2022-12-12T12:17:35Z

/label backport-risk-assessed

mffiedler · 2022-12-12T15:16:56Z

/label cherry-pick-approved

openshift-ci-robot · 2022-12-12T15:56:50Z

/retest-required

Remaining retests: 0 against base HEAD 0d88535 and 2 for PR HEAD c14fa49 in total

kyrtapz · 2022-12-12T18:19:38Z

/retest-required

kyrtapz · 2022-12-21T13:53:11Z

/retest-required

openshift-ci · 2022-12-21T15:46:12Z

@kyrtapz: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-hypershift	`c14fa49`	link	false	`/test e2e-hypershift`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

kyrtapz · 2022-12-21T15:49:47Z

/retest-required

openshift-ci-robot · 2022-12-21T17:39:23Z

@kyrtapz: All pull requests linked via external trackers have merged:

openshift/cluster-network-operator#1641

Jira Issue OCPBUGS-3490 has been moved to the MODIFIED state.

In response to this:

This cherry picks the improvements to ovn-k startup from #1579

/cc @martinkennelly @dcbw

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

dcbw added 3 commits November 28, 2022 12:06

ovn: remove unused variable

b7f0002

(cherry picked from commit 9d22f87)

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 28, 2022

kyrtapz changed the title ~~[release-4.11] OVN-Kubernetes: Prefer oldest nodes~~ [release-4.11] OCPBUGS-3490 OVN-Kubernetes: Prefer oldest nodes Nov 29, 2022

kyrtapz changed the title ~~[release-4.11] OCPBUGS-3490 OVN-Kubernetes: Prefer oldest nodes~~ [release-4.11] OCPBUGS-3490: OVN-Kubernetes: Prefer oldest nodes Nov 29, 2022

openshift-ci-robot added the jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. label Nov 29, 2022

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 29, 2022

openshift-ci bot assigned dcbw Dec 8, 2022

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Dec 8, 2022

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 8, 2022

openshift-ci bot added the backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. label Dec 12, 2022

openshift-ci bot assigned anuragthehatter, asood-rh, eurijon, huiran0826, jechen0648, mffiedler, qiowang721, rbbratta and weliang1 Dec 12, 2022

openshift-ci bot added the cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. label Dec 12, 2022

openshift-merge-robot merged commit ae7c9d9 into openshift:release-4.11 Dec 21, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[release-4.11] OCPBUGS-3490: OVN-Kubernetes: Prefer oldest nodes #1641

[release-4.11] OCPBUGS-3490: OVN-Kubernetes: Prefer oldest nodes #1641

kyrtapz commented Nov 28, 2022 •

edited

openshift-ci bot commented Nov 28, 2022

openshift-ci bot commented Nov 28, 2022

kyrtapz commented Nov 28, 2022

openshift-ci bot commented Nov 29, 2022

openshift-ci bot commented Nov 29, 2022

openshift-ci bot commented Nov 29, 2022

openshift-ci-robot commented Nov 29, 2022

kyrtapz commented Nov 29, 2022

openshift-ci-robot commented Nov 29, 2022

kyrtapz commented Nov 29, 2022

kyrtapz commented Nov 29, 2022

martinkennelly commented Nov 29, 2022

kyrtapz commented Dec 6, 2022

martinkennelly commented Dec 6, 2022

kyrtapz commented Dec 8, 2022

kyrtapz commented Dec 8, 2022

dcbw commented Dec 8, 2022

openshift-ci bot commented Dec 8, 2022

jcaamano commented Dec 12, 2022

mffiedler commented Dec 12, 2022

openshift-ci-robot commented Dec 12, 2022

kyrtapz commented Dec 12, 2022

kyrtapz commented Dec 21, 2022

openshift-ci bot commented Dec 21, 2022 •

edited

kyrtapz commented Dec 21, 2022

openshift-ci-robot commented Dec 21, 2022

[release-4.11] OCPBUGS-3490: OVN-Kubernetes: Prefer oldest nodes #1641

[release-4.11] OCPBUGS-3490: OVN-Kubernetes: Prefer oldest nodes #1641

Conversation

kyrtapz commented Nov 28, 2022 • edited

openshift-ci bot commented Nov 28, 2022

openshift-ci bot commented Nov 28, 2022

kyrtapz commented Nov 28, 2022

openshift-ci bot commented Nov 29, 2022

openshift-ci bot commented Nov 29, 2022

openshift-ci bot commented Nov 29, 2022

openshift-ci-robot commented Nov 29, 2022

kyrtapz commented Nov 29, 2022

openshift-ci-robot commented Nov 29, 2022

kyrtapz commented Nov 29, 2022

kyrtapz commented Nov 29, 2022

martinkennelly commented Nov 29, 2022

kyrtapz commented Dec 6, 2022

martinkennelly commented Dec 6, 2022

kyrtapz commented Dec 8, 2022

kyrtapz commented Dec 8, 2022

dcbw commented Dec 8, 2022

openshift-ci bot commented Dec 8, 2022

jcaamano commented Dec 12, 2022

mffiedler commented Dec 12, 2022

openshift-ci-robot commented Dec 12, 2022

kyrtapz commented Dec 12, 2022

kyrtapz commented Dec 21, 2022

openshift-ci bot commented Dec 21, 2022 • edited

kyrtapz commented Dec 21, 2022

openshift-ci-robot commented Dec 21, 2022

kyrtapz commented Nov 28, 2022 •

edited

openshift-ci bot commented Dec 21, 2022 •

edited