Bug 2021097: Set Upgradeable: false when HA workloads are incorrectly spread #1472

jan--f · 2021-11-08T13:12:37Z

… spread (#1431)

Merge pull request Bug 1995924: Set Upgradeable: false when HA workloads are incorrectly spread #1330 from dgrisonnet/ha-upgradeable

Bug 1995924: Set Upgradeable: false when HA workloads are incorrectly spread

pkg/rebalancer: sort resources for deletion

The resources that are marked for deletion are sorted by their PVC
creation timestamp, from the newest to the oldest to make the deletion
consistent.

Signed-off-by: Damien Grisonnet dgrisonn@redhat.com

pkg: improve rebalancer logging

Signed-off-by: Damien Grisonnet dgrisonn@redhat.com

pkg/rebalancer: sort based on pod names

Sort PVCs by their creation timestamps, from the newest to the oldest to
make sure that the oldest PVC is retained in case all of them are
annotated. If some PVCs have the same creation timestamp, they will be
sorted based on their pod name.

Signed-off-by: Damien Grisonnet dgrisonn@redhat.com

pkg/rebalancer: split annotation removal

Add EnsurePVCsAreNotAnnoted function that makes sure that none of the
PVCs of the given workload have the
openshift.io/cluster-monitoring-drop-pvc annotation after the
rebalancing is done. In case one of the PVC has the annotation, it will
be removed to prevent deleting the PVC in a future cycle.

Signed-off-by: Damien Grisonnet dgrisonn@redhat.com

pkg/rebalancer: sort resources to delete by age

Signed-off-by: Damien Grisonnet dgrisonn@redhat.com

test/e2e: fix framework

Signed-off-by: Damien Grisonnet dgrisonn@redhat.com

Co-authored-by: OpenShift Merge Robot openshift-merge-robot@users.noreply.github.com
(cherry picked from commit 8500e0f)

I added CHANGELOG entry for this change.
No user facing changes, so no entry in CHANGELOG was needed.

openshift-ci · 2021-11-08T13:12:42Z

@jan--f: This pull request references Bugzilla bug 2021097, which is invalid:

expected dependent Bugzilla bug 1995924 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE), but it is ON_QA instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Bug 2021097: Set Upgradeable: false when HA workloads are incorrectly spread

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jan--f · 2021-11-09T11:42:38Z

/hold
Investigating a CI issue with the master PR and still resolving backport issues.

jan--f · 2021-11-15T12:49:59Z

/retest

jan--f · 2021-11-15T14:51:18Z

/retest

jan--f · 2021-11-16T07:42:02Z

/retest

… spread (openshift#1431) * Merge pull request openshift#1330 from dgrisonnet/ha-upgradeable Bug 1995924: Set `Upgradeable: false` when HA workloads are incorrectly spread * pkg/rebalancer: sort resources for deletion The resources that are marked for deletion are sorted by their PVC creation timestamp, from the newest to the oldest to make the deletion consistent. Signed-off-by: Damien Grisonnet <dgrisonn@redhat.com> * pkg: improve rebalancer logging Signed-off-by: Damien Grisonnet <dgrisonn@redhat.com> * pkg/rebalancer: sort based on pod names Sort PVCs by their creation timestamps, from the newest to the oldest to make sure that the oldest PVC is retained in case all of them are annotated. If some PVCs have the same creation timestamp, they will be sorted based on their pod name. Signed-off-by: Damien Grisonnet <dgrisonn@redhat.com> * pkg/rebalancer: split annotation removal Add EnsurePVCsAreNotAnnoted function that makes sure that none of the PVCs of the given workload have the openshift.io/cluster-monitoring-drop-pvc annotation after the rebalancing is done. In case one of the PVC has the annotation, it will be removed to prevent deleting the PVC in a future cycle. Signed-off-by: Damien Grisonnet <dgrisonn@redhat.com> * pkg/rebalancer: sort resources to delete by age Signed-off-by: Damien Grisonnet <dgrisonn@redhat.com> * test/e2e: fix framework Signed-off-by: Damien Grisonnet <dgrisonn@redhat.com> Co-authored-by: OpenShift Merge Robot <openshift-merge-robot@users.noreply.github.com> (cherry picked from commit 8500e0f)

On master this was added in d3c1e5a. This can be considered a partial backport. Signed-off-by: Jan Fajerski <jfajersk@redhat.com>

jan--f · 2021-11-16T14:27:45Z

/retest

jan--f · 2021-11-17T08:54:43Z

/retest

jan--f · 2021-11-17T13:37:56Z

/bugzilla refresh

openshift-ci · 2021-11-17T13:38:06Z

@jan--f: This pull request references Bugzilla bug 2021097, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

6 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.9.z) matches configured target release for branch (4.9.z)
bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)
dependent bug Bugzilla bug 1995924 is in the state VERIFIED, which is one of the valid states (VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE))
dependent Bugzilla bug 1995924 targets the "4.10.0" release, which is one of the valid target releases: 4.10.0
bug has dependents

Requesting review from QA contact:
/cc @juzhao

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jan--f · 2021-11-17T13:38:53Z

/unhold
I think this is ready to review and merge.

simonpasquier · 2021-11-17T14:02:03Z

I'll add this at the top of my review list :)

simonpasquier · 2021-11-17T16:17:55Z

/lgtm

simonpasquier · 2021-11-17T16:18:23Z

/retest

simonpasquier · 2021-11-17T16:19:49Z

/skip

simonpasquier · 2021-11-17T16:21:28Z

/label backport-risk-assessed

openshift-bot · 2021-11-17T16:43:37Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-11-17T16:55:39Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-11-17T17:56:38Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-11-17T18:46:41Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-11-17T18:59:39Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-11-17T19:36:38Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-11-17T21:07:44Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-11-17T21:59:38Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-11-17T23:04:38Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-ci · 2021-11-17T23:19:01Z

@jan--f: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/versions	`b445978`	link	false	`/test versions`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-bot · 2021-11-17T23:30:39Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-11-18T00:22:39Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-11-18T00:35:39Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-11-18T01:01:38Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-11-18T01:14:39Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

juzhao · 2021-11-18T03:45:49Z

tested with PR, bound PVs for prometheus, and schedule prometheus pods to one same node, Upgradeable is False now
/lgtm

juzhao · 2021-11-18T03:46:13Z

/label qe-approved

juzhao · 2021-11-18T03:46:26Z

/label cherry-pick-approved

openshift-ci · 2021-11-18T03:46:38Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jan--f, juzhao, simonpasquier

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [jan--f,simonpasquier]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci · 2021-11-18T03:47:28Z

@jan--f: All pull requests linked via external trackers have merged:

openshift/cluster-monitoring-operator#1472

Bugzilla bug 2021097 has been moved to the MODIFIED state.

In response to this:

Bug 2021097: Set Upgradeable: false when HA workloads are incorrectly spread

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci bot added bugzilla/severity-high Referenced Bugzilla bug's severity is high for the branch this PR is targeting. bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Nov 8, 2021

openshift-ci bot requested review from arajkumar and fpetkovski November 8, 2021 13:14

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 8, 2021

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 9, 2021

simonpasquier mentioned this pull request Nov 12, 2021

Bug 1933847: enable hard affinity + PodDisruptionBudget for Prometheus and Thanos Ruler pods #1341

Merged

2 tasks

jan--f force-pushed the bp-1995924-4.9 branch 2 times, most recently from eb827f5 to 5ff3f13 Compare November 15, 2021 10:36

dgrisonnet and others added 2 commits November 16, 2021 11:13

e2e/framework: add context for to e2e and framework module

b445978

On master this was added in d3c1e5a. This can be considered a partial backport. Signed-off-by: Jan Fajerski <jfajersk@redhat.com>

jan--f force-pushed the bp-1995924-4.9 branch from 5ff3f13 to b445978 Compare November 16, 2021 10:14

openshift-ci bot added bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. and removed bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Nov 17, 2021

openshift-ci bot requested a review from juzhao November 17, 2021 13:38

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 17, 2021

openshift-ci bot added the backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. label Nov 17, 2021

openshift-ci bot assigned simonpasquier Nov 17, 2021

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Nov 17, 2021

openshift-ci bot added the qe-approved Signifies that QE has signed off on this PR label Nov 18, 2021

openshift-ci bot assigned juzhao Nov 18, 2021

openshift-ci bot added the cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. label Nov 18, 2021

openshift-merge-robot merged commit ae7b0a3 into openshift:release-4.9 Nov 18, 2021

Bug 2021097: Set Upgradeable: false when HA workloads are incorrectly spread #1472

Bug 2021097: Set Upgradeable: false when HA workloads are incorrectly spread #1472

Conversation

jan--f commented Nov 8, 2021

openshift-ci bot commented Nov 8, 2021

jan--f commented Nov 9, 2021

jan--f commented Nov 15, 2021

jan--f commented Nov 15, 2021

jan--f commented Nov 16, 2021

jan--f commented Nov 16, 2021

jan--f commented Nov 17, 2021

jan--f commented Nov 17, 2021

openshift-ci bot commented Nov 17, 2021

jan--f commented Nov 17, 2021

simonpasquier commented Nov 17, 2021

simonpasquier commented Nov 17, 2021

simonpasquier commented Nov 17, 2021

simonpasquier commented Nov 17, 2021

simonpasquier commented Nov 17, 2021

openshift-bot commented Nov 17, 2021

openshift-bot commented Nov 17, 2021

openshift-bot commented Nov 17, 2021

openshift-bot commented Nov 17, 2021

openshift-bot commented Nov 17, 2021

openshift-bot commented Nov 17, 2021

openshift-bot commented Nov 17, 2021

openshift-bot commented Nov 17, 2021

openshift-bot commented Nov 17, 2021

openshift-ci bot commented Nov 17, 2021 • edited

openshift-bot commented Nov 17, 2021

openshift-bot commented Nov 18, 2021

openshift-bot commented Nov 18, 2021

openshift-bot commented Nov 18, 2021

openshift-bot commented Nov 18, 2021

juzhao commented Nov 18, 2021

juzhao commented Nov 18, 2021

juzhao commented Nov 18, 2021

openshift-ci bot commented Nov 18, 2021

openshift-ci bot commented Nov 18, 2021

openshift-ci bot commented Nov 17, 2021 •

edited