ETCD-612: Added a callback to provide additional pre-conditions for installation #1749

jubittajohn · 2024-06-14T18:46:15Z

Added the callback shouldRevisionInstall to provide additional conditions for installation

These changes are consumed by : openshift/cluster-etcd-operator#1278

openshift-ci-robot · 2024-06-14T18:54:48Z

@jubittajohn: This pull request references ETCD-612 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.17.0" version, but no target version was set.

In response to this:

…onditions for installation

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2024-06-14T19:21:55Z

@jubittajohn: This pull request references ETCD-612 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.17.0" version, but no target version was set.

In response to this:

Added the callback shouldRevisionInstall to provide additional conditions for installation

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

pkg/operator/staticpod/controller/installer/installer_controller.go

tjungblu · 2024-06-17T09:29:21Z

pkg/operator/staticpod/controller/installer/installer_controller.go

@@ -442,6 +449,16 @@ func (c *InstallerController) manageInstallationPods(ctx context.Context, operat
 		return true, requeueAfter, nil
 	}

+	// Check if revision should be installed based on if the quorum is about to be violated
+	if c.shouldRevisionInstall != nil {


as far as I understand, this blocks the entire controller, which we don't want, we want the current revisions to continue to rollout.

We can be more granular here and just block the ensureInstallerPod routine from spawning just the installer pod. That way it wouldn't block existing rollouts. Having said that, I think there are generally very little tests around this, so we need to run a couple of payload tests in cluster-etcd-operator (CEO) as well

tjungblu · 2024-06-21T14:30:52Z

pkg/operator/staticpod/controller/installer/installer_controller.go

+	// checks if new revision should be rolled out
+	if c.shouldRevisionInstall != nil {
+		shouldInstall, err := c.shouldRevisionInstall()
+		if !shouldInstall {


shouldn't that be

Suggested change

if !shouldInstall {

if err != nil {

return err

}

if !shouldInstall {

return nil

}

maybe also add an info logging statement here, so we know if it was skipped

@tjungblu But only if there is error the requeue will happen as per the current logic here. https://github.com/openshift/library-go/blob/master/pkg/operator/staticpod/controller/installer/installer_controller.go#L481.

then we need to rethink the ensureInstallerPod signature potentially. I don't think we should trigger an event for this for example:

c.eventRecorder.Warningf("InstallerPodFailed", "Failed to create installer pod for revision %d count %d on node %q: %v", currNodeState.TargetRevision, currNodeState.LastFailedCount, currNodeState.NodeName, err)

that's misleading, because there was no actual error that failed the installation. We just skipped it.

tjungblu · 2024-06-21T14:34:54Z

pkg/operator/staticpod/controller/installer/installer_controller_test.go

+				kubeClient:      fake.NewSimpleClientset(),
+				eventRecorder:   eventstesting.NewTestingEventRecorder(t),
+				shouldRevisionInstall: func() (bool, error) {
+					return false, fmt.Errorf("revision shouldn't be installed")


ah okay, now I get it. If you want to use an error for control flow, that's fine. But then we don't really need the boolean to denote whether to skip it or not.

I (personally) would use the error for real errors though.

pkg/operator/staticpod/controllers.go

tjungblu · 2024-06-21T14:42:56Z

pkg/operator/staticpod/controller/installer/installer_controller_test.go

+				},
+				originalOperatorStatus: &operatorv1.StaticPodOperatorStatus{
+					LatestAvailableRevision: 1,
+					NodeStatuses: []operatorv1.NodeStatus{


I only very superficially looked through your tests, but I'm generally missing one with multiple NodeStatuses.

Maybe more meaningful for etcd would be a test with three nodes at those revisions:

master-0 = 4
master-1 = 3 (would be next to get to 4)
master-2 = 3

Now our quorum guard says master-0 went away (eg, machine was turned off). What happens to the remaining revisions?

I would expect this to block the installation of rev 4 on master-1 and master-2.
Maybe also think of a couple of such tests, maybe with a more structured permutation approach. We can discuss this also next Tuesday.

jubittajohn · 2024-06-21T17:44:54Z

/assign @dusk125

pkg/operator/staticpod/controller/installer/installer_controller.go

openshift-ci-robot · 2024-06-25T17:43:45Z

@jubittajohn: This pull request references ETCD-612 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.17.0" version, but no target version was set.

In response to this:

Added the callback shouldRevisionInstall to provide additional conditions for installation

This changes are consumed by : openshift/cluster-etcd-operator#1278

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

pkg/operator/staticpod/controller/installer/installer_controller.go

openshift-ci-robot · 2024-06-26T14:11:34Z

@jubittajohn: This pull request references ETCD-612 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.17.0" version, but no target version was set.

In response to this:

Added the callback shouldRevisionInstall to provide additional conditions for installation

These changes are consumed by : openshift/cluster-etcd-operator#1278

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

tjungblu · 2024-06-26T14:35:19Z

/lgtm (non binding)

can you please also squash all commits into one and remove the WIP please?

tjungblu · 2024-06-26T14:36:35Z

/assign @dgrisonnet

openshift-ci · 2024-06-26T14:47:39Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: jubittajohn, tjungblu
Once this PR has been reviewed and has the lgtm label, please ask for approval from dgrisonnet. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

pkg/operator/staticpod/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

dgrisonnet · 2024-06-28T13:04:25Z

pkg/operator/staticpod/controller/installer/installer_controller.go

@@ -98,6 +98,9 @@ type InstallerController struct {
 	clock            clock.Clock
 	installerBackOff func(count int) time.Duration
 	fallbackBackOff  func(count int) time.Duration
+
+	// shouldRevisionInstall is a callback function that determines whether a new revision should be installed
+	shouldRevisionInstall func() (bool, error)


I think we should follow the same precondition pattern that we have in the static resource controller: https://github.com/openshift/library-go/blob/master/pkg/operator/staticresourcecontroller/static_resource_controller.go#L55-L59. Meaning that we would allow for any condition to be defined to gate the revision process.

Also, we should pass on a context in that method and propagate it all the way down in case we need to cancel it for some reasons. Especially in https://github.com/openshift/cluster-etcd-operator/blob/master/pkg/operator/ceohelpers/bootstrap.go#L139 and maybe some other functions that don't take it yet.

good idea, @jubittajohn can we create it slightly different like this?

type StaticPodInstallerPreconditionsFuncType func(ctx context.Context) (bool, error)

and then the struct value could look like this

installPrecondition StaticPodInstallerPreconditionsFuncType

that way is more consistent with the other packages.

dgrisonnet · 2024-06-28T13:14:24Z

pkg/operator/staticpod/controller/installer/installer_controller.go

+// returns whether or not requeue and if an error happened while creating installer pod
+func (c *InstallerController) ensureInstallerPod(ctx context.Context, operatorSpec *operatorv1.StaticPodOperatorSpec, ns *operatorv1.NodeStatus) (bool, error) {
+	// checks if new revision should be rolled out
+	if c.shouldRevisionInstall != nil {


I think this check should be moved to manageInstallationPods because this function is supposed to be only about creating the installer pod.

we could maybe even move it inside the sync function before the call to manageInstallationPods since we want to skip the installation process entirely when the preconditions are not fulfilled.

yeah we thought so, but it's better to continue the current rollout. See #1749 (comment)

good point. it made me notice that we might be subject to the following issue: #1749 (comment)

dusk125 · 2024-06-28T20:53:31Z

lgtm, but I'll let Damien have the last word

dgrisonnet · 2024-07-02T14:50:42Z

pkg/operator/staticpod/controller/installer/installer_controller.go

+			return true, err
+		}
+		if !shouldInstall {
+			return true, nil


Let's imagine the following scenario:
t0: sync for a new revision occurs, preconditions are true, the installer pod is created
t1: preconditions became false while installation is still in progress
t2: sync happens again, ensureInstallerPod returns true for requeuing because the installer pod can't be installed due to the preconditions not being met.

When that happens, we return early in

library-go/pkg/operator/staticpod/controller/installer/installer_controller.go

Lines 500 to 503 in f152bd5

if requeue {

klog.V(4).Infof("Requeuing the creation of installer pod for revision %d on node %q", currNodeState.TargetRevision, currNodeState.NodeName)

return true, 0, nil

}

which means that the statuses will not be updated unless the preconditions are met again.

To prevent the whole flow from being dependent on the preconditions, we should make sure that the preconditions are only evaluated when the installer pod isn't already present.

maybe I'm off, but wouldn't t2 not go through the ensureInstallerPod branch? the previous if condition should be true?

if operatorStatus.LatestAvailableRevision > currNodeState.TargetRevision { // no backoff if new revision is pending }

@jubittajohn can we add a unit test for the condition that @dgrisonnet mentioned? I think he has a valid point that we definitely must update the node status for ongoing installations, even when the precondition is false.

@dgrisonnet As pointed out , ensureInstallerPod returns true at t2.
@tjungblu The if condition mentioned would evaluate to false at t2 allowing the installer creation process to continue. The mentioned condition only evaluates to true when a later revision is pending compared to the one currently being rolled out. Therefore, the ensureInstallerPod is invoked.

To address this, I have added a check as suggested to ensure that preconditions are only evaluated when the installer pod isn't already present. Additionally, I have added a unit test for this scenario to confirm the behavior.

Could you please review the new changes ?

looks good, thanks for the test case :)

pkg/operator/staticpod/controller/installer/installer_controller.go

tjungblu · 2024-07-09T08:03:49Z

pkg/operator/staticpod/controller/installer/installer_controller.go

+				return true, err
+			}
+			if !shouldInstall {
+				return true, nil


can we leave a log statement here, too, please?

Suggested change

return true, nil

klog.Infof("Skipping the creation of installer pod [%s] because of precondition check", installerPodName)

return true, nil

just as an example, feel free to find a more fitting log statement

Signed-off-by: jubittajohn <jujohn@redhat.com> Added unit test Signed-off-by: jubittajohn <jujohn@redhat.com> Preconditions are only evaluated when the installer pod isn't already present Signed-off-by: jubittajohn <jujohn@redhat.com>

openshift-ci · 2024-07-09T14:28:30Z

@jubittajohn: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 14, 2024

openshift-ci bot requested review from dgrisonnet and tkashem June 14, 2024 18:46

jubittajohn changed the title ~~WIP: Added the callback shouldRevisionInstall to provide additional c…~~ WIP: ETCD-612: Added the callback shouldRevisionInstall to provide additional c… Jun 14, 2024

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jun 14, 2024

jubittajohn mentioned this pull request Jun 14, 2024

WIP: ETCD-612: Added a function to check if the quorum is safe openshift/cluster-etcd-operator#1278

Open

lance5890 reviewed Jun 15, 2024

View reviewed changes

pkg/operator/staticpod/controller/installer/installer_controller.go Outdated Show resolved Hide resolved

tjungblu reviewed Jun 17, 2024

View reviewed changes

pkg/operator/staticpod/controller/installer/installer_controller.go Outdated Show resolved Hide resolved

tjungblu reviewed Jun 17, 2024

View reviewed changes

jubittajohn force-pushed the skip-static-pod-rollouts branch 3 times, most recently from bf42b26 to d1afb2e Compare June 21, 2024 14:07

jubittajohn requested review from tjungblu and lance5890 June 21, 2024 14:20

tjungblu reviewed Jun 21, 2024

View reviewed changes

pkg/operator/staticpod/controllers.go Outdated Show resolved Hide resolved

tjungblu reviewed Jun 21, 2024

View reviewed changes

openshift-ci bot assigned dusk125 Jun 21, 2024

jubittajohn force-pushed the skip-static-pod-rollouts branch from d1afb2e to 8e7bd43 Compare June 25, 2024 14:42

tjungblu reviewed Jun 25, 2024

View reviewed changes

pkg/operator/staticpod/controller/installer/installer_controller.go Outdated Show resolved Hide resolved

jubittajohn force-pushed the skip-static-pod-rollouts branch from 54dda05 to ddabfa2 Compare June 25, 2024 20:12

jubittajohn requested a review from tjungblu June 25, 2024 20:24

lance5890 reviewed Jun 26, 2024

View reviewed changes

pkg/operator/staticpod/controller/installer/installer_controller.go Outdated Show resolved Hide resolved

jubittajohn force-pushed the skip-static-pod-rollouts branch from 5f21a8d to 713d859 Compare June 26, 2024 13:51

openshift-ci bot assigned dgrisonnet Jun 26, 2024

jubittajohn force-pushed the skip-static-pod-rollouts branch from 713d859 to 8d7fddd Compare June 26, 2024 14:44

jubittajohn changed the title ~~WIP: ETCD-612: Added the callback shouldRevisionInstall to provide additional c…~~ ETCD-612: Added the callback shouldRevisionInstall to provide additional conditions for installation Jun 26, 2024

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 26, 2024

dgrisonnet reviewed Jun 28, 2024

View reviewed changes

jubittajohn changed the title ~~ETCD-612: Added the callback shouldRevisionInstall to provide additional conditions for installation~~ ETCD-612: Added a callback to provide additional pre-conditions for installation Jun 28, 2024

jubittajohn force-pushed the skip-static-pod-rollouts branch from 119fa6c to f152bd5 Compare June 28, 2024 20:42

jubittajohn requested a review from dgrisonnet June 28, 2024 20:44

dgrisonnet reviewed Jul 2, 2024

View reviewed changes

jubittajohn requested review from dgrisonnet and lance5890 July 8, 2024 17:54

tjungblu reviewed Jul 9, 2024

View reviewed changes

pkg/operator/staticpod/controller/installer/installer_controller.go Outdated Show resolved Hide resolved

tjungblu reviewed Jul 9, 2024

View reviewed changes

jubittajohn force-pushed the skip-static-pod-rollouts branch from 424dac8 to a73116f Compare July 9, 2024 14:10

jubittajohn force-pushed the skip-static-pod-rollouts branch from a73116f to 4b49b84 Compare July 9, 2024 14:13

jubittajohn requested a review from tjungblu July 9, 2024 14:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ETCD-612: Added a callback to provide additional pre-conditions for installation #1749

ETCD-612: Added a callback to provide additional pre-conditions for installation #1749

jubittajohn commented Jun 14, 2024 •

edited

Loading

openshift-ci-robot commented Jun 14, 2024 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Jun 14, 2024 •

edited by openshift-ci bot

Loading

tjungblu Jun 17, 2024

tjungblu Jun 21, 2024

tjungblu Jun 21, 2024

jubittajohn Jun 21, 2024

tjungblu Jun 21, 2024

tjungblu Jun 21, 2024

tjungblu Jun 21, 2024

jubittajohn commented Jun 21, 2024

openshift-ci-robot commented Jun 25, 2024 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Jun 26, 2024 •

edited by openshift-ci bot

Loading

tjungblu commented Jun 26, 2024

tjungblu commented Jun 26, 2024

openshift-ci bot commented Jun 26, 2024

dgrisonnet Jun 28, 2024

tjungblu Jun 28, 2024

dgrisonnet Jun 28, 2024

dgrisonnet Jun 28, 2024

tjungblu Jun 28, 2024

dgrisonnet Jul 2, 2024

dusk125 commented Jun 28, 2024

dgrisonnet Jul 2, 2024

tjungblu Jul 2, 2024

jubittajohn Jul 8, 2024 •

edited

Loading

tjungblu Jul 9, 2024

tjungblu Jul 9, 2024

tjungblu Jul 9, 2024

tjungblu Jul 9, 2024

openshift-ci bot commented Jul 9, 2024

-		if !shouldInstall {
+		if err != nil {
+    		   return err
+		}
+		if !shouldInstall {
+		  return nil
+		}

	if requeue {
	klog.V(4).Infof("Requeuing the creation of installer pod for revision %d on node %q", currNodeState.TargetRevision, currNodeState.NodeName)
	return true, 0, nil
	}

	return true, nil
	klog.Infof("Skipping the creation of installer pod [%s] because of precondition check", installerPodName)
	return true, nil

ETCD-612: Added a callback to provide additional pre-conditions for installation #1749

Are you sure you want to change the base?

ETCD-612: Added a callback to provide additional pre-conditions for installation #1749

Conversation

jubittajohn commented Jun 14, 2024 • edited Loading

openshift-ci-robot commented Jun 14, 2024 • edited by openshift-ci bot Loading

openshift-ci-robot commented Jun 14, 2024 • edited by openshift-ci bot Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jubittajohn commented Jun 21, 2024

openshift-ci-robot commented Jun 25, 2024 • edited by openshift-ci bot Loading

openshift-ci-robot commented Jun 26, 2024 • edited by openshift-ci bot Loading

tjungblu commented Jun 26, 2024

tjungblu commented Jun 26, 2024

openshift-ci bot commented Jun 26, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dusk125 commented Jun 28, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jubittajohn Jul 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

openshift-ci bot commented Jul 9, 2024

jubittajohn commented Jun 14, 2024 •

edited

Loading

openshift-ci-robot commented Jun 14, 2024 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Jun 14, 2024 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Jun 25, 2024 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Jun 26, 2024 •

edited by openshift-ci bot

Loading

jubittajohn Jul 8, 2024 •

edited

Loading