New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

🐛 Power off nodes upon deletion #1176

Merged

metal3-io-bot merged 1 commit into metal3-io:main from honza:bmh-delete-2022-10-20

Jul 6, 2023

Member

honza commented Oct 20, 2022

This is a continuation of #816 which in turn tries to fix #410.

Co-authored-by: Sandhya Dasu sadasu@redhat.com @sadasu

metal3-io-bot added the size/L label

Member

elfosardo commented Oct 20, 2022

/test-centos-integration-main
/test-ubuntu-integration-main

Member

elfosardo commented Oct 20, 2022

/lgtm

metal3-io-bot added the lgtm label

Member

furkatgofurov7 commented Oct 20, 2022

Heads up: centOS failure is unrelated to these changes, we are facing issues with CI.

xref: https://kubernetes.slack.com/archives/CHD49TLE7/p1666273231196429

zhouhao3 reviewed

View reviewed changes

pkg/provisioner/ironic/delete_test.go Outdated Show resolved Hide resolved

Cellebyte commented Nov 8, 2022

what is the progress here?

honza force-pushed the bmh-delete-2022-10-20 branch from 72efc2a to 8b41131 Compare

November 29, 2022 14:11

metal3-io-bot removed the lgtm label

Member

elfosardo commented Nov 29, 2022

/test-centos-integration-main
/test-ubuntu-integration-main

Member

elfosardo commented Nov 29, 2022

/lgtm

metal3-io-bot added the lgtm label

dtantsur reviewed

View reviewed changes

pkg/provisioner/ironic/ironic.go Outdated Show resolved Hide resolved

Member

dtantsur commented Dec 5, 2022

/lgtm cancel

One issue inline, otherwise looking good.

metal3-io-bot removed the lgtm label

zaneb reviewed

View reviewed changes

controllers/metal3.io/baremetalhost_controller.go Outdated Show resolved Hide resolved

pkg/provisioner/ironic/ironic.go Outdated Show resolved Hide resolved

honza force-pushed the bmh-delete-2022-10-20 branch from 8b41131 to 2f4593e Compare

December 19, 2022 19:42

metal3-io-bot added size/M and removed size/L labels

zaneb reviewed

View reviewed changes

pkg/provisioner/ironic/ironic.go Outdated Show resolved Hide resolved

honza force-pushed the bmh-delete-2022-10-20 branch from 2f4593e to 277a166 Compare

December 19, 2022 20:31

zaneb reviewed

View reviewed changes

Member

zaneb left a comment

Refamiliarising myself with how all of this works 😅

controllers/metal3.io/baremetalhost_controller.go Outdated Show resolved Hide resolved

controllers/metal3.io/baremetalhost_controller.go Outdated Show resolved Hide resolved

controllers/metal3.io/baremetalhost_controller.go Outdated Show resolved Hide resolved

controllers/metal3.io/baremetalhost_controller.go Outdated Show resolved Hide resolved

honza force-pushed the bmh-delete-2022-10-20 branch from 277a166 to 77fdf84 Compare

March 17, 2023 18:30

zaneb reviewed

View reviewed changes

controllers/metal3.io/baremetalhost_controller.go Outdated

+              		if err != nil {
+              			if info.host.Status.ErrorCount < maxPowerOffRetryCount {
+              				return actionError{errors.Wrap(err, "failed to power off")}

Member

zaneb Mar 23, 2023

I'm not seeing anything addressing my earlier comment about an infinite error loop in cases where the node is missing from ironic and we don't have credentials to re-register it, which we handle in deprovisioning here and also now need to handle in deleting.

I wonder if this would all be made simpler by putting the new code into a separate actionPowerOffBeforeDeleting() method, so that the state machine code can easily distinguish between errors coming from the power off vs. the delete.

controllers/metal3.io/baremetalhost_controller.go Outdated

+              		}
+              	}
+              	info.host.Status.ErrorCount = 0

Member

zaneb Mar 23, 2023

As mentioned in the last part of https://github.com/metal3-io/baremetal-operator/pull/1176/files#r1053444986, iff the error count wasn't already 0 we'll want to return actionUpdate wrapping actionContinue on line 533.

controllers/metal3.io/baremetalhost_controller.go Outdated

+              				}
+              				return result
+              			}
+              		}

Member

zaneb Mar 23, 2023

We should probably also set Status.PoweredOn to false so that we are reporting what we've actually done.

honza force-pushed the bmh-delete-2022-10-20 branch from 77fdf84 to b61cb84 Compare

March 23, 2023 19:01

metal3-io-bot added size/L and removed size/M labels

Member Author

honza commented Mar 23, 2023

This might be overkill but the suggested changes led me to create a new step in the state machine. When a delete is requested, instead of StateDeleting, we now enter StatePoweringOffBeforeDelete. This separates the two operations nicely, and allows us to handle errors in the usual way (i.e. as was said above regarding skipping to delete when too many errors happen, or we don't have host creds).

When we clear ErrorCount, we do it as part of the actionComplete result, and thus don't need an explicit update (as is idiomatic in the existing code).

honza force-pushed the bmh-delete-2022-10-20 branch from b61cb84 to 1630c13 Compare

March 29, 2023 17:41

Member

elfosardo commented Apr 3, 2023

/test-centos-integration-main
/test-ubuntu-integration-main

Member

elfosardo commented Apr 3, 2023

/test-centos-e2e-integration-main

dtantsur reviewed

View reviewed changes

apis/metal3.io/v1alpha1/baremetalhost_types.go Outdated Show resolved Hide resolved

dtantsur reviewed

View reviewed changes

controllers/metal3.io/host_state_machine.go Outdated Show resolved Hide resolved

honza force-pushed the bmh-delete-2022-10-20 branch from 1630c13 to 2f38157 Compare

April 20, 2023 12:23

dtantsur reviewed

View reviewed changes

controllers/metal3.io/host_state_machine.go

               	return actResult
               }
+              func (hsm *hostStateMachine) handlePoweringOffBeforeDelete(info *reconcileInfo) actionResult {
+              	actResult := hsm.Reconciler.actionPowerOffBeforeDeleting(hsm.Provisioner, info)
+              	skipToDelete := func() actionResult {

Member

dtantsur May 25, 2023

nit: most of this function is repeating handleDeprovisioning. It would be great to refactor them.

Member

dtantsur commented May 25, 2023

/test-centos-e2e-integration-main
/test-ubuntu-integration-main
/lgtm

metal3-io-bot added the lgtm label

Member Author

honza commented May 31, 2023

/assign @zaneb

metal3-io-bot assigned zaneb

metal3-io-bot added the needs-rebase label


          Power off nodes upon deletion

6f65d8e

We introduce a new step in the state machine where the node goes through
a power off stage before it's deleted.  We attempt to power it off 3
times before giving up, and proceeding to the delete.

honza force-pushed the bmh-delete-2022-10-20 branch from 2f38157 to 6f65d8e Compare

June 9, 2023 20:00

metal3-io-bot removed lgtm needs-rebase labels

Member

elfosardo commented Jun 28, 2023

/lgtm

metal3-io-bot added the lgtm label

Member

dtantsur commented Jul 6, 2023

/test-centos-e2e-integration-main
/test-ubuntu-integration-main

Member

dtantsur commented Jul 6, 2023

/approve

Contributor

metal3-io-bot commented Jul 6, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dtantsur

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [dtantsur]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

metal3-io-bot added the approved label

metal3-io-bot merged commit 7654cd7 into metal3-io:main

10 checks passed

honza mentioned this pull request

🐛 Proceed to power off after deprovisioning #1312

Merged

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment