fix: check for server power state when in use

Sometimes if power off event is followed by server allocation
immediately, IPMI might "lie" about power status of the server.

Server is being powered off, while it reports power on status.

In this case Sidero might fail to power on an allocated Server.

Workaround that by checking for power state in this state.

Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>

app/sidero-controller-manager/controllers/server_controller.go

@@ -217,7 +217,8 @@ func (r *ServerReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctr
 			}
 		}
 
-		return f(true, ctrl.Result{})
+		// keep checking power state from time to time, as sometimes IPMI lies about the power state
+		return f(true, ctrl.Result{RequeueAfter: constants.PowerCheckPeriod})
 	case !s.Status.InUse && !s.Status.IsClean:
 		// when server is set to PXE boot to be wiped, ConditionPowerCycle is set to mark server
 		// as power cycled to avoid duplicate reboot attempts from subsequent Reconciles

app/sidero-controller-manager/pkg/constants/constants.go

@@ -14,6 +14,7 @@ const (
 	InitrdAsset = "initramfs.xz"
 
 	DefaultRequeueAfter = time.Second * 20
+	PowerCheckPeriod    = 5 * time.Minute
 
 	DefaultServerRebootTimeout = time.Minute * 20