operator: fix deadlock when running in kvstore mode

When running in kvstore mode, the start hook of the identity GC cell
blocks until the kvstore client has been initialized, which is performed
by the legacyCell start hook. Given that the identity GC cell was
registered first, and there are no explicit dependencies among the two,
its start hook was also executed first, causing a deadlock.

This commit changes the order in which the cells are registered as a
workaround, until the kvstore is refactored into a proper cell.

The current hooks execution order is the following:

function="gops.registerGopsHooks.func1 (cell.go:44)"
function="client.(*compositeClientset).onStart"
function="cmd.registerOperatorHooks.func1 (root.go:142)"
function="*resource.resource[*github.com/cilium/cilium/pkg/k8s/apis/cilium.io/v2alpha1.CiliumLoadBalancerIPPool].Start"
function="*resource.resource[*github.com/cilium/cilium/pkg/k8s/slim/k8s/api/core/v1.Service].Start"
function="*lbipam.LBIPAM.Start"
function="*resource.resource[*k8s.io/api/core/v1.Node].Start"
function="*resource.resource[*github.com/cilium/cilium/pkg/k8s/apis/cilium.io/v2.CiliumNode].Start"
function="*resource.resource[*github.com/cilium/cilium/pkg/k8s/slim/k8s/api/core/v1.Namespace].Start"
function="*resource.resource[*github.com/cilium/cilium/pkg/k8s/apis/cilium.io/v2.CiliumIdentity].Start"
function="cmd.(*legacyOnLeader).onStart"
function="identitygc.registerGC.func1 (gc.go:107)"

Fixes: b115951 ("operator: Refactor cilium identities GC to a cell")
Signed-off-by: Marco Iorio <marco.iorio@isovalent.com>

operator/cmd/root.go

@@ -112,9 +112,16 @@ var (
 			client.RegisterCRDsCell,
 			k8s.SharedResourcesCell,
 			lbipam.Cell,
-			identitygc.Cell,
 
 			legacyCell,
+
+			// When running in kvstore mode, the start hook of the identity GC
+			// cell blocks until the kvstore client has been initialized, which
+			// is performed by the legacyCell start hook. Hence, the identity GC
+			// cell is registered afterwards, to ensure the ordering of the
+			// setup operations. This is a hacky workaround until the kvstore is
+			// refactored into a proper cell.
+			identitygc.Cell,
 		),
 	)