egressgw: retry getIdentityLabels on failure

[jibi]

when the manager fails to resolve the identity of an endpoint to a set of labels, instead of discarding the endpoint update, enqueue the event and retry identity labels resolution following the same exponential backoff logic used by the identity allocator

testing

I tested this locally by forcing the identity resolution to randomly fail:

diff --git a/pkg/egressgateway/manager.go b/pkg/egressgateway/manager.go
index ed7f19399c..0b867b2d12 100644
--- a/pkg/egressgateway/manager.go
+++ b/pkg/egressgateway/manager.go
@@ -7,6 +7,7 @@ import (
        "container/heap"
        "context"
        "fmt"
+       "math/rand"
        "net"
        "sort"
        "time"
@@ -171,6 +172,10 @@ func NewEgressGatewayManager(p Params) *Manager {
 // getIdentityLabels waits for the global identities to be populated to the cache,
 // then looks up identity by ID from the cached identity allocator and return its labels.
 func (manager *Manager) getIdentityLabels(securityIdentity uint32) (labels.Labels, error) {
+       if rand.Intn(16) > 0 {
+               return nil, fmt.Errorf("fake error")
+       }
+

➜  cilium git:(pr/jibi/egressgw-retry-identity) ks logs cilium-2dc5v --timestamps | grep subsys=egress | grep coredns-5d78c9869d-44vlf
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init), install-cni-binaries (init)
2023-06-23T10:22:36.698593375Z level=warning msg="Failed to get identity labels for endpoint, will retry." error="fake error" k8sEndpointName=coredns-5d78c9869d-44vlf k8sNamespace=kube-system subsys=egressgateway
2023-06-23T10:22:37.590584472Z level=warning msg="Failed to get identity labels for endpoint, will retry." error="fake error" k8sEndpointName=coredns-5d78c9869d-44vlf k8sNamespace=kube-system subsys=egressgateway
2023-06-23T10:22:42.630015783Z level=warning msg="Failed to get identity labels for endpoint, will retry." error="fake error" k8sEndpointName=coredns-5d78c9869d-44vlf k8sNamespace=kube-system subsys=egressgateway
2023-06-23T10:22:44.710650728Z level=warning msg="Failed to get identity labels for endpoint, will retry." error="fake error" k8sEndpointName=coredns-5d78c9869d-44vlf k8sNamespace=kube-system subsys=egressgateway
2023-06-23T10:22:46.871488814Z level=warning msg="Failed to get identity labels for endpoint, will retry." error="fake error" k8sEndpointName=coredns-5d78c9869d-44vlf k8sNamespace=kube-system subsys=egressgateway
2023-06-23T10:22:52.190670670Z level=warning msg="Failed to get identity labels for endpoint, will retry." error="fake error" k8sEndpointName=coredns-5d78c9869d-44vlf k8sNamespace=kube-system subsys=egressgateway
2023-06-23T10:22:55.113841789Z level=warning msg="Failed to get identity labels for endpoint, will retry." error="fake error" k8sEndpointName=coredns-5d78c9869d-44vlf k8sNamespace=kube-system subsys=egressgateway
2023-06-23T10:22:58.952487945Z level=debug msg="Added CiliumEndpoint" k8sEndpointName=coredns-5d78c9869d-44vlf k8sNamespace=kube-system subsys=egressgateway

Fixes: #26158 (hopefully 🤞)

[ti-mo]

I got pinged for TH review on pkg/backoff, could you assign a CODEOWNER to that package in this PR? 🙏

[bimmlerd]

CODEOWNERS change lgtm

[jibi]

Thanks for the pointers @giorio94 , I switched this PR to a workqueue.

The approach I'm taking is the follow: there's an endpointEventRetryQueue workqueue that simply tracks the ID of the CiliumEndpoint that should be processed/reprocessed, and then in a separate pendingEndpointEvents object we store the ID -> CiliumEndpoint mapping.

This ensures we can deal nicely with CiliumEndpoints that get updated/deleted while in the workqueue: whenever we receive a new event, any existing one for the same ID gets removed from the queue (as otherwise we could end up retrying processing old revisions of a given CiliumEndpoint).

tested with a random fake error:

2023-06-30T08:04:33.847665474Z level=debug msg="Added CiliumEgressGatewayPolicy" ciliumEgressGatewayPolicyName=egress subsys=egressgateway
2023-06-30T08:04:34.300917210Z level=warning msg="Failed to get identity labels for endpoint" error="fake error" k8sEndpointName=ubuntu-1 k8sNamespace=default subsys=egressgateway
2023-06-30T08:04:34.300955321Z level=warning msg="Failed to get identity labels for endpoint" error="fake error" k8sEndpointName=ubuntu-1 k8sNamespace=default subsys=egressgateway
2023-06-30T08:04:34.321451715Z level=warning msg="Failed to get identity labels for endpoint" error="fake error" k8sEndpointName=ubuntu-1 k8sNamespace=default subsys=egressgateway
2023-06-30T08:04:34.361744042Z level=warning msg="Failed to get identity labels for endpoint" error="fake error" k8sEndpointName=ubuntu-1 k8sNamespace=default subsys=egressgateway
2023-06-30T08:04:34.442374668Z level=warning msg="Failed to get identity labels for endpoint" error="fake error" k8sEndpointName=ubuntu-1 k8sNamespace=default subsys=egressgateway
2023-06-30T08:04:34.602854941Z level=warning msg="Failed to get identity labels for endpoint" error="fake error" k8sEndpointName=ubuntu-1 k8sNamespace=default subsys=egressgateway
2023-06-30T08:04:34.923616503Z level=warning msg="Failed to get identity labels for endpoint" error="fake error" k8sEndpointName=ubuntu-1 k8sNamespace=default subsys=egressgateway
2023-06-30T08:04:35.564425666Z level=warning msg="Failed to get identity labels for endpoint" error="fake error" k8sEndpointName=ubuntu-1 k8sNamespace=default subsys=egressgateway
2023-06-30T08:04:36.845466966Z level=warning msg="Failed to get identity labels for endpoint" error="fake error" k8sEndpointName=ubuntu-1 k8sNamespace=default subsys=egressgateway
2023-06-30T08:04:39.405585974Z level=debug msg="Added CiliumEndpoint" k8sEndpointName=ubuntu-1 k8sNamespace=default subsys=egressgateway
2023-06-30T08:04:39.405952893Z level=debug msg="Egress gateway policy applied" destinationCIDR=0.0.0.0/0 egressIP=0.0.0.0 gatewayIP=0.0.0.0 sourceIP=10.244.2.32 subsys=egressgateway
2023-06-30T08:04:39.405959833Z level=debug msg="Egress gateway policy applied" destinationCIDR=8.8.8.8/32 egressIP=0.0.0.0 gatewayIP=0.0.0.1 sourceIP=10.244.2.32 subsys=egressgateway

[giorio94]

The workqueue implementation looks good to me.
I wonder whether it could make sense to leverage it also during insertion for increased uniformity. WDYT?

[giorio94]

Looks great to me!

[jibi]

/test