Delayed identity cleanup on operator restart

[skmatti]

Is there an existing issue for this?

• I have searched the existing issues

What happened?

On operator start up, all identities are marked as alive irrespective of whether an identity has the delete annotation

cilium/operator/identitygc/crd_gc.go

Line 52 in bb6decf

igc.heartbeatStore.markAlive(event.Object.Name, time.Now())

The delete annotation is added on an identity in

cilium/operator/identitygc/crd_gc.go

Line 125 in bb6decf

identity.Annotations[identitybackend.HeartBeatAnnotation] = timeNow.Format(time.RFC3339Nano)

As the default heartbeat timeout is 30 minutes, an identity that was marked for deletion with an annotation is marked alive on operator start up and delays the cleanup by 30 minutes.

I think an identity with the deletion annotation should not marked alive again. Is there an issue with this?.

Cilium Version

1.12

Kernel Version

not applicable

Kubernetes Version

1.27

Sysdump

No response

Relevant log output

No response

Anything else?

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[jonohart]

The larger problem that this causes is every time the operator restarts, identity GC is delayed again by at least the ID timeout value (30 mins default). The operator will restart it loses leader election due to k8s api calls failing. If this happens often enough, say once per hour, then the operator GC logic keeps delaying the GC of identities further and further in the future which allows identities to accumulate until the policy maps fill up.

[github-actions]

This issue has been automatically marked as stale because it has not
had recent activity. It will be closed if no further activity occurs.

[github-actions]

This issue has not seen any activity since it was marked stale.
Closing.

[skmatti]

@dlapcevic @aojea Is this addressed with #27752, if so can you open this and reassign to yourself.

If this needs to be handled separately, can you open so others can take a look?. Thanks!

[dlapcevic]

Yes, this will be addressed with #27752.

The plan is to immediately clean up unused Cilium Identities. They will anyway be cleaned up when operator starts.

[dlapcevic]

/assign @dlapcevic