identity: stop double-update of selector cache and regenerate when a local identity is allocated

[squeed]

As I was looking in to some issues, I discovered that we are still regenerating all endpoints whenever we allocated a new local identity. This is not necessary, since we already have an incremental policy update engine.

This has been the case for some time, but it is a lot more relevant since #29036, which moved FQDN policy over to this code-path.

Background:

The AllocateIdentity() function takes a parameter, notifyOwner. When this is true, any newly allocated identities are passed to the policy engine by the identity allocator and an endpoint regeneration is triggered. However, there is a funny inconsistency: if the identity is local (i.e. a CIDR identity), then notifyOwner is ignored and the policy engine is always updated. Incidentally, as part of updating the policy engine, all endpoints are regenerated.

However, the only source of local (CIDR) identities now is the IPCache, which already has its own policy update mechanism. There is one missed case -- fixed in this PR -- so there really is no need to ignore notifyOwner. This is, also, the only caller where AllocateIdentity(notifyOwner = false).

The fix:

1. Push allocated identities in to the policy engine with the legacy / synchronous ipcache AllocateCIDRs() / releaseCIDRIdentities() calls too. These methods are only have one caller: the ServiceHandler that translates toServices: policy rules to a CIDR selector.
2. Make the local identity allocator respect notifyOwner, and don't pass policy updates when it is false.

What's changed?

It's useful to look at the flow, just to understand the full context of what's changed

Before:

1. Something calls ipcache.UpsertMetadata(prefix, ...)
2. The ipcache calls AllocateIdentity(labels, notifyOwner = false)
3. The allocator calls SelectorCache.UpdateIdentities(...) and RegenerateAllEndpoints()
4. The ipcache also calls SelectorCache.UpdateIdentities() and UpdatePolicyMaps()

After:

1. Something calls ipcache.UpsertMetadata(prefix, ...)
2. The ipcache calls AllocateIdentity(labels, notifyOwner = false)
3. The ipcache calls SelectorCache.UpdateIdentities() and UpdatePolicyMaps()

So, this change is safe, in that it is not possible for a new identity to be missed.

Fixes: #29681
Fixes: #29846

[squeed]

This may also fix #29846, which is an error message due to slow FQDN policy updates. Regeneration blocks incremental policy updates, which causes delays, which causes that error message.

[squeed]

/test
(edit: all green!)

[squeed]

/test
(Previous run was 100% green, just paranoia)
result: a few red. One bing.com flake, one case of #29532 (run), one possible flake that involved nothing related to this codepath - no policy, pod-to-pod.

[squeed]

/test

[gandro]

Given that this is a follow-up to a refactor which (to my knowledge) landed in v1.15, should we also backport this?

[gandro]

Looks good for IPCache (I mainly focused if the new usage is consistent with how we do things in the async path), thanks. The overall reasoning makes sense, but I'll defer the identity allocator review to other reviewers.

[squeed]

Given that this is a follow-up to a refactor which (to my knowledge) landed in v1.15, should we also backport this?

Definitely, yes, this should be backported. It is causing CI flakes and annoying lock contention.

It's also not actually directly related to the FQDN refactor. This is a long-standing bug. However, we have the confidence to fix this now that there are only a very limited set of users of the synchronous APIs.

[squeed]

/test
(result: lots of failures to 1.1.1.1; we must have hit a rate limit)

[squeed]

/test

[squeed]

All green!

[pchaigno]

Approving for ci-structure.

[joestringer]

I believe this is correct, but I raised a couple of points below for additional due diligence. The one conclusion I can come to here is that we can delete quite a lot of complicated logic once we finally migrate the k8sServiceHandler over to the new ipcache model 🎉

I also took a look over the callers who are passing the notifyOwner flag to the identity subsystem and I agree with your assessment that this should be safe / only removing a double-trigger.

I suspect that there's also a subsequent improvement to re-evaluate the role of notifications in the identity subsystem and potentially even remove them. Maybe we could prepare a short CFP to describe the design impacts of that change and float it around for discussion to confirm that's the right path forward.