Fix potential deadlock in pod identity updates

[jrajahalme]

Make IdentitySelectionUpdated() callbacks lock-free by queueing them
while still holding selectorcache lock (to keep FIFO order) and
calling from a goroutine not holding any locks. This prevents
deadlocks caused by the implementation of IdentitySelectionUpdated()
taking locks such as endpoint or selectorcache locks.

Potential deadlock in pod identity updates has been fixed.

Fixes: #16236
Fixes: #16300

[jrajahalme]

test-me-please

[aanm]

the unit tests have failed:

 	 # github.com/cilium/cilium/pkg/fqdn/dnsproxy [github.com/cilium/cilium/pkg/fqdn/dnsproxy.test]
	 pkg/fqdn/dnsproxy/proxy_test.go:184:36: not enough arguments in call to testSelectorCache.UpdateIdentities
	 	have ("github.com/cilium/cilium/pkg/identity/cache".IdentityCache, nil)
	 	want ("github.com/cilium/cilium/pkg/identity/cache".IdentityCache, "github.com/cilium/cilium/pkg/identity/cache".IdentityCache, *sync.WaitGroup)
	 make: *** [tests-privileged] Error 1

[jrajahalme]

test-me-please

[jrajahalme]

the unit tests have failed:

 	 # github.com/cilium/cilium/pkg/fqdn/dnsproxy [github.com/cilium/cilium/pkg/fqdn/dnsproxy.test]
	 pkg/fqdn/dnsproxy/proxy_test.go:184:36: not enough arguments in call to testSelectorCache.UpdateIdentities
	 	have ("github.com/cilium/cilium/pkg/identity/cache".IdentityCache, nil)
	 	want ("github.com/cilium/cilium/pkg/identity/cache".IdentityCache, "github.com/cilium/cilium/pkg/identity/cache".IdentityCache, *sync.WaitGroup)
	 make: *** [tests-privileged] Error 1

Fixed, thanks!

[jrajahalme]

test-me-please

[jrajahalme]

Test flake #14598 hit in test-1.21-4.9, seems unrelated (deathstar.default.svc.cluster.local not resolved by kube-dns even though the service exists and has 3 backends).