Introduce an EndpointsWatcher cache structure

[mateiidavid]

Currently, the control plane does not support indexing and discovering resources across cluster boundaries. In a multicluster set-up, it is desirable to have access to endpoint data (and by extension, any configuration associated with that endpoint) to support pod-to-pod communication. Linkerd's destination service should be extended to support reading discovery information from multiple sources (i.e. API Servers).

This change introduces an EndpointsWatcher cache. On creation, the cache registers a pair of event handlers:

• One handler to read multicluster secrets that embed kubeconfig files. For each such secret, the cache creates a corresponding EndpointsWatcher to read remote discovery information.
• Another handle to evict entries from the cache when a multicluster secret is deleted.

Additionally, a set of tests have been added that assert the behaviour of the cache when secrets are created and/or deleted. The cache will be used by the destination service to do look-ups for services that belong to another cluster, and that are running in a "remote discovery" mode.

---

Signed-off-by: Matei David matei@buoyant.io

[adleong]

direction looks good here. we'll want to be able to stop the informers when the secret is deleted, which I think we can do with the channel passed into Sync. we may also want to add a Stop function to EndpointsWatcher to unregister the event listeners to avoid memory leaks, but maybe stopping the informers and removing the EndpointsWatcher from the cache is sufficient to getting it all eligible for gc.

[adleong]

we could even go a step further and sync each of these in their own goroutine to sync them in parallel.

[alpeb]

Following up on our off-github convo, I think the race occurs between these two calls NewFakeAPI and NewFakeMetadataAPI, because they rely on a call to ToRuntimeObject() which uses this global struct instance from scheme/register.go, which holds the map that is contended over:

var Scheme = runtime.NewScheme()

Maybe a simple fix could be to have ToRuntimeObject() receive as an argument the scheme instance, which could be instantiated separately for these two API clients.

[alpeb]

I think this line is not needed.

[alpeb]

Why is that we're not interested in Update events? Can you add a comment about that?

[alpeb]

You can slim down InitializeMetadataAPI body now:

func InitializeMetadataAPI(kubeConfig string, resources ...APIResource) (*MetadataAPI, error) {
	config, err := k8s.GetConfig(kubeConfig, "")
	if err != nil {
		return nil, fmt.Errorf("error configuring Kubernetes API client: %w", err)
	}

        return InitializeMetadataAPIForConfig(config, resources...)
}

[alpeb]

Exciting!

Left some non-critical comments above. The one thing that might be annoying are the sporadic test failures though.

Also, fyi, EndpointSlices entered GA in k8s 1.21, which is our oldest supported version, at which point the EndpointSlices feature gate defaults to true. And in more recent versions the feature gate is gone altogether. So you could just assume that's enabled to simplify things, but leaving that at your discretion 😉