Proposal: Declarative Configuration of Multi-Cluster Ingress

[samcgardner]

At the moment, it does not appear to be possible to configure multi-cluster Ingress in a purely declarative fashion (i.e. via an Ingress resource). I'm going to summarise the state of play as I understand it and then propose a solution (which I am willing to implement).

There is currently a tool and Ingress class associated with multi-cluster Ingress, namely kubemci and the related class gce-multi-cluster. These tools assume the following workflow:

1. Write an Ingress object
2. Build a kubeconfig file listing all target clusters
3. Run kubemci to apply this Ingress to all target clusters

This appears to work well in my testing, but is problematic from an operational standpoint. My use-case relies heavily on declarative configuration, and the requirement to run kubemci repeatedly is challenging. Additionally, kubemci does not support Network Endpoint Groups which, reading between the lines, appear to be the focus of Google's efforts to support cluster Ingress moving forwards.

Accordingly, I think a new Ingress class that leverages NEGs to configure multi-cluster Ingress would be useful for me and other GKE users who rely on declarative configuration but wish to serve multiple clusters from an Ingress. I'd like to propose a new Ingress class which will attempt to configure itself in a purely declarative fashion as follows:

1. Read back all existing L7 load balancers and see if one exists for the Ingress. If not, create it.
2. Ensure a target proxy exists for the load balancer.
3. Ensure a backend service exists for the load balancer.
4. Ensure that NEGs for this cluster only exist and are registered with the backend service.

I believe this should be sufficient to allow declarative configuration of multi-cluster Ingress, but before I drop a PR on you folks I'd like to check a) to see whether you have any feedback on this proposal (as I'm sure you understand the internal of the GCLB much better than I do) and b) that you have bandwidth to review.

Thanks for reading!

[rramkumar1]

Adding @bowei so he can respond to this.

/assign @bowei

[bowei]

How do the other cluster's endpoints get added to the LB in this case?

[samcgardner]

The core idea is for the controller in each cluster to set the backend service's endpoints to current contents ∪ that cluster's endpoints. To outline the full procedure in a simple two-cluster case with no concurrency issues:

1. Cluster 0 is created with a declarative multi-cluster Ingress configured
2. Ingress controller in cluster zero (I₀ hereafter for brevity) creates a global forwarding rule
3. I₀ creates a target proxy
4. I₀ creates a URL map
5. I₀ creates a backend service
6. I₀ creates NEGs (NEG₀)
7. I₀ sets the backends for the backend service to be the current contents (i.e. {}) ∪ NEG₀
8. Cluster 1 is created with the same Ingress object configured
9. I₁ skips creating a target proxy, URL map, and backend service as these already exist
10. I₁ sets backends for the backend service to be NEG₀ ∪ NEG₁

The weaknesses I can see in this are:

1. Target proxies, URL maps, and backend servies will need to either have configurable names or very predictable ones, and there's a risk of clobbering things as the controller can't know what it does or doesn't manage
2. NEGs will never get cleaned up, so use-cases which involve actually removing NEGs from a backend service will leak them unless they actually delete the NEG itself this is nonsense, the Ingress manages creating the NEGs so its finalizer can clean them up

[wy100101]

I'm interested in this because the current solution is so close to what I need OOTB. Today I can declaratively create a GCLB for a service running in GKE just by creating a service with the right annotation, and the corresponding ingress. Unfortunately, I can't create the same deployment in another cluster in another region that automatically gets added to the existing GCLB as additional backends.

I've tried adding the ingress using the same reserved IP and that doesn't work. I can add the negs to original LB, but that gets overwritten eventually, and probably isn't practical.

I just need a way to say add this service as a backend to an existing LB through some sort of annotation, and I'll have exactly what I need.

[samcgardner]

I'm closing this because I think it's pretty clear at this juncture that this issue is not a going concern.