Fix and test multicluster with .global stub domain

[nmittler]

As of Istio 1.8, we no longer recommend using the .global stub domain for multi-primary configurations.

It was never well tested and has been in some state of broken since 1.6, as shown in bug reports:

• Istio 1.8 not doing multicluster routing using service entry & global dns stubbing #29308
• Multicluster mTLS doesn't work between 1.6.8 <-> 1.7.1 and 1.7.1 <-> 1.7.1 versions #27321
• [multi-cluster] Raise "HTTP/1.1 503 Service Unavailable" sometimes, when send request cross clusters. #26990

We should add some minimal testing and get it working again to ease users migration story from earlier (alpha) versions of multicluster.

[ ] Docs
[ ] Installation
[x] Networking
[ ] Performance and Scalability
[ ] Extensions and Telemetry
[ ] Security
[x] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
[ ] Upgrade

Expected behavior

Steps to reproduce the bug

Version (include the output of istioctl version --remote and kubectl version --short and helm version --short if you used Helm)

How was Istio installed?

Environment where the bug was observed (cloud vendor, OS, etc)

Additionally, please consider running istioctl bug-report and attach the generated cluster-state tarball to this issue.
Refer cluster state archive for more details.

[kenthua]

Use Case: StatefulSet replicas (redis & cockroachdb) - multi-cluster / multi-network

• Works in 1.6 & 1.7
• ServiceEntry with .global host was used to refer to individual replicas in the other cluster.
• In 1.8 while pinging the host x.global resolves to a 240.x IP as set in the service entry. Traffic doesn't appear to be flowing
• If .global is going away, thoughts on how this use case may be approached withe new .svc.cluster.local path?

[howardjohn]

@nmittler is this still desired?

[nmittler]

@howardjohn yeah, but it's not a blocker for 1.9. Removed the milestone

[stevenctl]

Think we will do this in 1.9.x? Or is it 1.10

[nmittler]

Not entirely sure if we have bandwidth ... it would be good to get in 1.9.x. I suspect 1.10 at the earliest

[brian-avery]

@nmittler are we pushing this off for 1.11?

[nmittler]

@brian-avery Yes, I've updated the milestone.

[nmittler]

@stevenctl I'm assigning to you so we don't forget about this. Feel free to prioritize or re-assign.

[ryantking]

@nmittler @stevenctl Are we pushing this to 1.12?

[stevenctl]

Either pushing or dropping in favor of something else.

[rytswd]

Sorry to jump in, but I'm curious how this .global-like communication support will be treated going forward. With the remote secret and Kubernetes API server integration since v1.8, multicluster is much easier to configure and mostly automatic, but that also means it doesn't provide the granular control of exposed services. I personally have this use case of connecting from one cluster to the other, but only for a limited number of services. I managed to get this to work with ServiceEntry defined for both source and target clusters for services I need (more details in #32848 (comment)). The .global-like setup is also useful when you want to specifically hit the service in a remote cluster (.global-like, as I understand any custom domain can be used). AFAIK, the new multicluster setup using istioctl x create-remote-secret cannot achieve this. Please let me know if there is a better way!

Although it is quite cumbersome to define many ServiceEntry resources in multiple clusters, it is still great to have such an option available. I understand this is not the most standard setup, and am not sure how this plays with other multicluster related efforts and changes such as Kubernetes MCS, but possibly worth having a dedicated section in the documentation. If there is anything I can help with, I'd be more than happy to!

[stevenctl]

For granular control, we still have the clusterLocal option in mesh config although it's not the best API.

We're also adding some new locality load balancing configuration options, which again isn't the best form of policy for this since it only sets priority and doesn't completely restrict using the different endpoints. istio/api#2043

I think MCS is at the root of how we tackle this problem, but we need our own stable API for it that works with the newer multi-cluster model.

[rytswd]

Thanks for the details, those are interesting, but indeed do not provide the granular control I'm after.

Although I understand there will be further changes with MCS and other API updates, using ServiceEntry + DestinationRule + Gateway resources to get the granular multicluster control is stable from Istio API point of view, and potentially worth adding documentation for? Even with the new multicluster model Istio introduces in the future, I imagine the above resources will keep their backward compatibility, and having doc test for such use case may be helpful for migration when the new model is ready?

If that sounds like a possibility, I can work on creating a new dedicated doc with test steps to see how that works with other doc. I've never worked on the doc test, but the steps look to be well documented so I can give it a try 👍

[Kmoneal]

@stevenctl @nmittler Are we pushing this to 1.13?

[nmittler]

@Kmoneal yes, I've updated the milestone.

[nmittler]

@rytswd

I personally have this use case of connecting from one cluster to the other, but only for a limited number of services. I managed to get this to work with ServiceEntry defined for both source and target clusters for services I need (more details in #32848 (comment)). The .global-like setup is also useful when you want to specifically hit the service in a remote cluster (.global-like, as I understand any custom domain can be used). AFAIK, the new multicluster setup using istioctl x create-remote-secret cannot achieve this. Please let me know if there is a better way!

Could you expand a bit more on your use case? Are you just trying to call service endpoints in a particular cluster? Say, from a client running in Cluster-1, you want to call Service-A endpoints residing in Cluster-2 only?

[rytswd]

@nmittler

Yes, that's correct. In our use case, Cluster-1 is a strictly regulated environment, and thus it needs to know all the endpoints in a declarative manner, which allows smooth auditing. Cluster-2 has many services that Cluster-1 does not need to talk to, and thus we limit this by specifying the list of services with ServiceEntry at the moment. Also, Cluster-2 cannot know about Cluster-1, so istioctl x create-remote-secret wasn't an option.

If I'm not mistaken, the same can be achieved with the MCS ServiceExport, but that would require GKE or Submariner from my understanding. We have been using Istio for a while, so MCS was not an option back then. We are currently on an older version of Istio, working on the upgrades to start experimenting with MCS, so that may be the way forward instead.

[nmittler]

@rytswd I think we almost have the pieces necessary to do what you want in a simpler way. See #35705

[rytswd]

@nmittler Great, thanks for the pointer - just for me to better understand this, does this mean to add a new label on Service-A in Cluster-2, and add DR / VS in Cluster-2 only? I'm assuming this means the two clusters would be using istioctl x create-remote-secret - wouldn't that allow other services in Cluster-2 to be made available for Cluster-1 still? Or is my assumption incorrect?

[stevenctl]

You won't have to add the labels on the Service, just use the label as a selector in the DR/VS. IIUC The DR/VS should exist in each cluster, and the VS will have a per-cluster variation to select the "targetCluster" from the different subsets.

[rytswd]

Ah sorry I misunderstood that part, with values.global.multiClutser.clusterName, the label should be propagated to each Pod already. I'm still not clear what to provide in DR/VS for each cluster - it would be great to see some examples once the change is in

[nmittler]

So the DR will be something like:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: mysvc-dr
spec:
  host: mysvc.myns.svc.cluster.local
  subsets:
  - name: cluster-1
    labels:
      topology.istio.io/cluster: cluster-1
  - name: cluster-2
    labels:
      topology.istio.io/cluster: cluster-2

Then in the VS you can route however you'd like based on the labels of the source workload. In the example below, I make all requests stay in the same cluster (e.g. clients in cluster-1 only call endpoints that are also in cluster-1):

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: mysvc-vs
spec:
  hosts:
  - mysvc.myns.svc.cluster.local
  http:
  - name: "cluster-1-local"
    match:
    - sourceLabels:
        topology.istio.io/cluster: "cluster-1"
    route:
    - destination:
        host: mysvc.myns.svc.cluster.local
        subset: cluster-1
  - name: "cluster-2-local"
    match:
    - sourceLabels:
        topology.istio.io/cluster: "cluster-2"
    route:
    - destination:
        host: mysvc.myns.svc.cluster.local
        subset: cluster-2

[nmittler]

@stevenctl @howardjohn IIRC this is no longer feasible since a number of other changes have widened the gap with getting the stub domain working properly. I believe with workarounds such as #29335 (comment) the need for the stub domain is significantly reduced. Can we just close this?

[stevenctl]

SGTM – that seems to be where every conversation we've had ended up.