Grpc not returning Unavailable when no longer exists

[owenhaynes]

Welcome!

• Yes, I've searched similar issues on GitHub and didn't find any.
• Yes, I've searched similar issues on the Traefik community forum and didn't find any.

What did you do?

2 identical set ups

Cluster 1 
GCP internal loadbalencer
Traefik
sniStrict enabled
kubernetes
ingress route pointing to grpc backend service.  Host Rule

Cluster 2 
GCP internal loadbalencer
Traefik
sniStrict enabled
kubernetes
ingress route pointing to grpc backend service. Host Rule

DNS record which points to Cluster 1 internal loadbalancer with TTL 1

Had active GRPC connections over DNS record.

Updated the DNS to point at Cluster 2.

Deleted the ingressroute from Cluster 1

Active GRPC connections started getting 404 (Not implemented) on calls. Expecting them to redo the connection 404 is a valid grpc return code.

What did you see instead?

The active GRPC connections to be closed as the route no longer exists and tell grpc to goway for any new connections to Cluster 1 with that host name as its no longer valid for Cluster 1.

What version of Traefik are you using?

2.10.1

What is your environment & configuration?

Kubernetees

args

      - "--global.checknewversion"
      - "--global.sendanonymoususage"
      - "--providers.kubernetesingress=true"
      - "--providers.kubernetescrd.allowCrossNamespace=true"
      - "--api.dashboard=true"
      - "--accesslog=true"
      - "--accesslog.format=json"
      - "--log.level=ERROR"
      - "--log.format=json"
      - "--metrics=true"
      - "--metrics.prometheus=true"
      - "--entryPoints.websecure.http.tls=true"
      - "--entryPoints.websecure.http.tls.options=default"
      - "--serverstransport.rootcas=/run/secrets/kubernetes.io/serviceaccount/ca.crt"

If applicable, please paste the log output in DEBUG level

No response

[nmengin]

Hello @owenhaynes ,

Thanks for reaching out!

After discussing this issue with the team, we think that your problem is linked to a configuration issue.
To keep the repository focused, we ask that all questions be asked in the community forum.

It is pretty active, so you might find that your question has already been answered there. If not, you can ask and get help from other community members pretty quickly.