clustermesh: traffic continues to be routed to unresponsive cluster #20729
Labels
kind/bug
This is a bug in the Cilium logic.
needs/triage
This issue requires triaging to establish severity and next steps.
sig/datapath
Impacts bpf/ or low-level forwarding details, including map management and monitor messages.
stale
The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale.
Comments
shadowspore
added
kind/bug
aanm
added
the
sig/datapath
|
This issue has been automatically marked as stale because it has not |
github-actions
bot
added
the
stale
|
This issue has not seen any activity since it was marked stale. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is there an existing issue for this?
What happened?
Two clusters in mesh created by
kindtool (repo). Cilium clustermesh connectivity tests was passed.Both clusters have http echo server deployments, exposed using kubernetes service with
io.cilium/global-service: "true"annotation.Clients are running inside of each cluster and sending requests to the service.
Similar to how x-wings access the rebel-base in the examples.
Cluster1 cilium status --verbose output
Cluster1 cilium-health status --probe output
root@mesh-1-control-plane:/home/cilium# cilium-health status --probe Probe time: 2022-08-01T10:54:39Z Nodes: mesh-1/mesh-1-control-plane (localhost): Host connectivity to 172.18.0.3: ICMP to stack: OK, RTT=1.167103ms HTTP to agent: OK, RTT=146.093µs Endpoint connectivity to 10.1.0.108: ICMP to stack: OK, RTT=1.187763ms HTTP to agent: OK, RTT=281.742µs mesh-1/mesh-1-worker: Host connectivity to 172.18.0.2: ICMP to stack: OK, RTT=1.18302ms HTTP to agent: OK, RTT=313.354µs Endpoint connectivity to 10.1.1.72: ICMP to stack: OK, RTT=1.210518ms HTTP to agent: OK, RTT=476.553µs mesh-2/mesh-2-control-plane: Host connectivity to 172.18.0.5: ICMP to stack: OK, RTT=1.214629ms HTTP to agent: OK, RTT=467.258µs Endpoint connectivity to 10.2.0.55: ICMP to stack: OK, RTT=1.140983ms HTTP to agent: OK, RTT=687.659µs mesh-2/mesh-2-worker: Host connectivity to 172.18.0.4: ICMP to stack: OK, RTT=1.225698ms HTTP to agent: OK, RTT=331.337µs Endpoint connectivity to 10.2.1.86: ICMP to stack: OK, RTT=1.183777ms HTTP to agent: OK, RTT=461.408µsCluster2 cilium status --verbose output
Cluster2 cilium-health status --probe output
root@mesh-2-control-plane:/home/cilium# cilium-health status --probe Probe time: 2022-08-01T10:56:38Z Nodes: mesh-2/mesh-2-control-plane (localhost): Host connectivity to 172.18.0.5: ICMP to stack: OK, RTT=1.117653ms HTTP to agent: OK, RTT=228.252µs Endpoint connectivity to 10.2.0.55: ICMP to stack: OK, RTT=1.079569ms HTTP to agent: OK, RTT=425.034µs mesh-1/mesh-1-control-plane: Host connectivity to 172.18.0.3: ICMP to stack: OK, RTT=1.03381ms HTTP to agent: OK, RTT=469.295µs Endpoint connectivity to 10.1.0.108: ICMP to stack: OK, RTT=1.082403ms HTTP to agent: OK, RTT=671.207µs mesh-1/mesh-1-worker: Host connectivity to 172.18.0.2: ICMP to stack: OK, RTT=1.059099ms HTTP to agent: OK, RTT=370.719µs Endpoint connectivity to 10.1.1.72: ICMP to stack: OK, RTT=1.080848ms HTTP to agent: OK, RTT=671.118µs mesh-2/mesh-2-worker: Host connectivity to 172.18.0.4: ICMP to stack: OK, RTT=1.114142ms HTTP to agent: OK, RTT=331.615µs Endpoint connectivity to 10.2.1.86: ICMP to stack: OK, RTT=1.124719ms HTTP to agent: OK, RTT=535.21µsThe problem is, if one of the clusters becomes unresponsive (
docker pausecluster1 nodes in this case), cilium continues routing traffic to that cluster.Clients start getting http timeout errors.
Health probe is not working:
Cluster2 cilium status --verbose output after cluster1 outage
Cilium Version
Kernel Version
Linux 5.15.0-41-generic
Kubernetes Version
Sysdump
cilium-sysdump-20220801-111040.zip
Relevant log output
No response
Anything else?
No response
Code of Conduct
The text was updated successfully, but these errors were encountered: