get "No route to host" with minikube and flannel

[STRRL]

Bug Report

It seems HTTPChaos does not work with CNI flannel.

I also tried with minikube start --cni cilium, and it works well.

What version of Kubernetes are you using?

  ~ ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── at  14:05:01
❯ minikube start --cni flannel
😄  minikube v1.25.2 on Arch "rolling"
✨  Using the docker driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=4, Memory=14000MB) ...
🐳  Preparing Kubernetes v1.23.3 on Docker 20.10.12 ...
    ▪ kubelet.housekeeping-interval=5m
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring Flannel (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
  ~ ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── took  21s at  14:06:00
❯ minikube version            
minikube version: v1.25.2
commit: 362d5fdc0a3dbee389b3d3f1034e8023e72bd3a7-dirty
  ~ ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── at ⎈ minikube at  14:06:56
❯ k version                                
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.2", GitCommit:"f66044f4361b9f1f96f0053dd46cb7dce5e990a8", GitTreeState:"archive", BuildDate:"2022-06-18T07:33:51Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.3", GitCommit:"816c97ab8cff8a1c72eccca1026f7820e93e0d25", GitTreeState:"clean", BuildDate:"2022-01-25T21:19:12Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"linux/amd64"}
  ~ ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── at ⎈ minikube at  14:06:59
❯ k cluster-info                                             
Kubernetes control plane is running at https://192.168.49.2:8443
CoreDNS is running at https://192.168.49.2:8443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

What version of Chaos Mesh are you using?

❯ k -n chaos-mesh exec chaos-controller-manager-599c7f95cb-bm4gg -- /usr/local/bin/chaos-controller-manager -version
Controller manager Version: version.Info{GitVersion:"v2.2.0", GitCommit:"22a1cf57398e35da58dd391b79b64f3071e2863b", BuildDate:"2022-04-29T13:15:01Z", GoVersion:"go1.18", Compiler:"gc", Platform:"linux/amd64"}

What did you do?

• k create deployment --image nginx:latest --replicas 3 nginx
• create this HTTPChaos:

kind: HTTPChaos
apiVersion: chaos-mesh.org/v1alpha1
metadata:
  namespace: default
  name: nginx-delay-2s
spec:
  selector:
    namespaces:
      - default
    labelSelectors:
      app: nginx
  mode: all
  target: Response
  delay: 2s
  port: 80
  path: '*'

What did you expect to see?

• the costed duration of curl from one nginx pod to another nginx pod should similar with 2 seconds

What did you see instead?

❯ k get pods -o wide                                                     
NAME                     READY   STATUS    RESTARTS   AGE     IP            NODE       NOMINATED NODE   READINESS GATES
nginx-7c658794b9-pxp8b   1/1     Running   0          2m12s   10.244.0.9    minikube   <none>           <none>
nginx-7c658794b9-s9jtl   1/1     Running   0          2m12s   10.244.0.10   minikube   <none>           <none>
nginx-7c658794b9-sppqm   1/1     Running   0          2m12s   10.244.0.8    minikube   <none>           <none>
  ~ ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── at ⎈ minikube at  14:11:17
❯ k exec -ti nginx-7c658794b9-pxp8b -- bash -c "time curl 10.244.0.8"
curl: (7) Failed to connect to 10.244.0.8 port 80: No route to host

real    0m3.053s
user    0m0.005s
sys     0m0.000s
command terminated with exit code 7
  ~ ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── took  3s at ⎈ minikube at  14:11:23
❯ k exec -ti nginx-7c658794b9-pxp8b -- bash -c "time curl 10.244.0.9"
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

real    0m0.005s
user    0m0.004s
sys     0m0.000s
  ~ ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── at ⎈ minikube at  14:11:25
❯ k exec -ti nginx-7c658794b9-pxp8b -- bash -c "time curl 10.244.0.10"
curl: (7) Failed to connect to 10.244.0.10 port 80: No route to host

real    0m3.047s
user    0m0.005s
sys     0m0.000s
command terminated with exit code 7

Output of chaosctl

[STRRL]

PTAL /cc @Andrewmatilde @Hexilee

[STRRL]

It would also failed by accessing with service ClusterIP:

❯ time curl 10.111.14.225:80
curl: (7) Failed to connect to 10.111.14.225 port 80 after 18437 ms: No route to host
curl 10.111.14.225:80  0.01s user 0.00s system 0% cpu 18.444 total

[STRRL]

Once I remove the HTTPChaos, everything goes well. Fortunately, it could be recovered.

  ~ ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── at ⎈ minikube at  14:14:58
❯ k delete httpchaos.chaos-mesh.org nginx-delay-2s           
httpchaos.chaos-mesh.org "nginx-delay-2s" deleted
  ~ ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── at  14:15:51
❯ time curl 10.111.14.225:80
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
curl 10.111.14.225:80  0.00s user 0.00s system 87% cpu 0.005 total

[Andrewmatilde]

@STRRL I find this may because some problem in linux bridge fdb, & trying to fix it in chaos-mesh/chaos-tproxy#51 .

[YangKeao]

Close this issue as #3412 merged.