Calico can't live without a default route #8481
Comments
|
As far as I know, Calico doesn't depend on the NO default route, Can you show the manifests of the calico-node and logs? |
|
Manifest calico.yaml.txt |
|
Some logs with proxy-mode=iptables: |
|
After I added the default route, the calico-node is running. I think it has something to do with iptables MASQUERADE? |
|
Without a default route, I can't access the apiserver at the node through the address of the kubernetes service, but access to the endpoint is fine. After adding the default route, everything works fine. ➜ test git:(e687fb89) ✗ nsenter -t 108333 -n
cyclinder3# curl -k https://10.233.0.1:443
curl: (7) Couldn't connect to server
cyclinder3# curl -k https://10.233.0.1:443
curl: (7) Couldn't connect to server
cyclinder3# curl -k https://172.25.0.3:6443
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
"reason": "Forbidden",
"details": {},
"code": 403
}
cyclinder3#
cyclinder3# ip r a default via 172.25.0.1 dev eth0
cyclinder3# curl -k https://10.233.0.1:6443
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
"reason": "Forbidden",
"details": {},
"code": 403
}This doesn't seem to have anything to do with calico, it's more like iptables behavior. |
|
No, it works with just about every other cni-plugin:
so this is a Calico-only issue. I may be that Calico can live without a default route, but can't be installed without it? I start with an empty cluster |
|
This seems very strange, there is no default route and DNAT doesn't seem to work. I'm not sure how other CNIs work, these iptables rules were created by kube-proxy |
|
You are right. With proxy-mode=iptables flannel doesn't work without a default route either. My default setup is proxy-mode=ipvs so I didn't notice, sorry. But with proxy-mode=ipvs the kubernetes service address works from a main netns on a node, even with Calico: |
|
I guess this is a K8s "peculiarity" after all, not a Calico issue. I don't think it's a big deal, everybody has a default route, right... Some security buff many refuse to set it though, so it should probably be documented. I'll write an issue on K8s... Should I close this one? Or do you want to track it? It took some time to troubleshoot 😄 |
|
Moved to kubernetes/kubernetes#123120 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Expected Behavior
Calico should work even if the K8s nodes has no default route.
Other CNI-plugins can handle this case. I have tested: Cilium, Flannel, Kindnet and Antrea.
Current Behavior
If there is no default route the K8s
kubernetesservice becomes unavailable (and likely all services but I don't get that far).With proxy-mode=ipvs the
calico-kube-controllersnever becomes "ready" and restarts:12.0.0.1 is the ip of the "kubernetes" service.
With proxy-mode=iptables it's even worse:
Possible Solution
No idea
Steps to Reproduce (for bugs)
Start a K8s cluster with Calico and no default route on the nodes.
Context
Experiment in a virtual test-cluster. When no router VMs are started, the K8s nodes don't get a default route.
No impact on any production, and no hurry to fix this.
Your Environment
The text was updated successfully, but these errors were encountered: