"No route to host" after initial installation #977

kyori19 · 2019-10-28T12:51:15Z

I tried to install k3s on ubuntu 16.04 and 18.04 and same error happens.

$ curl -sfL https://get.k3s.io | sh -

I used this command to install k3s.

Version:

$ k3s -v
k3s version v0.10.0 (f9888ca3)

Describe the bug

$ kubectl get pods -A
NAMESPACE     NAME                                      READY   STATUS             RESTARTS   AGE
kube-system   coredns-57d8bbb86-bvxhz                   1/1     Running            0          46s
kube-system   helm-install-traefik-p5hvk                0/1     Error              1          46s
kube-system   local-path-provisioner-58fb86bdfd-4hqbt   0/1     CrashLoopBackOff   2          46s

These two pods crash with "No route to host" errors.

$ kubectl logs -n kube-system helm-install-traefik-p5hvk
...
+ helm repo update --strict
Hang tight while we grab the latest from your chart repositories...
...Skip local chart repository
Error: Update Failed. Check log for details
...Unable to get an update from the "stable" chart repository (https://kubernetes-charts.storage.googleapis.com):
        Get https://kubernetes-charts.storage.googleapis.com/index.yaml: dial tcp: lookup kubernetes-charts.storage.googleapis.com on 10.43.0.10:53: read udp 10.42.0.4:38460->10.43.0.10:53: read: no route to host
...

To Reproduce
Just install k3s with this command: curl -sfL https://get.k3s.io | sh -

Expected behavior
k3s works fine.

Actual behavior
Two pods don't work.

I'm a beginner about kubernetes, so there may be something overlooked...

The text was updated successfully, but these errors were encountered:

davidnuzik · 2019-11-01T17:22:17Z

There shouldn't be any issues with k3s here; or at least I'm not aware of any issues presently. Is this issue still happening for you? It looks like a possible networking issue or something else funky is going on. Can the host reach kubernetes-charts.storage.googleapis.com ?

Have you taken a look at the network and ensured everything looks good?

kyori19 · 2019-11-01T22:26:49Z

Since I did nothing special other than installing k3s, I think the network is working fine.

$ ping kubernetes-charts.storage.googleapis.com
PING kubernetes-charts.storage.googleapis.com (172.217.26.48) 56(84) bytes of data.
64 bytes from nrt12s17-in-f16.1e100.net (172.217.26.48): icmp_seq=1 ttl=58 time=2.15 ms
64 bytes from nrt12s17-in-f16.1e100.net (172.217.26.48): icmp_seq=2 ttl=58 time=2.11 ms
64 bytes from nrt12s17-in-f16.1e100.net (172.217.26.48): icmp_seq=3 ttl=58 time=2.13 ms
64 bytes from nrt12s17-in-f16.1e100.net (172.217.26.48): icmp_seq=4 ttl=58 time=2.11 ms
64 bytes from nrt12s17-in-f16.1e100.net (172.217.26.48): icmp_seq=5 ttl=58 time=2.17 ms
64 bytes from nrt12s17-in-f16.1e100.net (172.217.26.48): icmp_seq=6 ttl=58 time=2.11 ms
64 bytes from nrt12s17-in-f16.1e100.net (172.217.26.48): icmp_seq=7 ttl=58 time=2.14 ms
64 bytes from nrt12s17-in-f16.1e100.net (172.217.26.48): icmp_seq=8 ttl=58 time=2.12 ms
64 bytes from nrt12s17-in-f16.1e100.net (172.217.26.48): icmp_seq=9 ttl=58 time=2.14 ms
64 bytes from nrt12s17-in-f16.1e100.net (172.217.26.48): icmp_seq=10 ttl=58 time=2.12 ms
64 bytes from nrt12s17-in-f16.1e100.net (172.217.26.48): icmp_seq=11 ttl=58 time=2.16 ms
^C
--- kubernetes-charts.storage.googleapis.com ping statistics ---
11 packets transmitted, 11 received, 0% packet loss, time 10012ms
rtt min/avg/max/mdev = 2.114/2.137/2.178/0.065 ms

The host can reach kubernetes-charts.storage.googleapis.com.

And I realized that there's no helm-install pods now. I don't know why it was disappeared.

Now there is two pods and one of them failed to start with same error.

$ kubectl get pods -A
NAMESPACE     NAME                                      READY   STATUS             RESTARTS   AGE
kube-system   coredns-57d8bbb86-bvxhz                   1/1     Running            0          4d9h
kube-system   local-path-provisioner-58fb86bdfd-4hqbt   0/1     CrashLoopBackOff   1241       4d9h

$ kubectl logs local-path-provisioner-58fb86bdfd-4hqbt -n kube-system
time="2019-11-01T22:24:49Z" level=fatal msg="Error starting daemon: Cannot start Provisioner: failed to get Kubernetes server version: Get https://10.43.0.1:443/version?timeout=32s: dial tcp 10.43.0.1:443: connect: no route to host"

erikwilson · 2019-11-01T22:42:46Z

What is the output of ip a and ip route?

kyori19 · 2019-11-01T22:45:44Z

Here.

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 02:00:17:00:7a:99 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.2/24 brd 10.0.0.255 scope global ens3
       valid_lft forever preferred_lft forever
    inet6 fe80::17ff:fe00:7a99/64 scope link
       valid_lft forever preferred_lft forever
3: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8950 qdisc noqueue state UNKNOWN group default
    link/ether 72:c9:9b:5e:04:90 brd ff:ff:ff:ff:ff:ff
    inet 10.42.0.0/32 brd 10.42.0.0 scope global flannel.1
       valid_lft forever preferred_lft forever
    inet6 fe80::70c9:9bff:fe5e:490/64 scope link
       valid_lft forever preferred_lft forever
4: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8950 qdisc noqueue state UP group default qlen 1000
    link/ether 8a:7d:d4:85:9f:30 brd ff:ff:ff:ff:ff:ff
    inet 10.42.0.1/24 scope global cni0
       valid_lft forever preferred_lft forever
    inet6 fe80::887d:d4ff:fe85:9f30/64 scope link
       valid_lft forever preferred_lft forever
5: veth3077c663@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8950 qdisc noqueue master cni0 state UP group default
    link/ether fa:93:80:e2:ca:d6 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::f893:80ff:fee2:cad6/64 scope link
       valid_lft forever preferred_lft forever
6: vethb6909f4b@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8950 qdisc noqueue master cni0 state UP group default
    link/ether e6:14:78:1f:03:5a brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::e414:78ff:fe1f:35a/64 scope link
       valid_lft forever preferred_lft forever

$ ip route
default via 10.0.0.1 dev ens3
10.0.0.0/24 dev ens3  proto kernel  scope link  src 10.0.0.2
10.42.0.0/24 dev cni0  proto kernel  scope link  src 10.42.0.1

erikwilson · 2019-11-01T22:53:07Z

Thanks! Also, iptables --version and iptables-save?

kyori19 · 2019-11-01T22:55:48Z

OK

$ iptables --version
iptables v1.6.0

$ iptables-save
<no output>

erikwilson · 2019-11-01T23:13:21Z

Interesting that iptables-save has no output, the errors make sense if you have no iptables entries. Would you mind attaching the contents of /var/log/syslog to this issue? Are you able to give any more info about the environment you are using? (eg cloud or vm)

kyori19 · 2019-11-02T13:12:36Z

I'm using vm on Oracle Cloud.

And this is syslog.
syslog.txt

erikwilson · 2019-11-05T16:39:20Z

Thanks!
The logs currently do not have enough data as old info has been rotated out.
Would it be possible to show the syslog from a fresh system, with k3s startup?

kyori19 · 2019-11-06T10:46:27Z

I installed on new machine.
How about this?

syslog.txt

erikwilson · 2019-11-06T17:56:44Z

Thanks, nothing pops out to me from the logs, looks like it should be creating iptables entries.

It looks like the network devices and route are created, but iptables-save is showing no entries (are you running the command as root?). Is Oracle doing anything special with iptables or firewalls here?

kyori19 · 2019-11-06T21:35:35Z

Sorry... I didn't use sudo when check iptables-save...

This is the output of sudo iptables-save.

$ sudo iptables-save
# Generated by iptables-save v1.6.0 on Wed Nov  6 21:31:52 2019
*nat
:PREROUTING ACCEPT [6:270]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [17:1130]
:POSTROUTING ACCEPT [17:1130]
:KUBE-MARK-DROP - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-NODEPORTS - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-SEP-6KJXLLADRY4FRUOJ - [0:0]
:KUBE-SEP-HSRFD4S3AHFNRYNO - [0:0]
:KUBE-SEP-VOJ5KSDSMKF2NGVM - [0:0]
:KUBE-SEP-YP32F2HEJI64ZYJN - [0:0]
:KUBE-SERVICES - [0:0]
:KUBE-SVC-ERIFXISQEP7F7OF4 - [0:0]
:KUBE-SVC-JD5MR3NA4I4DYORP - [0:0]
:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
:KUBE-SVC-TCOU7JCQXEZGVUNU - [0:0]
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -s 10.42.0.0/16 -d 10.42.0.0/16 -j RETURN
-A POSTROUTING -s 10.42.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE
-A POSTROUTING ! -s 10.42.0.0/16 -d 10.42.0.0/24 -j RETURN
-A POSTROUTING ! -s 10.42.0.0/16 -d 10.42.0.0/16 -j MASQUERADE
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
-A KUBE-SEP-6KJXLLADRY4FRUOJ -s 10.42.0.2/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-6KJXLLADRY4FRUOJ -p tcp -m tcp -j DNAT --to-destination 10.42.0.2:9153
-A KUBE-SEP-HSRFD4S3AHFNRYNO -s 10.0.0.2/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-HSRFD4S3AHFNRYNO -p tcp -m tcp -j DNAT --to-destination 10.0.0.2:6443
-A KUBE-SEP-VOJ5KSDSMKF2NGVM -s 10.42.0.2/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-VOJ5KSDSMKF2NGVM -p udp -m udp -j DNAT --to-destination 10.42.0.2:53
-A KUBE-SEP-YP32F2HEJI64ZYJN -s 10.42.0.2/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-YP32F2HEJI64ZYJN -p tcp -m tcp -j DNAT --to-destination 10.42.0.2:53
-A KUBE-SERVICES ! -s 10.42.0.0/16 -d 10.43.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.43.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU
-A KUBE-SERVICES ! -s 10.42.0.0/16 -d 10.43.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.43.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-SVC-ERIFXISQEP7F7OF4
-A KUBE-SERVICES ! -s 10.42.0.0/16 -d 10.43.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:metrics cluster IP" -m tcp --dport 9153 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.43.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:metrics cluster IP" -m tcp --dport 9153 -j KUBE-SVC-JD5MR3NA4I4DYORP
-A KUBE-SERVICES ! -s 10.42.0.0/16 -d 10.43.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.43.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
-A KUBE-SVC-ERIFXISQEP7F7OF4 -j KUBE-SEP-YP32F2HEJI64ZYJN
-A KUBE-SVC-JD5MR3NA4I4DYORP -j KUBE-SEP-6KJXLLADRY4FRUOJ
-A KUBE-SVC-NPX46M4PTMTKRN6Y -j KUBE-SEP-HSRFD4S3AHFNRYNO
-A KUBE-SVC-TCOU7JCQXEZGVUNU -j KUBE-SEP-VOJ5KSDSMKF2NGVM
COMMIT
# Completed on Wed Nov  6 21:31:52 2019
# Generated by iptables-save v1.6.0 on Wed Nov  6 21:31:52 2019
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [152:35760]
:InstanceServices - [0:0]
:KUBE-EXTERNAL-SERVICES - [0:0]
:KUBE-FIREWALL - [0:0]
:KUBE-FORWARD - [0:0]
:KUBE-SERVICES - [0:0]
-A INPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A INPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes externally-visible service portals" -j KUBE-EXTERNAL-SERVICES
-A INPUT -j KUBE-FIREWALL
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p udp -m udp --sport 123 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -m comment --comment "kubernetes forwarding rules" -j KUBE-FORWARD
-A FORWARD -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -s 10.42.0.0/16 -j ACCEPT
-A FORWARD -d 10.42.0.0/16 -j ACCEPT
-A OUTPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -j KUBE-FIREWALL
-A OUTPUT -d 169.254.0.0/16 -j InstanceServices
-A InstanceServices -d 169.254.0.2/32 -p tcp -m owner --uid-owner 0 -m tcp --dport 3260 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.2.0/24 -p tcp -m owner --uid-owner 0 -m tcp --dport 3260 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.0.2/32 -p tcp -m tcp --dport 80 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.169.254/32 -p udp -m udp --dport 53 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.169.254/32 -p tcp -m tcp --dport 53 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.0.3/32 -p tcp -m owner --uid-owner 0 -m tcp --dport 80 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.0.4/32 -p tcp -m tcp --dport 80 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.169.254/32 -p udp -m udp --dport 67 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.169.254/32 -p udp -m udp --dport 69 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.169.254/32 -p udp -m udp --dport 123 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.0.0/16 -p tcp -m tcp -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j REJECT --reject-with tcp-reset
-A InstanceServices -d 169.254.0.0/16 -p udp -m udp -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j REJECT --reject-with icmp-port-unreachable
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
-A KUBE-FORWARD -m conntrack --ctstate INVALID -j DROP
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x4000/0x4000 -j ACCEPT
-A KUBE-FORWARD -s 10.42.0.0/16 -m comment --comment "kubernetes forwarding conntrack pod source rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-FORWARD -d 10.42.0.0/16 -m comment --comment "kubernetes forwarding conntrack pod destination rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
COMMIT
# Completed on Wed Nov  6 21:31:52 2019

erikwilson · 2019-11-06T21:41:58Z

Looks like Oracle's firewall is breaking things.

kyori19 · 2019-11-06T21:49:53Z

What should I do?

vlastoun · 2019-11-10T20:08:03Z

same problem, using oracle cloud ubuntu

 kubectl logs local-path-provisioner-58fb86bdfd-hfxtz --namespace kube-system

time="2019-11-10T20:05:51Z" level=fatal msg="Error starting daemon: Cannot start Provisioner: failed to get Kubernetes server version: Get https://10.43.0.1:443/version?timeout=32s: dial tcp 10.43.0.1:443: connect: no route to host"

apollo13 · 2019-11-11T15:36:05Z

@erikwilson After also running into this issue I dug deeper and found the culprit. I am on CentOS7 with the following iptables default configuration:

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT

By default this setup only allows SSH and rejects all other input and forward requests. Now when k3s starts itself the rules look like this (forward only):

-A FORWARD -m comment --comment "kubernetes forwarding rules" -j KUBE-FORWARD
-A FORWARD -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -s 10.42.0.0/16 -j ACCEPT
-A FORWARD -d 10.42.0.0/16 -j ACCEPT

As you can see while the kubernetes forwarding rules and service portal are created properly, the flannel rules are just appened last and as such will just fail. You can find the relevant code in:
https://github.com/rancher/k3s/blob/405f85aa025fe4ce2f74f19916eeadb5dcac8e08/pkg/agent/flannel/flannel.go#L70-L71

Interestingly enough there is a similar rule in the KUBE-FORWARD chain:

-A KUBE-FORWARD -s 10.42.0.0/16 -m comment --comment "kubernetes forwarding conntrack pod source rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-FORWARD -d 10.42.0.0/16 -m comment --comment "kubernetes forwarding conntrack pod destination rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

But this one only accepts already established connections, so it won't help here. All in all I do not think that the existing firewalls are to blame here, but k3s creating it's rules in the wrong place.

apollo13 · 2019-11-11T15:52:12Z

Note: Just moving the rules before the REJECT doesn't seem to be enough, I have to check where else it triggers failures :( [Removing the default REJECT does fix it]

apollo13 · 2019-11-11T16:04:12Z

Okay, I have got it. The default INPUT rules also drop everything, so one needs to (for instance) add:

iptables -I INPUT 3 -s 10.42.0.0/16 -j ACCEPT
iptables -I INPUT 3 -d 10.42.0.0/16 -j ACCEPT

The better option is probably to add rules to allow traffic through the cni0 interface. Either way, to summarize I can say:
If there are rules in the foward or input chains that reject any traffic, k3s will not work

EDIT: Note that when I refer to 10.43 I mean flannel on my box and 10.42 is cni.

apollo13 · 2019-11-11T16:16:35Z

A working iptables configuration with comments where I added rules and what they look like:

# Generated by iptables-save v1.4.21 on Mon Nov 11 16:51:13 2019
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [51:25393]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
# Added my rules
-A INPUT -s 10.42.0.0/16 -j ACCEPT
-A INPUT -d 10.42.0.0/16 -j ACCEPT
-A INPUT -j LOG --log-prefix "IPTables-Reject-Input: " --log-level 4
# End added my rules
-A INPUT -j REJECT --reject-with icmp-host-prohibited
# Added my rules
-A FORWARD -s 10.42.0.0/16 -j ACCEPT
-A FORWARD -d 10.42.0.0/16 -j ACCEPT
-A FORWARD -j LOG --log-prefix "IPTables-Reject-Forward: " --log-level 4
# End added my rules
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT
# Completed on Mon Nov 11 16:51:13 2019
# Generated by iptables-save v1.4.21 on Mon Nov 11 16:51:13 2019
*nat
:PREROUTING ACCEPT [17:964]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
COMMIT
# Completed on Mon Nov 11 16:51:13 2019

In general I think it would be good to allow INPUTs on cni0 instead of the source addr, but it illustrates the issue nicely. If you drop the added accepts the kernel will show you the rejections:

Nov 11 17:10:49 infra01 kernel: IPTables-Reject-Input: IN=cni0 OUT= PHYSIN=veth3a22aa74 MAC=3e:3a:0a:8c:95:78:72:c4:80:d8:db:e9:08:00 SRC=10.42.0.2 DST=172.22.3.180 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=20683 DF PROTO=TCP SPT=36444 DPT=6443 WINDOW=28200 RES=0x00 SYN URGP=0

Here the pod tries to talk to the kube master on the same host but the INPUT rule forbids it.

BlackTurtle123 · 2019-12-25T16:17:06Z

I guess my issue is also connected with this: #1247

maci0 · 2020-05-24T12:14:27Z

A working iptables configuration with comments where I added rules and what they look like:

# Generated by iptables-save v1.4.21 on Mon Nov 11 16:51:13 2019
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [51:25393]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
# Added my rules
-A INPUT -s 10.42.0.0/16 -j ACCEPT
-A INPUT -d 10.42.0.0/16 -j ACCEPT
-A INPUT -j LOG --log-prefix "IPTables-Reject-Input: " --log-level 4
# End added my rules
-A INPUT -j REJECT --reject-with icmp-host-prohibited
# Added my rules
-A FORWARD -s 10.42.0.0/16 -j ACCEPT
-A FORWARD -d 10.42.0.0/16 -j ACCEPT
-A FORWARD -j LOG --log-prefix "IPTables-Reject-Forward: " --log-level 4
# End added my rules
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT
# Completed on Mon Nov 11 16:51:13 2019
# Generated by iptables-save v1.4.21 on Mon Nov 11 16:51:13 2019
*nat
:PREROUTING ACCEPT [17:964]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
COMMIT
# Completed on Mon Nov 11 16:51:13 2019

In general I think it would be good to allow INPUTs on cni0 instead of the source addr, but it illustrates the issue nicely. If you drop the added accepts the kernel will show you the rejections:

Nov 11 17:10:49 infra01 kernel: IPTables-Reject-Input: IN=cni0 OUT= PHYSIN=veth3a22aa74 MAC=3e:3a:0a:8c:95:78:72:c4:80:d8:db:e9:08:00 SRC=10.42.0.2 DST=172.22.3.180 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=20683 DF PROTO=TCP SPT=36444 DPT=6443 WINDOW=28200 RES=0x00 SYN URGP=0

Here the pod tries to talk to the kube master on the same host but the INPUT rule forbids it.

you're the man! this fixes it for me on centos8.
now we just need to get it working with firewalld.
for plain podman i had to add the firewalld plugin to the cni config, maybe this needs to be done here as well ?

tuananh · 2020-12-21T16:23:21Z

what's the proper fix for this?

Edit: since i was using microk8s before, i did a iptable flush before installiing k3s and it works now

stale · 2021-07-30T22:45:39Z

This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 180 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.

ya-mouse · 2022-11-13T21:10:33Z

Just in case if somebody faced the same on Oracle Cloud 9:

sudo firewall-cmd --add-port=6443/tcp

No other changes to iptables needed.

Also add Ingress Rule limiting to the internal network:

To join a master node use internal node's hostname or IP (i.e. 10.0.0.X):

sudo k3s agent --server https://k3s-master.subnetXXX.vcnYYYY.oraclevcn.com:6443 --token "${TOKEN}"

davidnuzik added [zube]: To Triage kind/question No code change, just asking/answering a question labels Nov 1, 2019

davidnuzik added this to the Backlog milestone Nov 1, 2019

erikwilson removed the kind/question No code change, just asking/answering a question label Nov 5, 2019

lerit mentioned this issue Feb 4, 2020

Exec shell and view logs do not work when import k3s cluster rancher/rancher#25236

Closed

davidnuzik added the [zube]: Unscheduled label May 1, 2020

zube bot added [zube]: To Triage and removed [zube]: To Triage labels May 1, 2020

davidnuzik added the [zube]: Unscheduled label May 1, 2020

zube bot removed the [zube]: To Triage label May 1, 2020

ttpcodes mentioned this issue Jun 3, 2020

[BUG]Longhorn doesn't work on K3S + CentOS + SELinux longhorn/longhorn#1273

Closed

jrcichra mentioned this issue Jul 21, 2020

(Oracle Cloud) - k3s installing master - dial tcp 10.43.0.1:443: connect: no route to host #2046

Closed

martijnjanssen mentioned this issue Oct 26, 2020

nixos/k3s: dns not working (reply from unexpected source) NixOS/nixpkgs#98766

Open

stale bot added the status/stale label Jul 30, 2021

stale bot closed this as completed Aug 13, 2021

davesargrad mentioned this issue May 27, 2022

Though Ceph and CephFileSystem seem perfectly healthy PVC stays pending rook/rook#10294

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

"No route to host" after initial installation #977

"No route to host" after initial installation #977

kyori19 commented Oct 28, 2019

davidnuzik commented Nov 1, 2019

kyori19 commented Nov 1, 2019

erikwilson commented Nov 1, 2019

kyori19 commented Nov 1, 2019

erikwilson commented Nov 1, 2019

kyori19 commented Nov 1, 2019

erikwilson commented Nov 1, 2019

kyori19 commented Nov 2, 2019

erikwilson commented Nov 5, 2019

kyori19 commented Nov 6, 2019

erikwilson commented Nov 6, 2019

kyori19 commented Nov 6, 2019

erikwilson commented Nov 6, 2019

kyori19 commented Nov 6, 2019

vlastoun commented Nov 10, 2019 •

edited

Loading

apollo13 commented Nov 11, 2019

apollo13 commented Nov 11, 2019

apollo13 commented Nov 11, 2019 •

edited

Loading

apollo13 commented Nov 11, 2019

BlackTurtle123 commented Dec 25, 2019

maci0 commented May 24, 2020

tuananh commented Dec 21, 2020 •

edited

Loading

stale bot commented Jul 30, 2021

ya-mouse commented Nov 13, 2022

"No route to host" after initial installation #977

"No route to host" after initial installation #977

Comments

kyori19 commented Oct 28, 2019

davidnuzik commented Nov 1, 2019

kyori19 commented Nov 1, 2019

erikwilson commented Nov 1, 2019

kyori19 commented Nov 1, 2019

erikwilson commented Nov 1, 2019

kyori19 commented Nov 1, 2019

erikwilson commented Nov 1, 2019

kyori19 commented Nov 2, 2019

erikwilson commented Nov 5, 2019

kyori19 commented Nov 6, 2019

erikwilson commented Nov 6, 2019

kyori19 commented Nov 6, 2019

erikwilson commented Nov 6, 2019

kyori19 commented Nov 6, 2019

vlastoun commented Nov 10, 2019 • edited Loading

apollo13 commented Nov 11, 2019

apollo13 commented Nov 11, 2019

apollo13 commented Nov 11, 2019 • edited Loading

apollo13 commented Nov 11, 2019

BlackTurtle123 commented Dec 25, 2019

maci0 commented May 24, 2020

tuananh commented Dec 21, 2020 • edited Loading

stale bot commented Jul 30, 2021

ya-mouse commented Nov 13, 2022

vlastoun commented Nov 10, 2019 •

edited

Loading

apollo13 commented Nov 11, 2019 •

edited

Loading

tuananh commented Dec 21, 2020 •

edited

Loading