Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

k3s service cannot start in RHEL with firewall enabled #401

Open
abdennour opened this issue Apr 26, 2019 · 4 comments
Open

k3s service cannot start in RHEL with firewall enabled #401

abdennour opened this issue Apr 26, 2019 · 4 comments

Comments

@abdennour
Copy link

@abdennour abdennour commented Apr 26, 2019

Describe the bug
I downloaded k3s on Redhat 7 and I wait for the cluster to be up, but it never be up.
I adjust firewalld to accept 6443/tcp, but no benefit.

To Reproduce
Steps to reproduce the behavior:

  1. Install curl -sfL https://get.k3s.io | sh -
  2. Check if the service k3s is up : watch -n 3 k3s kubectl get node.
  3. Adjust firewalld to allow 6443/tcp.
  4. Run again (3) then systemctl status k3s but we got:
systemctl status k3s -l
● k3s.service - Lightweight Kubernetes
   Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2019-04-26 20:45:29 UTC; 10min ago
     Docs: https://k3s.io
  Process: 4157 ExecStart=/usr/local/bin/k3s server (code=exited, status=1/FAILURE)
  Process: 4155 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
  Process: 4154 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
 Main PID: 4157 (code=exited, status=1/FAILURE)

Apr 26 20:45:28 control1 k3s[4157]: time="2019-04-26T20:45:28.832482434Z" level=info msg="Run: k3s kubectl"
Apr 26 20:45:28 control1 k3s[4157]: time="2019-04-26T20:45:28.832497807Z" level=info msg="k3s is up and running"
Apr 26 20:45:28 control1 systemd[1]: Started Lightweight Kubernetes.
Apr 26 20:45:28 control1 k3s[4157]: time="2019-04-26T20:45:28.936343517Z" level=info msg="Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log"
Apr 26 20:45:28 control1 k3s[4157]: time="2019-04-26T20:45:28.937828597Z" level=info msg="Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd"
Apr 26 20:45:28 control1 k3s[4157]: time="2019-04-26T20:45:28.942713193Z" level=info msg="Waiting for containerd startup: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial unix /run/k3s/containerd/containerd.sock: connect: connection refused\""
Apr 26 20:45:29 control1 k3s[4157]: containerd: exit status 1
Apr 26 20:45:29 control1 systemd[1]: k3s.service: main process exited, code=exited, status=1/FAILURE
Apr 26 20:45:29 control1 systemd[1]: Unit k3s.service entered failed state.
Apr 26 20:45:29 control1 systemd[1]: k3s.service failed.

Expected behavior
I expect to

Screenshots
If applicable, add screenshots to help explain your problem.
image

Additional context

  • uname -a : Linux control1 3.10.0-862.3.2.el7.x86_64 #1 SMP Tue May 15 18:22:15 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
@erikwilson

This comment has been minimized.

Copy link
Member

@erikwilson erikwilson commented Apr 27, 2019

There is more that needs to be done with the firewall setup, here are some other potential ports:
Kubernetes needs:

Master node(s):
TCP     6443*       Kubernetes API Server
TCP     10250       Kubelet API
TCP     10251       kube-scheduler
TCP     10252       kube-controller-manager
UDP     8285        flannel overlay network - udp backend

Worker nodes (minions):
TCP     10250       Kubelet API
TCP     30000-32767 NodePort Services
UDP     8285        flannel overlay network - udp backend

Also see https://github.com/coreos/coreos-kubernetes/blob/master/Documentation/kubernetes-networking.md

The firewall will also probably need to be setup to allow traffic between various interfaces.

I am marking this issue as kind/documentation because we should provide better requirements in docs in https://github.com/rancher/k3s#open-ports--network-security on what is needed for a firewall setup.

@erikwilson erikwilson changed the title k3s service cannot start in RHEL k3s service cannot start in RHEL with firewall enabled Apr 27, 2019
@thatarchguy

This comment has been minimized.

Copy link

@thatarchguy thatarchguy commented Apr 29, 2019

cat /var/lib/rancher/k3s/agent/containerd/containerd.log

I bet you'll see something like I was:

time="2019-04-29T16:15:41.094592934-04:00" level=info msg="containerd successfully booted in 0.002523s"
time="2019-04-29T16:15:41.097621565-04:00" level=info msg="Start subscribing containerd event"
time="2019-04-29T16:15:41.097658651-04:00" level=info msg="Start recovering state"
time="2019-04-29T16:15:41.097764992-04:00" level=info msg="Start event monitor"
time="2019-04-29T16:15:41.097784041-04:00" level=info msg="Start snapshots syncer"
time="2019-04-29T16:15:41.097793303-04:00" level=info msg="Start streaming server"
time="2019-04-29T16:15:41.098765431-04:00" level=error msg="Failed to start streaming server" error="listen tcp: lookup myhostname on 192.168.1.2:53: no such host"

I was seeing the same exact error as you on Arch linux. My hostname was not resolvable, so containerd was not starting.

@ifurther54

This comment has been minimized.

Copy link

@ifurther54 ifurther54 commented Jun 13, 2019

I installed this in Centos 7.6.
And I added the cni0 to firewall internal.

@xiaods

This comment has been minimized.

Copy link

@xiaods xiaods commented Nov 11, 2019

@thatarchguy check the hostname have a record the host

127.0.0.1 xxx.node.local

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.