k3s will listen on random ports #9724

bingfengfeifei · 2024-03-12T01:32:27Z

bingfengfeifei
Mar 12, 2024

Environmental Info:
K3s Version: k3s version v1.27.10+k3s2 (5b2ac88)
go version go1.20.13

Node(s) CPU architecture, OS, and Version:

Linux dev 4.19.90-2211.5.0.0178.22.uel20.aarch64 #1 SMP Thu Nov 24 10:33:07 CST 2022 aarch64 aarch64 aarch64 GNU/Linux
Cluster Configuration:

1 server 0 agent

Describe the bug:

# netstat -nlp | grep k3s
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      338698/k3s server   
tcp        0      0 127.0.0.1:10249         0.0.0.0:*               LISTEN      338698/k3s server   
tcp        0      0 127.0.0.1:10250         0.0.0.0:*               LISTEN      338698/k3s server   
tcp        0      0 198.19.11.11:2379       0.0.0.0:*               LISTEN      338698/k3s server   
tcp        0      0 127.0.0.1:2379          0.0.0.0:*               LISTEN      338698/k3s server   
tcp        0      0 198.19.11.11:6443       0.0.0.0:*               LISTEN      338698/k3s server   
tcp        0      0 127.0.0.1:6444          0.0.0.0:*               LISTEN      338698/k3s server   
tcp        0      0 198.19.11.11:2380       0.0.0.0:*               LISTEN      338698/k3s server   
tcp        0      0 127.0.0.1:2380          0.0.0.0:*               LISTEN      338698/k3s server   
tcp        0      0 127.0.0.1:2381          0.0.0.0:*               LISTEN      338698/k3s server   
tcp        0      0 127.0.0.1:2382          0.0.0.0:*               LISTEN      338698/k3s server   
tcp        0      0 127.0.0.1:10256         0.0.0.0:*               LISTEN      338698/k3s server   
tcp        0      0 127.0.0.1:10257         0.0.0.0:*               LISTEN      338698/k3s server   
tcp        0      0 127.0.0.1:10258         0.0.0.0:*               LISTEN      338698/k3s server   
tcp        0      0 127.0.0.1:10259         0.0.0.0:*               LISTEN      338698/k3s server   
tcp        0      0 127.0.0.1:10260         0.0.0.0:*               LISTEN      338698/k3s server   
tcp6       0      0 :::38497                :::*                    LISTEN      338698/k3s server

When using netstat to check the listening port, a random port 38497 will be exposed. I want to use k3s on a single machine and not expose any ports at 0.0.0.0.
Steps To Reproduce:

Installed K3s:
Use the following configure and install commands
config.yaml

node-name: "gaea"
bind-address: "198.19.11.11"
cluster-cidr: "198.19.123.0/24,198:19:123::/56"
service-cidr: "198.19.161.0/24,198:19:161::/112"
cluster-dns: "198.19.161.10"
service-node-port-range: "0-65535"
disable-network-policy: true
disable:
  - traefik
  - local-storage
  - metrics-server
data-dir: "/data/rancher/k3s"
default-local-storage-path: "/data/local-storage"
kubelet-arg:
  - "address=127.0.0.1"
kube-cloud-controller-manager-arg:
  - "webhook-bind-address=127.0.0.1"
flannel-iface: vxlan

install command:

INSTALL_K3S_SKIP_DOWNLOAD=true INSTALL_K3S_EXEC='--docker --node-ip=198.19.11.11,198:19:11:11:: node-external-ip=198.19.11.11,198:19:11:11:: --cluster-init' ./k3s-install.sh

Expected behavior:

All ports are listening on node-ip 198.19.11.11 or 127.0.0.1, not on 0.0.0.0
Or you can modify the listening IP of the port through configuration, but I did not find the relevant parameters.
Actual behavior:

tcp6       0      0 :::38497                :::*                    LISTEN      338698/k3s server

Additional context / logs:

Answered by brandond

Apr 17, 2024

Looks like it's the --streaming-bind-addr flag:
https://github.com/Mirantis/cri-dockerd/blob/b138f5226ae901b99ea34d40ab1eaed1c26445a4/cmd/server.go#L171-L183

We could probably take a look at setting this to a fixed value in future releases.

Tracking this in #9965

View full answer

bingfengfeifei · 2024-03-22T02:14:01Z

bingfengfeifei
Mar 22, 2024
Author

After my observation, this port is the spdy port of kubelet, and the kubectl exec command will communicate with this port.
When I use kubeadm to create a k8s cluster, the port will listen on 127.0.0.1 by default.

tcp   LISTEN 0      4096         127.0.0.1:46825             0.0.0.0:*     users:(("kubelet",pid=2219,fd=12))

However, when created using k3s, the port will be monitored at all addresses, which exposes the port to the outside by default, which may cause security risks. How can I set the listening port to listen on localhost?

0 replies

mcarbonneaux · 2024-04-06T22:46:09Z

mcarbonneaux
Apr 6, 2024

i've configured my k3s with --bind-address and K3s listen on the configured ip, but not for Kubelet metrics port (10250) and spdy port of kubelet:

# netstat -lnput | grep k3s
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      1033195/k3s server
tcp        0      0 127.0.0.1:10249         0.0.0.0:*               LISTEN      1033195/k3s server
tcp        0      0 127.0.0.1:10258         0.0.0.0:*               LISTEN      1033195/k3s server
tcp        0      0 127.0.0.1:10259         0.0.0.0:*               LISTEN      1033195/k3s server
tcp        0      0 127.0.0.1:10256         0.0.0.0:*               LISTEN      1033195/k3s server
tcp        0      0 127.0.0.1:10257         0.0.0.0:*               LISTEN      1033195/k3s server
tcp        0      0 192.168.0.1:6443        0.0.0.0:*               LISTEN      1033195/k3s server
tcp        0      0 127.0.0.1:6444          0.0.0.0:*               LISTEN      1033195/k3s server
tcp6       0      0 :::10250                :::*                    LISTEN      1033195/k3s server
tcp6       0      0 :::44233                :::*                    LISTEN      1033195/k3s server

how to force k3s to listen on 127.0.0.1 for spdy port, and set address to bind-address for Kubelet metrics port ?

0 replies

mcarbonneaux · 2024-04-06T22:50:55Z

mcarbonneaux
Apr 6, 2024

ok i found how to force kublet metrics port listen:
#4135 (comment)

# netstat -lnput | grep k3s
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      1118252/k3s server
tcp        0      0 127.0.0.1:10249         0.0.0.0:*               LISTEN      1118252/k3s server
tcp        0      0 127.0.0.1:10258         0.0.0.0:*               LISTEN      1118252/k3s server
tcp        0      0 127.0.0.1:10259         0.0.0.0:*               LISTEN      1118252/k3s server
tcp        0      0 127.0.0.1:10256         0.0.0.0:*               LISTEN      1118252/k3s server
tcp        0      0 127.0.0.1:10257         0.0.0.0:*               LISTEN      1118252/k3s server
tcp        0      0 192.168.0.1:6443        0.0.0.0:*               LISTEN      1118252/k3s server
tcp        0      0 192.168.0.1:10250       0.0.0.0:*               LISTEN      1118252/k3s server
tcp        0      0 127.0.0.1:6444          0.0.0.0:*               LISTEN      1118252/k3s server
tcp6       0      0 :::38803                :::*                    LISTEN      1118252/k3s server

1 reply

bingfengfeifei Apr 17, 2024
Author

However, the kubelet spdy port is still listening on 0.0.0.0. If it is created through kubeadm, it will listen on localhost by default.

manuelbuil · 2024-04-16T14:00:31Z

manuelbuil
Apr 16, 2024
Collaborator

@bingfengfeifei what config are you using? I did not manage to reproduce that env. I never see a random port listening:

$> sudo ss -lntup | grep k3s
tcp   LISTEN 0      4096                         127.0.0.1:10256      0.0.0.0:*    users:(("k3s-agent",pid=2427,fd=44))     
tcp   LISTEN 0      4096                         127.0.0.1:10249      0.0.0.0:*    users:(("k3s-agent",pid=2427,fd=41))     
tcp   LISTEN 0      4096                         127.0.0.1:10248      0.0.0.0:*    users:(("k3s-agent",pid=2427,fd=31))     
tcp   LISTEN 0      4096                         127.0.0.1:6444       0.0.0.0:*    users:(("k3s-agent",pid=2427,fd=7))      
tcp   LISTEN 0      4096                                 *:10250            *:*    users:(("k3s-agent",pid=2427,fd=32))

4 replies

bingfengfeifei Apr 17, 2024
Author

config.yaml

node-name: "gaea"
bind-address: "198.19.11.11"
cluster-cidr: "198.19.123.0/24,198:19:123::/56"
service-cidr: "198.19.161.0/24,198:19:161::/112"
cluster-dns: "198.19.161.10"
service-node-port-range: "0-65535"
disable-network-policy: true
disable:
  - traefik
  - local-storage
  - metrics-server
data-dir: "/data/rancher/k3s"
default-local-storage-path: "/data/local-storage"
kubelet-arg:
  - "address=127.0.0.1"
kube-cloud-controller-manager-arg:
  - "webhook-bind-address=127.0.0.1"
flannel-iface: vxlan

install command

INSTALL_K3S_SKIP_DOWNLOAD=true INSTALL_K3S_EXEC='--docker --node-ip=198.19.11.11,198:19:11:11:: node-external-ip=198.19.11.11,198:19:11:11:: --cluster-init' ./k3s-install.sh

I used Docker as the runtime of k3s. It may be due to this reason.

bingfengfeifei Apr 17, 2024
Author

When I remove the --docker startup parameter, the random port listening on 0.0.0.0 disappears

u_str LISTEN 0      4096                                   /data/rancher/k3s/kubelet/pod-resources/2202050811 61130661               * 0        users:(("k3s-server",pid=2648662,fd=172))                        
u_str LISTEN 0      4096                                         /var/lib/kubelet/device-plugins/kubelet.sock 61133096               * 0        users:(("k3s-server",pid=2648662,fd=179))                        
u_str LISTEN 0      4096                                                                            kine.sock 61129704               * 0        users:(("k3s-server",pid=2648662,fd=12))                         
u_str LISTEN 0      4096                                            /run/k3s/containerd/containerd.sock.ttrpc 61130932               * 0        users:(("containerd",pid=2648681,fd=11))                         
u_str LISTEN 0      4096                                                  /run/k3s/containerd/containerd.sock 61130933               * 0        users:(("containerd",pid=2648681,fd=12))                         
tcp   LISTEN 0      4096                                                                            127.0.0.1:10248            0.0.0.0:*        users:(("k3s-server",pid=2648662,fd=173))                        
tcp   LISTEN 0      4096                                                                            127.0.0.1:10249            0.0.0.0:*        users:(("k3s-server",pid=2648662,fd=207))                        
tcp   LISTEN 0      4096                                                                            127.0.0.1:10250            0.0.0.0:*        users:(("k3s-server",pid=2648662,fd=169))                        
tcp   LISTEN 0      4096                                                                         198.19.11.11:6443             0.0.0.0:*        users:(("k3s-server",pid=2648662,fd=13))                         
tcp   LISTEN 0      4096                                                                            127.0.0.1:6444             0.0.0.0:*        users:(("k3s-server",pid=2648662,fd=18))                         
tcp   LISTEN 0      4096                                                                            127.0.0.1:10256            0.0.0.0:*        users:(("k3s-server",pid=2648662,fd=208))                        
tcp   LISTEN 0      4096                                                                            127.0.0.1:10257            0.0.0.0:*        users:(("k3s-server",pid=2648662,fd=156))                        
tcp   LISTEN 0      4096                                                                            127.0.0.1:10258            0.0.0.0:*        users:(("k3s-server",pid=2648662,fd=189))                        
tcp   LISTEN 0      4096                                                                            127.0.0.1:10259            0.0.0.0:*        users:(("k3s-server",pid=2648662,fd=147))

When I added the --docker startup parameter, the listener appeared on a random port at 0.0.0.0

u_str LISTEN 0      4096                                     /data/rancher/k3s/kubelet/pod-resources/91477519 61139941               * 0        users:(("k3s-server",pid=2650910,fd=173))                        
u_str LISTEN 0      4096                                         /var/lib/kubelet/device-plugins/kubelet.sock 61142157               * 0        users:(("k3s-server",pid=2650910,fd=154))                        
u_str LISTEN 0      4096                                                      /run/k3s/cri-dockerd/3569649467 61140314               * 0        users:(("k3s-server",pid=2650910,fd=149))                        
u_str LISTEN 0      4096                                                                            kine.sock 61139788               * 0        users:(("k3s-server",pid=2650910,fd=12))                         
tcp   LISTEN 0      4096                                                                            127.0.0.1:10248            0.0.0.0:*        users:(("k3s-server",pid=2650910,fd=174))                        
tcp   LISTEN 0      4096                                                                            127.0.0.1:10249            0.0.0.0:*        users:(("k3s-server",pid=2650910,fd=211))                        
tcp   LISTEN 0      4096                                                                            127.0.0.1:10250            0.0.0.0:*        users:(("k3s-server",pid=2650910,fd=170))                        
tcp   LISTEN 0      4096                                                                         198.19.11.11:6443             0.0.0.0:*        users:(("k3s-server",pid=2650910,fd=15))                         
tcp   LISTEN 0      4096                                                                            127.0.0.1:6444             0.0.0.0:*        users:(("k3s-server",pid=2650910,fd=18))                         
tcp   LISTEN 0      4096                                                                            127.0.0.1:10256            0.0.0.0:*        users:(("k3s-server",pid=2650910,fd=209))                        
tcp   LISTEN 0      4096                                                                            127.0.0.1:10257            0.0.0.0:*        users:(("k3s-server",pid=2650910,fd=178))                        
tcp   LISTEN 0      4096                                                                            127.0.0.1:10258            0.0.0.0:*        users:(("k3s-server",pid=2650910,fd=172))                        
tcp   LISTEN 0      4096                                                                            127.0.0.1:10259            0.0.0.0:*        users:(("k3s-server",pid=2650910,fd=203))                        
tcp   LISTEN 0      4096                                                                                    *:42005                  *:*        users:(("k3s-server",pid=2650910,fd=146))

brandond Apr 17, 2024
Collaborator

If it is only present when using --docker, that would be coming from cri-dockerd. I don't know if it's configurable. Have you opened an issue at https://github.com/Mirantis/cri-dockerd ?

brandond Apr 17, 2024
Collaborator

Looks like it's the --streaming-bind-addr flag:
https://github.com/Mirantis/cri-dockerd/blob/b138f5226ae901b99ea34d40ab1eaed1c26445a4/cmd/server.go#L171-L183

We could probably take a look at setting this to a fixed value in future releases.

Tracking this in Set cri-dockerd streaming-bind-addr to fixed address #9965

Answer selected by brandond

mcarbonneaux · 2024-04-21T11:40:08Z

mcarbonneaux
Apr 21, 2024

If it is only present when using --docker, that would be coming from cri-dockerd. I don't know if it's configurable. Have you opened an issue at https://github.com/Mirantis/cri-dockerd ?

i confirm is linked to docker option

with docker option:

# ss -lnput | grep k3s
tcp   LISTEN 0      4096     192.168.0.1:6443       0.0.0.0:*    users:(("k3s-server",pid=42945,fd=13))
tcp   LISTEN 0      4096       127.0.0.1:6444       0.0.0.0:*    users:(("k3s-server",pid=42945,fd=14))
tcp   LISTEN 0      4096       127.0.0.1:10258      0.0.0.0:*    users:(("k3s-server",pid=42945,fd=196))
tcp   LISTEN 0      4096       127.0.0.1:10259      0.0.0.0:*    users:(("k3s-server",pid=42945,fd=157))
tcp   LISTEN 0      4096       127.0.0.1:10256      0.0.0.0:*    users:(("k3s-server",pid=42945,fd=201))
tcp   LISTEN 0      4096       127.0.0.1:10257      0.0.0.0:*    users:(("k3s-server",pid=42945,fd=173))
tcp   LISTEN 0      4096       127.0.0.1:10248      0.0.0.0:*    users:(("k3s-server",pid=42945,fd=169))
tcp   LISTEN 0      4096       127.0.0.1:10249      0.0.0.0:*    users:(("k3s-server",pid=42945,fd=203))
tcp   LISTEN 0      4096     192.168.0.1:10250      0.0.0.0:*    users:(("k3s-server",pid=42945,fd=170))
tcp   LISTEN 0      4096               *:40683            *:*    users:(("k3s-server",pid=42945,fd=145))

and without docker option:

# ss -lnput | grep k3s
tcp   LISTEN 0      4096     192.168.0.1:6443       0.0.0.0:*    users:(("k3s-server",pid=58939,fd=14))
tcp   LISTEN 0      4096       127.0.0.1:6444       0.0.0.0:*    users:(("k3s-server",pid=58939,fd=16))
tcp   LISTEN 0      4096       127.0.0.1:10258      0.0.0.0:*    users:(("k3s-server",pid=58939,fd=202))
tcp   LISTEN 0      4096       127.0.0.1:10259      0.0.0.0:*    users:(("k3s-server",pid=58939,fd=151))
tcp   LISTEN 0      4096       127.0.0.1:10256      0.0.0.0:*    users:(("k3s-server",pid=58939,fd=195))
tcp   LISTEN 0      4096       127.0.0.1:10257      0.0.0.0:*    users:(("k3s-server",pid=58939,fd=160))
tcp   LISTEN 0      4096       127.0.0.1:10248      0.0.0.0:*    users:(("k3s-server",pid=58939,fd=186))
tcp   LISTEN 0      4096       127.0.0.1:10249      0.0.0.0:*    users:(("k3s-server",pid=58939,fd=201))
tcp   LISTEN 0      4096     192.168.0.1:10250      0.0.0.0:*    users:(("k3s-server",pid=58939,fd=182))

0 replies

mcarbonneaux · 2024-04-21T14:57:18Z

mcarbonneaux
Apr 21, 2024

Looks like it's the --streaming-bind-addr flag

@brandond there a solution to fix that before PR while be validated ?

3 replies

brandond Apr 22, 2024
Collaborator

You can use a CI build? You can use any recent commit from one of the master/release branches with the INSTALL_K3S_COMMIT env var when running the install script.

If you don't want do to that, your other options are to install and configure your own cri-dockerd, or just wait.

mcarbonneaux Apr 24, 2024

there no solution like modifing manualy containerd configuration for exemple ?

brandond Apr 24, 2024
Collaborator

it's not in containerd, is it? It's in cri-dockerd. Which does not have any user-modifiable configuration flags.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

k3s will listen on random ports #9724

{{title}}

Replies: 6 comments 8 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

k3s will listen on random ports #9724

bingfengfeifei Mar 12, 2024

Replies: 6 comments · 8 replies

bingfengfeifei Mar 22, 2024 Author

mcarbonneaux Apr 6, 2024

mcarbonneaux Apr 6, 2024

bingfengfeifei Apr 17, 2024 Author

manuelbuil Apr 16, 2024 Collaborator

bingfengfeifei Apr 17, 2024 Author

bingfengfeifei Apr 17, 2024 Author

brandond Apr 17, 2024 Collaborator

brandond Apr 17, 2024 Collaborator

mcarbonneaux Apr 21, 2024

mcarbonneaux Apr 21, 2024

brandond Apr 22, 2024 Collaborator

mcarbonneaux Apr 24, 2024

brandond Apr 24, 2024 Collaborator

bingfengfeifei
Mar 12, 2024

Replies: 6 comments 8 replies

bingfengfeifei
Mar 22, 2024
Author

mcarbonneaux
Apr 6, 2024

mcarbonneaux
Apr 6, 2024

bingfengfeifei Apr 17, 2024
Author

manuelbuil
Apr 16, 2024
Collaborator

bingfengfeifei Apr 17, 2024
Author

bingfengfeifei Apr 17, 2024
Author

brandond Apr 17, 2024
Collaborator

brandond Apr 17, 2024
Collaborator

mcarbonneaux
Apr 21, 2024

mcarbonneaux
Apr 21, 2024

brandond Apr 22, 2024
Collaborator

brandond Apr 24, 2024
Collaborator