New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kube-proxy should modprobe iptables modules it needs #12459

Closed
asridharan opened this Issue Aug 10, 2015 · 26 comments

Comments

Projects
None yet
9 participants
@asridharan
Copy link
Contributor

asridharan commented Aug 10, 2015

Description of problem
I am trying to run kubernetes on a docker-based local setup using the following instructions:
http://kubernetes.io/v1.0/docs/getting-started-guides/docker.html

NOTE: I initially filed the issue with docker #15406. But, after analyses by the docker developers looks like all the errors are contained with the kubernetes code-base, so opening the issue here.

All the kuberenetes containers start cleanly, except the "kube-proxy" container.

~ $ docker ps --all
CONTAINER ID        IMAGE                                        COMMAND                CREATED             STATUS                           PORTS               NAMES
34fb39961d7a        gcr.io/google_containers/hyperkube:v0.21.2   "/hyperkube proxy --   54 minutes ago      Exited (255) 54 minutes ago                          dreamy_morse                                                                                             
bd4c5c6e89ab        gcr.io/google_containers/hyperkube:v0.21.2   "/hyperkube proxy --   54 minutes ago                                                           tender_babbage                                                                                           
dd493eaa6d65        gcr.io/google_containers/hyperkube:v0.21.2   "/hyperkube proxy --   About an hour ago   Exited (255) About an hour ago                       berserk_colden                                                                                           
88ca0fb914be        gcr.io/google_containers/hyperkube:v0.21.2   "/hyperkube schedule   About an hour ago   Up About an hour                                     k8s_scheduler.b725e775_k8s-master-127.0.0.1_default_9b44830745c166dfc6d027b0fc2df36d_f864727d            
031b1646492f        gcr.io/google_containers/hyperkube:v0.21.2   "/hyperkube apiserve   About an hour ago   Up About an hour                                     k8s_apiserver.70750283_k8s-master-127.0.0.1_default_9b44830745c166dfc6d027b0fc2df36d_fe01717c            
245658f9a17a        gcr.io/google_containers/hyperkube:v0.21.2   "/hyperkube controll   About an hour ago   Up About an hour                                     k8s_controller-manager.aad1ee8f_k8s-master-127.0.0.1_default_9b44830745c166dfc6d027b0fc2df36d_0fd7c4e8   
6566e67ab67e        gcr.io/google_containers/pause:0.8.0         "/pause"               About an hour ago   Up About an hour                                     k8s_POD.e4cc795_k8s-master-127.0.0.1_default_9b44830745c166dfc6d027b0fc2df36d_21bef762                   
ff98d0a62497        gcr.io/google_containers/hyperkube:v0.21.2   "/hyperkube kubelet    About an hour ago   Up About an hour                                     drunk_franklin                                                                                           
97a00a7de5f4        gcr.io/google_containers/etcd:2.0.9          "/usr/local/bin/etcd   About an hour ago   Up About an hour                                     ecstatic_hodgkin                                                                                         

On looking at the output of the kube-proxy container, looks like the remote API is throwing a error while launching the container:

$ docker logs dd493eaa6d65 
W0807 14:26:19.502305       1 server.go:86] Failed to start in resource-only container "/kube-proxy": mountpoint for cgroup not found
I0807 14:26:19.502667       1 proxier.go:121] Setting proxy IP to 192.168.75.10 and initializing iptables
F0807 14:26:19.509773       1 server.go:101] Unable to create proxer: failed to initialize iptables: error appending rule: exit status 1: iptables: No chain/target/match by that name.

This is a bit odd, cause all the cgroup mounts seem to be in place:

~ $ mount | grep cgroup
cgroup_root on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,relatime,size=10240k,mode=755)
openrc on /sys/fs/cgroup/openrc type cgroup (rw,nosuid,nodev,noexec,relatime,release_agent=/lib64/rc/sh/cgroup-release-agent.sh,name=openrc)
cpuset on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cpu on /sys/fs/cgroup/cpu type cgroup (rw,nosuid,nodev,noexec,relatime,cpu)
cpuacct on /sys/fs/cgroup/cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct)
memory on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
devices on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
freezer on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
blkio on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
perf_event on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
hugetlb on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)

Also, there doesn't seem to be an issue with starting any of the other containers. The only difference, I could see, in starting the kube-proxy container versus the other containers (etcd, api-server, scheduler) is that kube-proxy needs to be run with --priveleged . However, I did try running a ubuntu container with --privileged and it seems to be coming up fine.

docker version:
Client version: 1.7.1
Client API version: 1.19
Go version (client): go1.4
Git commit (client): 786b29d
OS/Arch (client): linux/amd64
Server version: 1.7.1
Server API version: 1.19
Go version (server): go1.4
Git commit (server): 786b29d
OS/Arch (server): linux/amd64

docker info:
Containers: 9
Images: 97
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 115
Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.18.4-aufs
Operating System: Gentoo/Linux
CPUs: 4
Total Memory: 3.862 GiB
Name: stitchserver
ID: NSPJ:6OQ3:FURV:IIES:UIEA:WO65:P3QG:SW2N:NFA6:UTBW:ATHA:4XLF

uname -a:

Linux stitchserver 3.18.4-aufs #4 SMP Thu Aug 6 10:53:19 PDT 2015 x86_64 QEMU Virtual CPU version 2.1.0 GenuineIntel GNU/Linux

Environment details (AWS, VirtualBox, physical, etc.):
Gentoo VM, running a custom kernel. Ran the check-confish.sh script to make sure all kernel configs are turned on:

warning: /proc/config.gz does not exist, searching other paths for kernel config...
info: reading kernel config from /boot/config-3.18.4-aufs ...

Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_DEVPTS_MULTIPLE_INSTANCES: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MACVLAN: enabled
- CONFIG_VETH: enabled
- CONFIG_BRIDGE: enabled (as module)
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_NF_NAT_IPV4: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_NF_NAT_NEEDED: enabled
- CONFIG_POSIX_MQUEUE: enabled

Optional Features:
- CONFIG_MEMCG_SWAP: enabled
- CONFIG_MEMCG_SWAP_ENABLED: enabled
- CONFIG_RESOURCE_COUNTERS: enabled
- CONFIG_BLK_CGROUP: enabled
- CONFIG_IOSCHED_CFQ: enabled
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CFS_BANDWIDTH: missing
- Storage Drivers:
  - "aufs":
    - CONFIG_AUFS_FS: enabled
    - CONFIG_EXT4_FS_POSIX_ACL: enabled
    - CONFIG_EXT4_FS_SECURITY: enabled
  - "btrfs":
    - CONFIG_BTRFS_FS: missing
  - "devicemapper":
    - CONFIG_BLK_DEV_DM: enabled
    - CONFIG_DM_THIN_PROVISIONING: enabled
    - CONFIG_EXT4_FS: enabled
    - CONFIG_EXT4_FS_POSIX_ACL: enabled
    - CONFIG_EXT4_FS_SECURITY: enabled
  - "overlay":
    - CONFIG_OVERLAY_FS: missing
    - CONFIG_EXT4_FS_SECURITY: enabled
    - CONFIG_EXT4_FS_POSIX_ACL: enabled
  - "zfs":
    - /dev/zfs: missing
    - zfs command: missing
    - zpool command: missing

How reproducible:
Every time.

Steps to Reproduce:

  1. Follow the steps listed in http://kubernetes.io/v1.0/docs/getting-started-guides/docker.html
  2. After each step maker sure the docker containers have started correctly.
  3. After step 3 you will the kube-proxy exits with the above issue.

Actual Results:
kube-proxy container fails to start

Expected Results:
kube-proxy should start and configure the iptables with the right entries.

Additional info:

This could be something specific to Gentoo. I am not seeing other users (on Ubuntu for e.g.,) complaining about kubernetes not starting.

@a-robinson

This comment has been minimized.

Copy link
Member

a-robinson commented Aug 10, 2015

Handing off to @brendandburns because hyperkube, feel free to pass on

@brendandburns

This comment has been minimized.

Copy link
Contributor

brendandburns commented Aug 11, 2015

Will try to reproduce.

@jakexks

This comment has been minimized.

Copy link
Contributor

jakexks commented Aug 12, 2015

I'm also seeing this issue on CoreOS stable (723.3.0)
Scratch that, my problem was unrelated but I still get the cryptic "server.go:85] Failed to start in resource-only container "/kube-proxy": mountpoint for cgroup not found" error which threw me off while reading through the logs!

@asridharan

This comment has been minimized.

Copy link
Contributor Author

asridharan commented Aug 13, 2015

Some more data on this:

  • Ran the hyperkube proxy container with debug level logs. Got the following output
 docker logs 2800a9ad3604
W0813 19:31:57.819018       1 server.go:86] Failed to start in resource-only container "/kube-proxy": mountpoint for cgroup not found
I0813 19:31:57.819251       1 util.go:499] Default route transits interface "eth0"
I0813 19:31:57.819427       1 util.go:346] Interface eth0 is up
I0813 19:31:57.819508       1 util.go:391] Interface "eth0" has 2 addresses :[192.168.75.10/24 fe80::5054:ff:fe0f:8f70/64].
I0813 19:31:57.819559       1 util.go:358] Checking addr  192.168.75.10/24.
I0813 19:31:57.819569       1 util.go:367] IP found 192.168.75.10
I0813 19:31:57.819577       1 util.go:397] valid IPv4 address for interface "eth0" found as 192.168.75.10.
I0813 19:31:57.819584       1 util.go:505] Choosing IP 192.168.75.10
I0813 19:31:57.819606       1 proxier.go:121] Setting proxy IP to 192.168.75.10 and initializing iptables
I0813 19:31:57.819645       1 iptables.go:193] running iptables -N [KUBE-PORTALS-CONTAINER -t nat]
I0813 19:31:57.822749       1 iptables.go:193] running iptables -C [PREROUTING -t nat -m comment --comment handle ClusterIPs; NOTE: this must be before the NodePort rules -j KUBE-PORTALS-CONTAINER]
I0813 19:31:57.824153       1 iptables.go:193] running iptables -I [PREROUTING -t nat -m comment --comment handle ClusterIPs; NOTE: this must be before the NodePort rules -j KUBE-PORTALS-CONTAINER]
F0813 19:31:57.826694       1 server.go:101] Unable to create proxer: failed to initialize iptables: error appending rule: exit status 1: iptables: No chain/target/match by that name.

Wanted to see why the iptables command was throwing an error, so started the hyperkube proxy container in bash mode, and ran the commands individually:

root@3c22ed75fe0f:/# iptables -t nat -N KUBE-PORTALS-CONTAINER
root@3c22ed75fe0f:/# iptables --list -t nat
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         

Chain KUBE-PORTALS-CONTAINER (0 references)   <<<< New chain is added
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            


root@3c22ed75fe0f:/# iptables -v -C PREROUTING -t nat -m comment --comment "handle ClusterIPs; NOTE: this must be before the NodePort rules" -j KUBE-PORTALS-CONTAINER
KUBE-PORTALS-CONTAINER  all opt -- in * out *  0.0.0.0/0  -> 0.0.0.0/0   /* handle ClusterIPs; NOTE: this must be before the NodePort rules */
iptables: No chain/target/match by that name.
root@3c22ed75fe0f:/# iptables -v -I PREROUTING -t nat -m comment --comment "handle ClusterIPs; NOTE: this must be before the NodePort rules" -j KUBE-PORTALS-CONTAINER
KUBE-PORTALS-CONTAINER  all opt -- in * out *  0.0.0.0/0  -> 0.0.0.0/0   /* handle ClusterIPs; NOTE: this must be before the NodePort rules */
iptables: No chain/target/match by that name.
root@3c22ed75fe0f:/# 

Looks like the last command insert is cribbing cause there are no rules in the KUBE-PORTALS-CONTAINER chain ?? So should there be rules inserted before it tries launching these commands?? Not sure why this is so specific to this environment.

@asridharan

This comment has been minimized.

Copy link
Contributor Author

asridharan commented Aug 15, 2015

Finally got this working :). Turns out there were a bunch of iptables kernel modules that hadn't been loaded and hence iptables was cribbing on the match command. Failing silently without actually indicating the modules were not present. Ran the local docker-based setup on a ubuntu 14.04 and figured out the missing kernel modules. This is what my gentoo lsmod output looks like after installing all the required kernel modules:

$ lsmod 
Module                  Size  Used by
xt_REDIRECT             1614  1 
xt_nat                  1785  1 
xt_comment               907  6 
nft_reject_ipv4         1057  0 
nft_reject              1317  1 nft_reject_ipv4
nf_tables              46411  1 nft_reject_ipv4
xt_conntrack            3113  1 
ipt_MASQUERADE          1053  1 
nf_nat_masquerade_ipv4     1769  1 ipt_MASQUERADE
iptable_nat             1735  1 
nf_nat_ipv4             4859  1 iptable_nat
xt_addrtype             2765  4 
iptable_filter          1368  1 
br_netfilter           10168  0 
nf_nat                 11713  4 nf_nat_ipv4,xt_nat,nf_nat_masquerade_ipv4,xt_REDIRECT
bridge                 80857  1 br_netfilter
stp                     1533  1 bridge
llc                     3409  2 stp,bridge
tun                    19649  0 
virtio_net             20521  0 
r8169                  69210  0 
8139too                20605  0 
8139cp                 20976  0 
mii                     3875  3 r8169,8139cp,8139too

Hadn't loaded modules like xt_comment, xt_REDIRECT, nft_reject .... Once I loaded the modules the kube-proxy container was able to start and was able to update the iptables correctly:

$ docker ps
CONTAINER ID        IMAGE                                  COMMAND                CREATED             STATUS              PORTS               NAMES
e5becc5e8ca6        hyperkube-local:0.0                    "/hyperkube proxy --   7 minutes ago       Up 7 minutes                            trusting_torvalds                                                                                       
d88f027ef946        hyperkube-local:0.0                    "/hyperkube apiserve   8 minutes ago       Up 8 minutes                            k8s_apiserver.9bb9f9f1_k8s-master-127.0.0.1_default_21ab8199930ad1907a58741ed16a2445_48e6f0bf           
7de1843455ab        hyperkube-local:0.0                    "/hyperkube controll   8 minutes ago       Up 8 minutes                            k8s_controller-manager.496e60c_k8s-master-127.0.0.1_default_21ab8199930ad1907a58741ed16a2445_aa408fcc   
936c21ffe734        hyperkube-local:0.0                    "/hyperkube schedule   8 minutes ago       Up 8 minutes                            k8s_scheduler.af30def2_k8s-master-127.0.0.1_default_21ab8199930ad1907a58741ed16a2445_7a17bdef           
69bbfc611fbb        gcr.io/google_containers/pause:0.8.0   "/pause"               8 minutes ago       Up 8 minutes                            k8s_POD.e4cc795_k8s-master-127.0.0.1_default_21ab8199930ad1907a58741ed16a2445_ed679bcf                  
d681af54cfbc        hyperkube-local:0.0                    "/hyperkube kubelet    8 minutes ago       Up 8 minutes                            kickass_thompson                                                                                        
5d752c32ebb1        gcr.io/google_containers/etcd:2.0.9    "/usr/local/bin/etcd   8 minutes ago       Up 8 minutes                            stoic_rosalind                                                                                          

Output from iptables:

$ sudo iptables -t nat -L
Password: 
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
KUBE-PORTALS-CONTAINER  all  --  anywhere             anywhere             /* handle ClusterIPs; NOTE: this must be before the NodePort rules */
DOCKER     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL
KUBE-NODEPORT-CONTAINER  all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL /* handle service NodePorts; NOTE: this must be the last rule in the chain */

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
KUBE-PORTALS-HOST  all  --  anywhere             anywhere             /* handle ClusterIPs; NOTE: this must be before the NodePort rules */
DOCKER     all  --  anywhere            !loopback/8           ADDRTYPE match dst-type LOCAL
KUBE-NODEPORT-HOST  all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL /* handle service NodePorts; NOTE: this must be the last rule in the chain */

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
MASQUERADE  all  --  172.17.0.0/16        anywhere            

Chain DOCKER (2 references)
target     prot opt source               destination         

Chain KUBE-NODEPORT-CONTAINER (1 references)
target     prot opt source               destination         

Chain KUBE-NODEPORT-HOST (1 references)
target     prot opt source               destination         

Chain KUBE-PORTALS-CONTAINER (1 references)
target     prot opt source               destination         
REDIRECT   tcp  --  anywhere             10.0.0.1             /* default/kubernetes: */ tcp dpt:https redir ports 42759

Chain KUBE-PORTALS-HOST (1 references)
target     prot opt source               destination         
DNAT       tcp  --  anywhere             10.0.0.1             /* default/kubernetes: */ tcp dpt:https to:192.168.75.10:42759

Should the requirement of kernel modules be documented somewhere. With standard distro's shouldn't bean issue, but for something like Gentoo might be?

PS: if we don't need to document these modules than we can go ahead and close this issue.

@thockin

This comment has been minimized.

Copy link
Member

thockin commented Aug 16, 2015

I'm not above forcefully modprobing them in kube-proxy, if we can get a
concise list. Other distros seem to load them automatically...

On Fri, Aug 14, 2015 at 6:13 PM, asridharan notifications@github.com
wrote:

Finally got this working :). Turns out there were a bunch of iptables
kernel modules that hadn't been loaded and hence iptables was cribbing on
the match command. Failing silently without actually indicating the modules
were not present. Ran the local docker-based setup on a ubuntu 14.04 and
figured out the missing kernel modules. This is what my gentoo lsmod output
looks like after installing all the required kernel modules:

$ lsmod Module Size Used byxt_REDIRECT 1614 1 xt_nat 1785 1 xt_comment 907 6 nft_reject_ipv4 1057 0 nft_reject 1317 1 nft_reject_ipv4nf_tables 46411 1 nft_reject_ipv4xt_conntrack 3113 1 ipt_MASQUERADE 1053 1 nf_nat_masquerade_ipv4 1769 1 ipt_MASQUERADEiptable_nat 1735 1 nf_nat_ipv4 4859 1 iptable_natxt_addrtype 2765 4 iptable_filter 1368 1 br_netfilter 10168 0 nf_nat 11713 4 nf_nat_ipv4,xt_nat,nf_nat_masquerade_ipv4,xt_REDIRECTbridge 80857 1 br_netfilterstp 1533 1 bridgellc 3409 2 stp,bridgetun 19649 0 virtio_net 20521 0 r8169 69210 0 8139too 20605 0 8139cp 20976 0 mii 3875 3 r8169,8139cp,8139too

Hadn't loaded modules like xt_comment, xt_REDIRECT, nft_reject .... Once I
loaded the modules the kube-proxy container was able to start and was able
to update the iptables correctly:

$ docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMESe5becc5e8ca6 hyperkube-local:0.0 "/hyperkube proxy -- 7 minutes ago Up 7 minutes trusting_torvalds d88f027ef946 hyperkube-local:0.0 "/hyperkube apiserve 8 minutes ago Up 8 minutes k8s_apiserver.9bb9f9f1_k8s-master-127.0.0.1_default_21ab8199930ad1907a58741ed16a2445_48e6f0bf 7de1843455ab hyperkube-local:0.0 "/hyperkube controll 8 minutes ago Up 8 minutes k8s_controller-manager.496e60c_k8s-master-127.0.0.1_default_21ab8199930ad1907a58741ed16a2445_aa408fcc 936c21ffe734 hyperkube-local:0.0 "/hyperkube schedule 8 minutes ago Up 8 minutes k8s_scheduler.af30def2_k8s-master-127.0.0.1_default_21ab8199930ad1907a58741ed16a2445_7a17bdef 69bbfc611fbb gcr.io/google_containers/pause:0.8.0 "/pause" 8 minutes ago Up 8 minutes k8s_POD.e4cc795_k8s-master-127.0.0.1_default_21ab8199930ad1907a58741ed16a2445_ed679bcf d681af54cfbc hyperkube-local:0.0 "/hyperkube kubelet 8 minutes ago Up 8 minutes kickass_thompson 5d752c32ebb1 gcr.io/google_containers/etcd:2.0.9 "/usr/local/bin/etcd 8 minutes ago Up 8 minutes stoic_rosalind

Output from iptables:

$ sudo iptables -t nat -LPassword: Chain PREROUTING (policy ACCEPT)target prot opt source destination KUBE-PORTALS-CONTAINER all -- anywhere anywhere /* handle ClusterIPs; NOTE: this must be before the NodePort rules /DOCKER all -- anywhere anywhere ADDRTYPE match dst-type LOCALKUBE-NODEPORT-CONTAINER all -- anywhere anywhere ADDRTYPE match dst-type LOCAL / handle service NodePorts; NOTE: this must be the last rule in the chain /
Chain INPUT (policy ACCEPT)target prot opt source destination
Chain OUTPUT (policy ACCEPT)target prot opt source destination KUBE-PORTALS-HOST all -- anywhere anywhere /
handle ClusterIPs; NOTE: this must be before the NodePort rules /DOCKER all -- anywhere !loopback/8 ADDRTYPE match dst-type LOCALKUBE-NODEPORT-HOST all -- anywhere anywhere ADDRTYPE match dst-type LOCAL / handle service NodePorts; NOTE: this must be the last rule in the chain /
Chain POSTROUTING (policy ACCEPT)target prot opt source destination MASQUERADE all -- 172.17.0.0/16 anywhere
Chain DOCKER (2 references)target prot opt source destination
Chain KUBE-NODEPORT-CONTAINER (1 references)target prot opt source destination
Chain KUBE-NODEPORT-HOST (1 references)target prot opt source destination
Chain KUBE-PORTALS-CONTAINER (1 references)target prot opt source destination REDIRECT tcp -- anywhere 10.0.0.1 /
default/kubernetes: / tcp dpt:https redir ports 42759
Chain KUBE-PORTALS-HOST (1 references)target prot opt source destination DNAT tcp -- anywhere 10.0.0.1 /
default/kubernetes: */ tcp dpt:https to:192.168.75.10:42759

Should the requirement of kernel modules be documented somewhere. With
standard distro's shouldn't bean issue, but for something like Gentoo might
be?

PS: if we don't need to document these modules than we can go ahead and
close this issue.


Reply to this email directly or view it on GitHub
#12459 (comment)
.

@asridharan

This comment has been minimized.

Copy link
Contributor Author

asridharan commented Aug 16, 2015

Will try to put a list here. Few of the modules, in my case, were compiled
into my kernel so will have to figure those out.

Thanks,
Avinash

On Sat, Aug 15, 2015 at 9:31 PM, Tim Hockin notifications@github.com
wrote:

I'm not above forcefully modprobing them in kube-proxy, if we can get a
concise list. Other distros seem to load them automatically...

On Fri, Aug 14, 2015 at 6:13 PM, asridharan notifications@github.com
wrote:

Finally got this working :). Turns out there were a bunch of iptables
kernel modules that hadn't been loaded and hence iptables was cribbing on
the match command. Failing silently without actually indicating the
modules
were not present. Ran the local docker-based setup on a ubuntu 14.04 and
figured out the missing kernel modules. This is what my gentoo lsmod
output
looks like after installing all the required kernel modules:

$ lsmod Module Size Used byxt_REDIRECT 1614 1 xt_nat 1785 1 xt_comment
907 6 nft_reject_ipv4 1057 0 nft_reject 1317 1 nft_reject_ipv4nf_tables
46411 1 nft_reject_ipv4xt_conntrack 3113 1 ipt_MASQUERADE 1053 1
nf_nat_masquerade_ipv4 1769 1 ipt_MASQUERADEiptable_nat 1735 1 nf_nat_ipv4
4859 1 iptable_natxt_addrtype 2765 4 iptable_filter 1368 1 br_netfilter
10168 0 nf_nat 11713 4
nf_nat_ipv4,xt_nat,nf_nat_masquerade_ipv4,xt_REDIRECTbridge 80857 1
br_netfilterstp 1533 1 bridgellc 3409 2 stp,bridgetun 19649 0 virtio_net
20521 0 r8169 69210 0 8139too 20605 0 8139cp 20976 0 mii 3875 3
r8169,8139cp,8139too

Hadn't loaded modules like xt_comment, xt_REDIRECT, nft_reject .... Once
I
loaded the modules the kube-proxy container was able to start and was
able
to update the iptables correctly:

$ docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
NAMESe5becc5e8ca6 hyperkube-local:0.0 "/hyperkube proxy -- 7 minutes ago Up
7 minutes trusting_torvalds d88f027ef946 hyperkube-local:0.0 "/hyperkube
apiserve 8 minutes ago Up 8 minutes
k8s_apiserver.9bb9f9f1_k8s-master-127.0.0.1_default_21ab8199930ad1907a58741ed16a2445_48e6f0bf
7de1843455ab hyperkube-local:0.0 "/hyperkube controll 8 minutes ago Up 8
minutes
k8s_controller-manager.496e60c_k8s-master-127.0.0.1_default_21ab8199930ad1907a58741ed16a2445_aa408fcc
936c21ffe734 hyperkube-local:0.0 "/hyperkube schedule 8 minutes ago Up 8
minutes
k8s_scheduler.af30def2_k8s-master-127.0.0.1_default_21ab8199930ad1907a58741ed16a2445_7a17bdef
69bbfc611fbb gcr.io/google_containers/pause:0.8.0 "/pause" 8 minutes ago
Up 8 minutes
k8s_POD.e4cc795_k8s-master-127.0.0.1_default_21ab8199930ad1907a58741ed16a2445_ed679bcf
d681af54cfbc hyperkube-local:0.0 "/hyperkube kubelet 8 minutes ago Up 8
minutes kickass_thompson 5d752c32ebb1 gcr.io/google_containers/etcd:2.0.9
"/usr/local/bin/etcd 8 minutes ago Up 8 minutes stoic_rosalind

Output from iptables:

$ sudo iptables -t nat -LPassword: Chain PREROUTING (policy
ACCEPT)target prot opt source destination KUBE-PORTALS-CONTAINER all --
anywhere anywhere /* handle ClusterIPs; NOTE: this must be before the
NodePort rules /DOCKER all -- anywhere anywhere ADDRTYPE match dst-type
LOCALKUBE-NODEPORT-CONTAINER all -- anywhere anywhere ADDRTYPE match
dst-type LOCAL /
handle service NodePorts; NOTE: this must be the last
rule in the chain /
Chain INPUT (policy ACCEPT)target prot opt source destination
Chain OUTPUT (policy ACCEPT)target prot opt source destination
KUBE-PORTALS-HOST all -- anywhere anywhere /
handle ClusterIPs; NOTE: this
must be before the NodePort rules /DOCKER all -- anywhere !loopback/8
ADDRTYPE match dst-type LOCALKUBE-NODEPORT-HOST all -- anywhere anywhere
ADDRTYPE match dst-type LOCAL /
handle service NodePorts; NOTE: this must
be the last rule in the chain /
Chain POSTROUTING (policy ACCEPT)target prot opt source destination
MASQUERADE all -- 172.17.0.0/16 anywhere
Chain DOCKER (2 references)target prot opt source destination
Chain KUBE-NODEPORT-CONTAINER (1 references)target prot opt source
destination
Chain KUBE-NODEPORT-HOST (1 references)target prot opt source destination
Chain KUBE-PORTALS-CONTAINER (1 references)target prot opt source
destination REDIRECT tcp -- anywhere 10.0.0.1 /
default/kubernetes: / tcp
dpt:https redir ports 42759
Chain KUBE-PORTALS-HOST (1 references)target prot opt source destination
DNAT tcp -- anywhere 10.0.0.1 /
default/kubernetes: */ tcp dpt:https to:
192.168.75.10:42759

Should the requirement of kernel modules be documented somewhere. With
standard distro's shouldn't bean issue, but for something like Gentoo
might
be?

PS: if we don't need to document these modules than we can go ahead and
close this issue.


Reply to this email directly or view it on GitHub
<
#12459 (comment)

.


Reply to this email directly or view it on GitHub
#12459 (comment)
.

@asridharan

This comment has been minimized.

Copy link
Contributor Author

asridharan commented Aug 17, 2015

I was putting a list together for all the iptables module required, but then realized that modprobe itself might fail if some of the modules might be compiled within the kernel. So, rather then taking care of every module being present, wouldn't it be better to throw a more elaborate error and exit when iptables commands cannot be executed?

The iptables command being executed are pre-canned so we know the match and target for each command. If the command fails it will be most probably because of a missing match or target. Can't we append a kubernetes specific error (say specific match module or target module might be missing) and exit? Then it will be up to the user to get his kernel configuration right.

@thockin

This comment has been minimized.

Copy link
Member

thockin commented Aug 17, 2015

I'd rather TRY to load modules we know we need first. If that succeeds,
hooray. If that is insufficient, then complain loudly. Better to not make
admins do it.

On Mon, Aug 17, 2015 at 3:14 PM, asridharan notifications@github.com
wrote:

I was putting a list together for all the iptables module required, but
then realized that modprobe itself might fail if some of the modules might
be compiled within the kernel. So, rather then taking care of every module
being present, wouldn't it be better to throw a more elaborate error and
exit when iptables commands cannot be executed?

The iptables command being executed are pre-canned so we know the match
and target for each command. If the command fails it will be most probably
because of a missing match or target. Can't we append a kubernetes specific
error (say specific match module or target module might be missing) and
exit? Then it will be up to the user to get his kernel configuration right.


Reply to this email directly or view it on GitHub
#12459 (comment)
.

@asridharan

This comment has been minimized.

Copy link
Contributor Author

asridharan commented Aug 17, 2015

Sounds reasonable. Will put up a list shortly.

@asridharan

This comment has been minimized.

Copy link
Contributor Author

asridharan commented Aug 17, 2015

xt_addrtype
xt_conntrack 
xt_comment   
xt_nat             
xt_REDIRECT 
nf_nat_ipv4
nf_nat_masquerade_ipv4  
nft_reject_ipv4 
nft_reject        
nf_tables        
nf_nat
nf_conntrack
iptable_filter
iptable_nat  

These are the ones that I had to load to get things working (with docker and kubernetes). So I guess we can try doing a modprobe on all these modules (catch being that some might be compiled in kernel), and run the iptables command and crib "vehemently" if an iptables command fails.

@sebgoa

This comment has been minimized.

Copy link
Member

sebgoa commented Aug 24, 2015

I got the same issue on Debian:jessie. the proxy container b.gcr.io/kuar/kube-proxy does not start.
Still trying to figure it out.

@asridharan

This comment has been minimized.

Copy link
Contributor Author

asridharan commented Aug 24, 2015

Run the container starting the kube-proxy with debug level logs. It might
give you more info.

docker run -d --net=host --privileged
gcr.io/google_containers/hyperkube:v0.21.2 /hyperkube proxy
--master=http://127.0.0.1:8080 --v=4

PS: Notice the --v being set to 4 (DEBUG)

On Mon, Aug 24, 2015 at 4:04 AM, runseb notifications@github.com wrote:

I got the same issue on Debian:jessie. the proxy container
b.gcr.io/kuar/kube-proxy does not start.
Still trying to figure it out.


Reply to this email directly or view it on GitHub
#12459 (comment)
.

@thockin thockin changed the title starting container "kube-proxy" fails with : mountpoint for cgroup not found kube-proxy should modprobe iptables modules it needs Aug 28, 2015

@thockin

This comment has been minimized.

Copy link
Member

thockin commented Aug 28, 2015

Renaming this from "starting container "kube-proxy" fails with : mountpoint for cgroup not found" to something more appropriate.

@thockin

This comment has been minimized.

Copy link
Member

thockin commented Aug 28, 2015

I spent some time looking at this today, and I can't reproduce it. Why does Gentoo not auto-load the modules?

@asridharan

This comment has been minimized.

Copy link
Contributor Author

asridharan commented Aug 31, 2015

@thockin , when you say you weren't able to reproduce it, is this on Gentoo or some other distro? Gentoo allows a lot of flexibility in terms of choosing the kernel options, and the specific modules that get loaded. Going through the Gentoo handbook would give a better idea about this. So on a gentoo installation, though a user might have "emerged" the iptables, its not necessary he would have loaded "all" the iptables modules.

Here is a link to the Gentoo handbook : https://wiki.gentoo.org/wiki/Handbook:AMD64

@thockin

This comment has been minimized.

Copy link
Member

thockin commented Aug 31, 2015

My concerns are:

a) I shouldn't have to load iptables modules - every doc I can find says so
b) This is guaranteed going to drift and be perpetually broken on gentoo

The latter is more significant - how do we head-off all the support
requests?

On Sun, Aug 30, 2015 at 11:01 PM, asridharan notifications@github.com
wrote:

@thockin https://github.com/thockin , when you say you weren't able to
reproduce it, is this on Gentoo or some other distro? Gentoo allows a lot
of flexibility in terms of choosing the kernel options, and the specific
modules that get loaded. Going through the Gentoo handbook would give a
better idea about this. So on a gentoo installation, though a user might
have "emerged" the iptables, its not necessary he would have loaded "all"
the iptables modules.


Reply to this email directly or view it on GitHub
#12459 (comment)
.

@asridharan

This comment has been minimized.

Copy link
Contributor Author

asridharan commented Aug 31, 2015

I agree with your concern. That's the reason I was suggesting earlier that we should leave the headache of actually loading the module to the user. We should inform him explicitly of the missing module, based on the iptables failure, but should let the user deal with the actual loading of the module.

We can also document the required modules upfront for people to have a look at, possibly in the pre-reqs or troubleshooting section.

@thockin

This comment has been minimized.

Copy link
Member

thockin commented Aug 31, 2015

yeah, that sounds OK. I thought maybe I actually was supposed to load
modules on all platforms and they just were pre-loaded on mine, but it's
not.

Is thsi something you can help with, since you have Gentoo and I do not?

On Mon, Aug 31, 2015 at 11:45 AM, asridharan notifications@github.com
wrote:

I agree with your concern. That's the reason I was suggesting earlier that
we should leave the headache of actually loading the module to the user. We
should inform him explicitly of the missing module, based on the iptables
failure, but should let the user deal with the actual loading of the
module.

We can also document the required modules upfront for people to have a
look at, possibly in the pre-reqs or troubleshooting section.


Reply to this email directly or view it on GitHub
#12459 (comment)
.

@asridharan

This comment has been minimized.

Copy link
Contributor Author

asridharan commented Aug 31, 2015

Sure,
I can have a look at it. Don't have much experience wil GO, but this should be an incentive to pick it up :)

@kalebwalton

This comment has been minimized.

Copy link

kalebwalton commented Nov 3, 2015

I probably missed it in this thread, but was there something done to resolve this? I see mention of making the output more appropriate, but don't see the commit. Sorry if I'm just spacing on it... and thanks :)

@asridharan

This comment has been minimized.

Copy link
Contributor Author

asridharan commented Nov 3, 2015

@kalebwalton this is my bad. I was supposed to take on this task. Couldn't get to this due to some work constraints so got delayed. Will edit the output of the iptables error and send a PR soon.

@utahcon

This comment has been minimized.

Copy link

utahcon commented Nov 11, 2015

Ran into this error on the CoreOS Alpha image today.

OS Version

Linux infraops-kubenode-004.iad.prod.zulily.com 4.2.0-coreos-r1 #2 SMP Thu Sep 17 04:31:47 UTC 2015 x86_64 Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz GenuineIntel GNU/Linux

Error

Nov 11 04:23:47 infraops-kubenode-005.iad.prod.zulily.com docker[4192]: W1111 04:23:47.127429       1 server.go:86] Failed to start in resource-only container "/kube-proxy": mkdir /sys/fs/cgroup/blkio/kube-proxy: read-only file system
Nov 11 04:23:47 infraops-kubenode-005.iad.prod.zulily.com docker[4192]: F1111 04:23:47.129901       1 server.go:101] Unable to create proxer: failed to initialize iptables: error creating chain "KUBE-PORTALS-CONTAINER": exit status 3: iptables v1.4.21: can't initialize iptables table `nat': Permission denied (you must be root)
Nov 11 04:23:47 infraops-kubenode-005.iad.prod.zulily.com docker[4192]: Perhaps iptables or your kernel needs to be upgraded

Container Definition

apiVersion: v1
kind: Pod
metadata:
name: kube-proxy
namespace: kube-system
spec:
hostNetwork: true
containers:
- name: kube-proxy
  image: gcr.io/google_containers/hyperkube:v1.0.6
  command:
  - /hyperkube
  - proxy
  - --master=https://infraops-kubemaster-001.iad.prod.zulily.com
  - --kubeconfig=/etc/kubernetes/worker-kubeconfig.yaml
  securityContext:
    priviledged: true
  volumeMounts:
    - mountPath: /etc/ssl/certs
      name: ssl-certs
    - mountPath: /etc/kubernetes/work-kubeconfig.yaml
      name: kubeconfig
      readOnly: true
    - mountPath: /etc/kubernetes/ssl
      name: etc-kube-ssl
      readOnly: true
volumes:
  - name: ssl-certs
    hostPath:
      path: /usr/share/ca-certificates
  - name: kubeconfig
    hostPath:
      path: /etc/kubernetes/worker-kubeconfig.yaml
  - name: etc-kube-ssl
    hostPath:
        path: /etc/kubernetes/ssl
@eghobo

This comment has been minimized.

Copy link

eghobo commented Nov 26, 2015

@asridharan: how have you installed br-netfilter to CentOS 7? without it proxy/iptables doesn't work ):

@thockin

This comment has been minimized.

Copy link
Member

thockin commented Nov 26, 2015

I suspect this is a side effect of running kube-proxy in a container and
also trying to run it in a cgroup standalone. Fixed in head.
On Nov 10, 2015 6:07 PM, "Adam Barrett" notifications@github.com wrote:

Ran into this error on the CoreOS Alpha image today.

Nov 11 04:03:18 infraops-kubenode-004.iad.prod.zulily.com docker[4180]: W1111 04:03:18.200810 1 server.go:86] Failed to start in resource-only c
Nov 11 04:03:18 infraops-kubenode-004.iad.prod.zulily.com docker[4180]: F1111 04:03:18.202958 1 server.go:101] Unable to create proxer: failed t
Nov 11 04:03:18 infraops-kubenode-004.iad.prod.zulily.com docker[4180]: Perhaps iptables or your kernel needs to be upgraded.

here is my container definition:

apiVersion: v1
kind: Pod
metadata:
name: kube-proxy
namespace: kube-system
spec:
hostNetwork: true
containers:

  • name: kube-proxy
    image: gcr.io/google_containers/hyperkube:v1.0.6
    command:
    • /hyperkube
    • proxy
    • --master=https://infraops-kubemaster-001.iad.prod.zulily.com
    • --kubeconfig=/etc/kubernetes/worker-kubeconfig.yaml
      securityContext:
      priviledged: true
      volumeMounts:
      • mountPath: /etc/ssl/certs
        name: ssl-certs
      • mountPath: /etc/kubernetes/work-kubeconfig.yaml
        name: kubeconfig
        readOnly: true
      • mountPath: /etc/kubernetes/ssl
        name: etc-kube-ssl
        readOnly: true
        volumes:
    • name: ssl-certs
      hostPath:
      path: /usr/share/ca-certificates
    • name: kubeconfig
      hostPath:
      path: /etc/kubernetes/worker-kubeconfig.yaml
    • name: etc-kube-ssl
      hostPath:
      path: /etc/kubernetes/ssl


Reply to this email directly or view it on GitHub
#12459 (comment)
.

@eghobo

This comment has been minimized.

Copy link

eghobo commented Nov 26, 2015

@thockin: at CentOS 7 I tried both scenarios, see errors below

container

W1124 22:44:06.883500       1 server.go:200] Failed to start in resource-only container "/kube-proxy": open /sys/fs/cgroup/cpu/kube-proxy/cpu.shares: no such file or directory
E1124 22:44:06.911791       1 proxier.go:193] Error removing pure-iptables proxy rule: error checking rule: exit status 2: iptables v1.4.21: Couldn't load target `KUBE-SERVICES':No such file or directory
Try `iptables -h' or 'iptables --help' for more information.
E1124 22:44:06.912930       1 proxier.go:197] Error removing pure-iptables proxy rule: error checking rule: exit status 2: iptables v1.4.21: Couldn't load target `KUBE-SERVICES':No such file or directory
Try `iptables -h' or 'iptables --help' for more information.

service

Nov 25 05:51:56 master-54788761-1-91199842 kube-proxy[14793]: W1125 05:51:56.344747   14793 server.go:200] Failed to start in resource-only container "/kube-proxy": write /sys/fs/cgroup/memory/kube-proxy/memory.swappiness: invalid argument
Nov 25 05:51:56 master-54788761-1-91199842 kube-proxy[14793]: E1125 05:51:56.352732   14793 proxier.go:197] Error removing userspace rule: error checking rule: exit status 2: iptables v1.4.21: Couldn't load target `KUBE-PORTALS-HOST':No such file or directory
Nov 25 05:51:56 master-54788761-1-91199842 kube-proxy[14793]: Try `iptables -h' or 'iptables --help' for more information.
Nov 25 05:51:56 master-54788761-1-91199842 kube-proxy[14793]: E1125 05:51:56.353877   14793 proxier.go:201] Error removing userspace rule: error checking rule: exit status 2: iptables v1.4.21: Couldn't load target `KUBE-PORTALS-CONTAINER':No such file or directory
Nov 25 05:51:56 master-54788761-1-91199842 kube-proxy[14793]: Try `iptables -h' or 'iptables --help' for more information.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment