iptables on OKD #2400

phoracek · 2019-06-19T14:47:51Z

We fail to configure iptables on OKD since they are not initialized on the host (and maybe we are also missing some kernel modules). I think we have two options: Enable and initialize iptables on OKD host or use nftables in case we run on OKD.

{"component":"virt-launcher","level":"error","msg":"failed to create nat rules for vm error: running [/usr/sbin/iptables -t nat -N KUBEVIRT_PREINBOUND --wait]: exit status 3: modprobe: FATAL: Module ip_tables not found in directory /lib/modules/4.18.0-80.1.2.el8_0.x86_64\niptables v1.6.2: can't initialize iptables table `nat': Table does not exist (do you need to insmod?)\nPerhaps iptables or your kernel needs to be upgraded.\n","pos":"podinterface.go:435","timestamp":"2019-06-19T14:29:18.393389Z"} {"component":"virt-launcher","level":"fatal","msg":"failed to prepared pod networking","pos":"podinterface.go:88","reason":"running [/usr/sbin/iptables -t nat -N KUBEVIRT_PREINBOUND --wait]: exit status 3: modprobe: FATAL: Module ip_tables not found in directory /lib/modules/4.18.0-80.1.2.el8_0.x86_64\niptables v1.6.2: can't initialize iptables table `nat': Table does not exist (do you need to insmod?)\nPerhaps iptables or your kernel needs to be upgraded.\n","timestamp":"2019-06-19T14:29:18.393440Z"} panic: running [/usr/sbin/iptables -t nat -N KUBEVIRT_PREINBOUND --wait]: exit status 3: modprobe: FATAL: Module ip_tables not found in directory /lib/modules/4.18.0-80.1.2.el8_0.x86_64 iptables v1.6.2: can't initialize iptables table `nat': Table does not exist (do you need to insmod?) Perhaps iptables or your kernel needs to be upgraded. goroutine 69 [running]: kubevirt.io/kubevirt/pkg/virt-launcher/virtwrap/network.(*PodInterface).Plug(0x22a98b0, 0xc000740300, 0xc00014e630, 0xc00034bbf0, 0xc000330000, 0x14dec7d, 0x4, 0x10, 0x12f7140) pkg/virt-launcher/virtwrap/network/podinterface.go:89 +0x536 kubevirt.io/kubevirt/pkg/virt-launcher/virtwrap/network.SetupNetworkInterfaces(0xc000740300, 0xc000330000, 0x0, 0x0) pkg/virt-launcher/virtwrap/network/network.go:83 +0x520 kubevirt.io/kubevirt/pkg/virt-launcher/virtwrap.(*LibvirtDomainManager).preStartHook(0xc0004a9340, 0xc000740300, 0xc000330000, 0x1698e00, 0x0, 0x166c820) pkg/virt-launcher/virtwrap/manager.go:728 +0x3ad kubevirt.io/kubevirt/pkg/virt-launcher/virtwrap.(*LibvirtDomainManager).SyncVMI(0xc0004a9340, 0xc000740300, 0xc0003ca900, 0x0, 0x0, 0x0) pkg/virt-launcher/virtwrap/manager.go:881 +0x5e2 kubevirt.io/kubevirt/pkg/virt-launcher/virtwrap/cmd-server.(*Launcher).SyncVirtualMachine(0xc00001c460, 0x1686540, 0xc00034acf0, 0xc00034ad20, 0xc00001c460, 0xc00034ac60, 0x135df40) pkg/virt-launcher/virtwrap/cmd-server/server.go:159 +0x81 kubevirt.io/kubevirt/pkg/handler-launcher-com/cmd/v1._Cmd_SyncVirtualMachine_Handler(0x1426cc0, 0xc00001c460, 0x1686540, 0xc00034acf0, 0xc00073eb40, 0x0, 0x0, 0x0, 0xc000582b00, 0x55c) bazel-out/k8-fastbuild/bin/pkg/handler-launcher-com/cmd/v1/linux_amd64_stripped/kubevirt_cmd_go_proto%!/(MISSING)kubevirt.io/kubevirt/pkg/handler-launcher-com/cmd/v1/cmd.pb.go:515 +0x23e google.golang.org/grpc.(*Server).processUnaryRPC(0xc00046af00, 0x1692140, 0xc00055e180, 0xc000734300, 0xc00049d2c0, 0x21e5760, 0x0, 0x0, 0x0) external/org_golang_google_grpc/server.go:971 +0x4a2 google.golang.org/grpc.(*Server).handleStream(0xc00046af00, 0x1692140, 0xc00055e180, 0xc000734300, 0x0) external/org_golang_google_grpc/server.go:1250 +0xd61 google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc000018320, 0xc00046af00, 0x1692140, 0xc00055e180, 0xc000734300) external/org_golang_google_grpc/server.go:690 +0x9f created by google.golang.org/grpc.(*Server).serveStreams.func1 external/org_golang_google_grpc/server.go:688 +0xa1 {"component":"virt-launcher","level":"error","msg":"dirty virt-launcher shutdown","pos":"virt-launcher.go:558","reason":"exit status 2","time

Kudos to @cynepco3hahue for reporting the issue.

The text was updated successfully, but these errors were encountered:

phoracek · 2019-06-19T14:48:16Z

/cc @dankenigsberg @SchSeba @booxter @qinqon

cynepco3hahue · 2019-06-19T14:52:14Z

Some details regarding the environment where it happened:

NAME                          STATUS   ROLES    AGE   VERSION             INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                                                   KERNEL-VERSION               CONTAINER-RUNTIME
test-1-hw7kn-master-0         Ready    master   24h   v1.13.4+9252851b0   192.168.126.11   <none>        Red Hat Enterprise Linux CoreOS 410.8.20190612.0 (Ootpa)   4.18.0-80.1.2.el8_0.x86_64   cri-o://1.13.9-1.rhaos4.1.gitd70609a.el8
test-1-hw7kn-worker-0-7w268   Ready    worker   24h   v1.13.4+9252851b0   192.168.126.51   <none>        Red Hat Enterprise Linux CoreOS 410.8.20190612.0 (Ootpa)   4.18.0-80.1.2.el8_0.x86_64   cri-o://1.13.9-1.rhaos4.1.gitd70609a.el8

Kernel modules

[root@test-1-hw7kn-master-0 ~]# lsmod
Module                  Size  Used by
vhost_net              24576  0
vhost                  49152  1 vhost_net
tap                    28672  1 vhost_net
tun                    49152  1 vhost_net
bridge                188416  0
stp                    16384  1 bridge
llc                    16384  2 bridge,stp
dummy                  16384  0
devlink                81920  0
loop                   32768  3
xt_recent              20480  1
veth                   16384  0
xt_statistic           16384  1
nf_conntrack_netlink    49152  0
xt_nat                 16384  1
xt_addrtype            16384  1
vxlan                  61440  0
ip6_udp_tunnel         16384  1 vxlan
udp_tunnel             16384  1 vxlan
openvswitch           155648  53
nf_conntrack_ipv6      20480  51
nf_nat_ipv6            16384  1 openvswitch
nf_conncount           16384  1 openvswitch
nf_defrag_ipv6         20480  2 nf_conntrack_ipv6,openvswitch
xt_conntrack           16384  1
ipt_REJECT             16384  1
nf_reject_ipv4         16384  1 ipt_REJECT
ipt_MASQUERADE         16384  1
xt_comment             16384  1
nft_counter            16384  552
xt_mark                16384  2
nft_compat             20480  784
nft_chain_nat_ipv4     16384  4
nf_conntrack_ipv4      16384  112
nf_defrag_ipv4         16384  1 nf_conntrack_ipv4
nf_nat_ipv4            16384  3 ipt_MASQUERADE,nft_chain_nat_ipv4,openvswitch
nf_nat                 36864  4 nf_nat_ipv6,nf_nat_ipv4,xt_nat,openvswitch
nf_conntrack          155648  11 xt_conntrack,nf_conntrack_ipv6,nf_conntrack_ipv4,nf_nat,nf_nat_ipv6,ipt_MASQUERADE,nf_nat_ipv4,xt_nat,openvswitch,nf_conntrack_netlink,nf_conncount
nf_tables             147456  1420 nft_compat,nft_chain_nat_ipv4,nft_counter
overlay               126976  154
nfnetlink              16384  3 nft_compat,nf_conntrack_netlink,nf_tables
ext4                  733184  1
mbcache                16384  1 ext4
jbd2                  122880  1 ext4
cirrus                 28672  1
ttm                   131072  1 cirrus
drm_kms_helper        200704  1 cirrus
sb_edac                24576  0
kvm_intel             245760  0
syscopyarea            16384  1 drm_kms_helper
sysfillrect            16384  1 drm_kms_helper
sysimgblt              16384  1 drm_kms_helper
fb_sys_fops            16384  1 drm_kms_helper
kvm                   745472  1 kvm_intel
drm                   520192  4 drm_kms_helper,cirrus,ttm
irqbypass              16384  1 kvm
i2c_piix4              24576  0
joydev                 24576  0
pcspkr                 16384  0
virtio_balloon         20480  0
xfs                  1474560  1
libcrc32c              16384  4 nf_conntrack,nf_nat,openvswitch,xfs
dm_multipath           32768  0
ata_generic            16384  0
crct10dif_pclmul       16384  0
crc32_pclmul           16384  0
crc32c_intel           24576  3
ata_piix               36864  0
libata                274432  2 ata_piix,ata_generic
ghash_clmulni_intel    16384  0
serio_raw              16384  0
virtio_net             53248  0
virtio_console         36864  0
net_failover           24576  1 virtio_net
virtio_blk             20480  3
failover               16384  1 net_failover
dm_mirror              28672  0
dm_region_hash         20480  1 dm_mirror
dm_log                 20480  2 dm_region_hash,dm_mirror
dm_mod                151552  3 dm_multipath,dm_log,dm_mirror
be2iscsi              126976  0
bnx2i                  57344  0
cnic                   73728  1 bnx2i
uio                    20480  1 cnic
cxgb4i                 49152  0
cxgb4                 413696  1 cxgb4i
libcxgbi               65536  1 cxgb4i
libcxgb                20480  1 libcxgbi
qla4xxx               307200  0
iscsi_boot_sysfs       16384  2 be2iscsi,qla4xxx
iscsi_tcp              20480  0
libiscsi_tcp           24576  3 libcxgbi,iscsi_tcp,cxgb4i
libiscsi               61440  7 be2iscsi,libiscsi_tcp,bnx2i,libcxgbi,iscsi_tcp,qla4xxx,cxgb4i
scsi_transport_iscsi   106496  6 be2iscsi,bnx2i,libcxgbi,iscsi_tcp,qla4xxx,libiscsi

I also can see it has iptables and iptables has nat table.

phoracek · 2019-06-19T14:58:20Z

The log complains about ip_tables module though.

cynepco3hahue · 2019-06-19T15:09:34Z

I meant that it has iptables binary but not the module, so I believe we need support for nf_tables, likes I said to Petr, I do not believe that RHCOS will add additional kernel module only because we are working only with iptables but we can try to speak with them.

cynepco3hahue · 2019-06-19T15:14:09Z

I think it is an urgent issue because it blocks all functional testing on top of OKD.

dankenigsberg · 2019-06-19T15:22:17Z

@phoracek Why do we touch iptables? Only for masquerade? Or also for something else?

phoracek · 2019-06-19T15:28:39Z

@dankenigsberg yes, it seems to be only under masquerade code.

SchSeba · 2019-06-19T16:39:57Z

I will take a look on that

SchSeba · 2019-06-21T15:55:28Z

Related to:

This PR check if iptables are enable and if not use nftables to configure rules for the masquerade interface binding. Fixes kubevirt#2400

fabiand · 2019-07-16T14:56:13Z

@cynepco3hahue @SchSeba anything blocking this work?

SchSeba · 2019-07-16T15:02:48Z

@fabiand waiting for this PR #2487 to be merge and I am fixing the review comments here #2430

fabiand · 2019-07-16T15:10:15Z

Thanks, that does not sound to bad.

This PR check if iptables are enable and if not use nftables to configure rules for the masquerade interface binding. Fixes kubevirt#2400

SchSeba added the sig/network label Jun 19, 2019

SchSeba self-assigned this Jun 19, 2019

SchSeba mentioned this issue Jun 25, 2019

Update libvirt image to 5.4.0 #2428

Closed

SchSeba added a commit to SchSeba/kubevirt that referenced this issue Jun 25, 2019

use nftable rules for coreOS servers on masquerade

bacaca5

This PR check if iptables are enable and if not use nftables to configure rules for the masquerade interface binding. Fixes kubevirt#2400

SchSeba added a commit to SchSeba/kubevirt that referenced this issue Jun 25, 2019

use nftable rules for coreOS servers on masquerade

7090993

This PR check if iptables are enable and if not use nftables to configure rules for the masquerade interface binding. Fixes kubevirt#2400

SchSeba mentioned this issue Jun 26, 2019

Use nftable rules for coreOS servers on masquerade #2430

Merged

SchSeba added a commit to SchSeba/kubevirt that referenced this issue Jun 30, 2019

use nftable rules for coreOS servers on masquerade

53469c5

This PR check if iptables are enable and if not use nftables to configure rules for the masquerade interface binding. Fixes kubevirt#2400

dankenigsberg mentioned this issue Jul 10, 2019

BZ1725848 Release Notes openshift/openshift-docs#15805

Merged

SchSeba added a commit to SchSeba/kubevirt that referenced this issue Jul 11, 2019

use nftable rules for coreOS servers on masquerade

8da4de2

This PR check if iptables are enable and if not use nftables to configure rules for the masquerade interface binding. Fixes kubevirt#2400

SchSeba added a commit to SchSeba/kubevirt that referenced this issue Jul 11, 2019

use nftable rules for coreOS servers on masquerade

c633f54

This PR check if iptables are enable and if not use nftables to configure rules for the masquerade interface binding. Fixes kubevirt#2400

SchSeba added a commit to SchSeba/kubevirt that referenced this issue Jul 16, 2019

use nftable rules for coreOS servers on masquerade

7b17549

This PR check if iptables are enable and if not use nftables to configure rules for the masquerade interface binding. Fixes kubevirt#2400

SchSeba added a commit to SchSeba/kubevirt that referenced this issue Jul 17, 2019

use nftable rules for coreOS servers on masquerade

50fbcc6

This PR check if iptables are enable and if not use nftables to configure rules for the masquerade interface binding. Fixes kubevirt#2400

SchSeba added a commit to SchSeba/kubevirt that referenced this issue Jul 17, 2019

use nftable rules for coreOS servers on masquerade

76fbc82

This PR check if iptables are enable and if not use nftables to configure rules for the masquerade interface binding. Fixes kubevirt#2400

SchSeba added a commit to SchSeba/kubevirt that referenced this issue Jul 22, 2019

use nftable rules for coreOS servers on masquerade

bbf3e8e

This PR check if iptables are enable and if not use nftables to configure rules for the masquerade interface binding. Fixes kubevirt#2400

SchSeba added a commit to SchSeba/kubevirt that referenced this issue Jul 28, 2019

use nftable rules for coreOS servers on masquerade

79feaa2

This PR check if iptables are enable and if not use nftables to configure rules for the masquerade interface binding. Fixes kubevirt#2400

cynepco3hahue closed this as completed in #2430 Jul 29, 2019

SchSeba added a commit to SchSeba/kubevirt that referenced this issue Jul 29, 2019

use nftable rules for coreOS servers on masquerade

206cb6a

This PR check if iptables are enable and if not use nftables to configure rules for the masquerade interface binding. Fixes kubevirt#2400

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

iptables on OKD #2400

iptables on OKD #2400

phoracek commented Jun 19, 2019 •

edited

phoracek commented Jun 19, 2019

cynepco3hahue commented Jun 19, 2019 •

edited

phoracek commented Jun 19, 2019

cynepco3hahue commented Jun 19, 2019

cynepco3hahue commented Jun 19, 2019

dankenigsberg commented Jun 19, 2019

phoracek commented Jun 19, 2019

SchSeba commented Jun 19, 2019

SchSeba commented Jun 21, 2019

fabiand commented Jul 16, 2019

SchSeba commented Jul 16, 2019

fabiand commented Jul 16, 2019

iptables on OKD #2400

iptables on OKD #2400

Comments

phoracek commented Jun 19, 2019 • edited

phoracek commented Jun 19, 2019

cynepco3hahue commented Jun 19, 2019 • edited

phoracek commented Jun 19, 2019

cynepco3hahue commented Jun 19, 2019

cynepco3hahue commented Jun 19, 2019

dankenigsberg commented Jun 19, 2019

phoracek commented Jun 19, 2019

SchSeba commented Jun 19, 2019

SchSeba commented Jun 21, 2019

fabiand commented Jul 16, 2019

SchSeba commented Jul 16, 2019

fabiand commented Jul 16, 2019

phoracek commented Jun 19, 2019 •

edited

cynepco3hahue commented Jun 19, 2019 •

edited