Flannel setting wrong MTU on AWS #841

sgarg1 · 2017-10-17T15:21:16Z

Flanneld is not detecting the host interface's mtu

Expected Behavior

Flanneld should detect the mtu of the host interface and should set the variable for DOCKER_OPT_MTU to interface MTU - 50, on AWS this should be 9001-50 = 8951. This is what I have in a different cluster running an older version.

Current Behavior

Flanneld version 0.8 which is bundled with CoreOS 1520.6.0 is not detecting the host interface's mtu and is therefore passing the wrong mtu values to docker.

Possible Solution

No idea how to

Steps to Reproduce (for bugs)

Install kubernets 1.7.7 on CoreOs using flanneld with vxlan on AWS.

Your Environment

Flannel version: 0.8
Backend used : vxlan
Etcd version: 3.1.5
Kubernetes version : 1.7.7
Operating System and version: CoreOS 1520.6.0

systemctl cat flanneld

#/usr/lib/systemd/system/flanneld.service
[Unit]
Description=flannel - Network fabric for containers (System Application Container)
Documentation=https://github.com/coreos/flannel
After=etcd.service etcd2.service etcd-member.service
Requires=flannel-docker-opts.service

[Service]
Type=notify
Restart=always
RestartSec=10s
TimeoutStartSec=300
LimitNOFILE=40000
LimitNPROC=1048576

Environment="FLANNEL_IMAGE_TAG=v0.8.0"
Environment="FLANNEL_OPTS=--ip-masq=true"
Environment="RKT_RUN_ARGS=--uuid-file-save=/var/lib/coreos/flannel-wrapper.uuid"
EnvironmentFile=-/run/flannel/options.env

ExecStartPre=/sbin/modprobe ip_tables
ExecStartPre=/usr/bin/mkdir --parents /var/lib/coreos /run/flannel
ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/lib/coreos/flannel-wrapper.uuid
ExecStart=/usr/lib/coreos/flannel-wrapper $FLANNEL_OPTS
ExecStop=-/usr/bin/rkt stop --uuid-file=/var/lib/coreos/flannel-wrapper.uuid

[Install]
WantedBy=multi-user.target

/etc/systemd/system/flanneld.service.d/50-network-config.conf
[Service]
OOMScoreAdjust=-500
Restart=always
RestartSec=10

ifconfig

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9001
        inet 10.0.3.41  netmask 255.255.255.0  broadcast 10.0.3.255
        inet6 fe80::4eb:f0ff:fee7:e566  prefixlen 64  scopeid 0x20<link>
        ether 06:eb:f0:e7:e5:66  txqueuelen 1000  (Ethernet)
        RX packets 10427602  bytes 7095274433 (6.6 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 12874424  bytes 10800681256 (10.0 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.1.68.0  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::449b:baff:fed4:2cfd  prefixlen 64  scopeid 0x20<link>
        ether 46:9b:ba:d4:2c:fd  txqueuelen 0  (Ethernet)
        RX packets 2866507  bytes 346105120 (330.0 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2860897  bytes 446694368 (426.0 MiB)
        TX errors 0  dropped 1055 overruns 0  carrier 0  collisions 0

cat /var/run/flannel/flannel_docker_opts.env

DOCKER_OPT_BIP="--bip=10.1.68.1/24"
DOCKER_OPT_IPMASQ="--ip-masq=false"
DOCKER_OPT_MTU="--mtu=1500"

cat /var/run/flannel/subnet.env

FLANNEL_NETWORK=10.1.0.0/16
FLANNEL_SUBNET=10.1.68.1/24
FLANNEL_MTU=1500
FLANNEL_IPMASQ=true

etcd values

 etcdctl get /coreos.com/network/subnets/10.1.68.0-24
{"PublicIP":"10.0.3.41","BackendType":"vxlan","BackendData":{"VtepMAC":"46:9b:ba:d4:2c:fd"}}

# etcdctl get /coreos.com/network/config
{ "Network": "10.1.0.0/16", "Backend": { "Type": "vxlan" } }

The text was updated successfully, but these errors were encountered:

sgarg1 · 2017-10-18T11:29:54Z

We upgraded to the Alpha version CoreOS 1562.1.0 with Flannel 0.9 and the problem still persists.

tomdee · 2017-10-20T19:39:14Z

@sgarg1 This is strange indeed. Could you share some of the output from flannel when it starts? You should see something like

I1020 12:01:57.728043    6015 main.go:470] Determining IP address of default interface
I1020 12:01:57.728273    6015 main.go:483] Using interface with name wlp58s0 and address 192.168.43.73
I1020 12:01:57.728283    6015 main.go:500] Defaulting external address to interface address (192.168.43.73)

This will confirm that it's actually choosing eth0 as the external address.

You can also run ip -d link show dev flannel.1 and check that you get the right device listed on the last line

11: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT group default 
    link/ether 0a:96:29:53:fa:33 brd ff:ff:ff:ff:ff:ff promiscuity 0 
    vxlan id 1 local 10.11.12.13 dev dummy0 srcport 0 0 dstport 8472 nolearning ageing 300 udpcsum addrgenmode none

tomdee · 2017-10-20T19:41:59Z

Also, if I create a dummy device

ip link add type dummy && ip link set dev dummy0 mtu 5000 && ip link set dev dummy0 up && ip addr add 10.11.12.13/32 dev dummy0

And start flannel

sudo dist/flanneld --iface dummy0

Then I see the correct MTU

cat /run/flannel/subnet.env
FLANNEL_NETWORK=10.105.128.0/21
FLANNEL_SUBNET=10.105.133.65/26
FLANNEL_MTU=4950
FLANNEL_IPMASQ=false

I'm going to close since I can't repro but please provide more info and I'll re-open if we can repro it.

mikkeloscar · 2017-11-08T11:31:18Z

I don't see anywhere in the vxlan backend that would actually change the MTU it just sets up the device which gets the default MTU of 1500 afaict.

@tomdee Did you use vxlan in the example above? I see that the udp backend as code for setting the MTU, maybe this is used? (https://github.com/coreos/flannel/blob/master/backend/udp/udp_network.go#L120)

Whether it makes sense to change the MTU when using vxlan I don't know?

tomdee · 2017-11-09T21:12:20Z

@mikkeloscar Yes I was using vxlan. I 'm pretty sure that the correct MTU is automatically set by linux (to the "host" interface -20)

mikkeloscar · 2017-11-09T22:32:56Z

We run 100s of Container Linux stable nodes on AWS and we see the same 1500 MTU value for all interfaces (except eth0 which is 9001). If I change the default MTU with a /etc/systemd/network/99-default.link file then I get that instead of 1500 but not -20.

This is our node configuration fwiw: https://github.com/zalando-incubator/kubernetes-on-aws/blob/dev/cluster/userdata-worker.yaml

tomdee · 2017-11-10T00:22:19Z

Does this resolve to an IP address on an interface that has the 9000 MTU?

mikkeloscar · 2017-11-10T07:30:10Z

Does this resolve to an IP address on an interface that has the 9000 MTU?

I'm sorry, I don't fully understand what you mean? Do you mean if flannel uses the eth0 interface? It always does, example from one flannel log:

1031 main.go:470] Determining IP address of default interface
1109 15:31:22.125676    1031 main.go:483] Using interface with name eth0 and address 172.31.9.74
1109 15:31:22.125695    1031 main.go:500] Defaulting external address to interface address (172.31.9.74)
1109 15:31:22.125785    1031 main.go:235] Created subnet manager: Etcd Local Manager with Previous Subnet: 0.0.0.0/0
1109 15:31:22.125796    1031 main.go:238] Installing signal handlers
1109 15:31:22.144787    1031 main.go:348] Found network config - Backend type: vxlan
1109 15:31:22.144839    1031 vxlan.go:119] VXLAN config: VNI=1 Port=0 GBP=false DirectRouting=false
1109 15:31:22.224149    1031 local_manager.go:234] Picking subnet in range 10.2.1.0 ... 10.2.255.0
1109 15:31:22.227551    1031 local_manager.go:220] Allocated lease (10.2.108.0/24) to current node (172.31.9.74)
1109 15:31:22.248275    1031 main.go:295] Wrote subnet file to /run/flannel/subnet.env
1109 15:31:22.248290    1031 main.go:299] Running backend.
nel - Network fabric for containers (System Application Container).
1109 15:31:22.254644    1031 ipmasq.go:75] Some iptables rules are missing; deleting and recreating rules
1109 15:31:22.254670    1031 ipmasq.go:97] Deleting iptables rule: -s 10.2.0.0/16 -d 10.2.0.0/16 -j RETURN
1109 15:31:22.255818    1031 ipmasq.go:97] Deleting iptables rule: -s 10.2.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE
1109 15:31:22.257049    1031 ipmasq.go:97] Deleting iptables rule: ! -s 10.2.0.0/16 -d 10.2.108.0/24 -j RETURN
1109 15:31:22.263636    1031 ipmasq.go:97] Deleting iptables rule: ! -s 10.2.0.0/16 -d 10.2.0.0/16 -j MASQUERADE
1109 15:31:22.264832    1031 ipmasq.go:85] Adding iptables rule: -s 10.2.0.0/16 -d 10.2.0.0/16 -j RETURN
1109 15:31:22.269451    1031 ipmasq.go:85] Adding iptables rule: -s 10.2.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE
1109 15:31:22.284971    1031 vxlan_network.go:56] watching for new subnet leases
1109 15:31:22.296313    1031 main.go:391] Waiting for 22h59m59.927033448s to renew lease
1109 15:31:22.296397    1031 ipmasq.go:85] Adding iptables rule: ! -s 10.2.0.0/16 -d 10.2.108.0/24 -j RETURN
1109 15:31:22.298935    1031 ipmasq.go:85] Adding iptables rule: ! -s 10.2.0.0/16 -d 10.2.0.0/16 -j MASQUERADE

And the interfaces on this host:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 06:2f:ce:c7:aa:3a brd ff:ff:ff:ff:ff:ff
    inet 172.31.9.74/21 brd 172.31.15.255 scope global dynamic eth0
       valid_lft 1883sec preferred_lft 1883sec
    inet6 fe80::42f:ceff:fec7:aa3a/64 scope link 
       valid_lft forever preferred_lft forever
3: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default 
    link/ether ca:d3:ce:dd:25:31 brd ff:ff:ff:ff:ff:ff
    inet 10.2.108.0/32 scope global flannel.1
       valid_lft forever preferred_lft forever
    inet6 fe80::c8d3:ceff:fedd:2531/64 scope link 
       valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:95:95:86:5d brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 scope global docker0
       valid_lft forever preferred_lft forever
5: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 0a:58:0a:02:6c:01 brd ff:ff:ff:ff:ff:ff
    inet 10.2.108.1/24 scope global cni0
       valid_lft forever preferred_lft forever
    inet6 fe80::60cf:e9ff:fed0:aab5/64 scope link 
       valid_lft forever preferred_lft forever

$ ip -d link show dev flannel.1
3: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default 
    link/ether ca:d3:ce:dd:25:31 brd ff:ff:ff:ff:ff:ff promiscuity 0 
    vxlan id 1 local 172.31.9.74 dev eth0 srcport 0 0 dstport 8472 nolearning ageing 300 udpcsum noudp6zerocsumtx noudp6zerocsumrx addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535

tomdee · 2017-12-07T19:36:51Z

I'm able to repro this on AWS now:

core@ip-172-31-12-123 ~ $ ip -d link show flannel.1
6: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default 
    link/ether 8e:23:05:4d:c8:af brd ff:ff:ff:ff:ff:ff promiscuity 0 
    vxlan id 1 local 172.31.12.123 dev eth0 srcport 0 0 dstport 8472 nolearning ageing 300 udpcsum noudp6zerocsumtx noudp6zerocsumrx addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 
core@ip-172-31-12-123 ~ $ ip -d link show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 0a:1b:aa:f3:64:00 brd ff:ff:ff:ff:ff:ff promiscuity 0 addrgenmode eui64 numtxqueues 2 numrxqueues 2 gso_max_size 65536 gso_max_segs 65535 
core@ip-172-31-12-123 ~ $

sgarg1 changed the title ~~Flannel AWS setting wrong MTU~~ Flannel setting wrong MTU on AWS Oct 17, 2017

tomdee added components/backend/aws components/backend/vxlan labels Oct 20, 2017

tomdee closed this as completed Oct 20, 2017

tomdee reopened this Nov 10, 2017

tomdee mentioned this issue Dec 7, 2017

backend/vxlan: Return correct MTU value #897

Merged

gunjan5 closed this as completed in #897 Dec 11, 2017

andyjeffries mentioned this issue Aug 28, 2019

curl command is not working in k3s container k3s-io/k3s#763

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Flannel setting wrong MTU on AWS #841

Flannel setting wrong MTU on AWS #841

sgarg1 commented Oct 17, 2017 •

edited

Loading

sgarg1 commented Oct 18, 2017

tomdee commented Oct 20, 2017

tomdee commented Oct 20, 2017 •

edited

Loading

mikkeloscar commented Nov 8, 2017 •

edited

Loading

tomdee commented Nov 9, 2017

mikkeloscar commented Nov 9, 2017

tomdee commented Nov 10, 2017

mikkeloscar commented Nov 10, 2017

tomdee commented Dec 7, 2017

Flannel setting wrong MTU on AWS #841

Flannel setting wrong MTU on AWS #841

Comments

sgarg1 commented Oct 17, 2017 • edited Loading

Expected Behavior

Current Behavior

Possible Solution

Steps to Reproduce (for bugs)

Your Environment

systemctl cat flanneld

ifconfig

cat /var/run/flannel/flannel_docker_opts.env

cat /var/run/flannel/subnet.env

etcd values

sgarg1 commented Oct 18, 2017

tomdee commented Oct 20, 2017

tomdee commented Oct 20, 2017 • edited Loading

mikkeloscar commented Nov 8, 2017 • edited Loading

tomdee commented Nov 9, 2017

mikkeloscar commented Nov 9, 2017

tomdee commented Nov 10, 2017

mikkeloscar commented Nov 10, 2017

tomdee commented Dec 7, 2017

sgarg1 commented Oct 17, 2017 •

edited

Loading

tomdee commented Oct 20, 2017 •

edited

Loading

mikkeloscar commented Nov 8, 2017 •

edited

Loading