No IP masquerade for traffic destined for outside the flannel network #1439

h0tbird · Jul 5, 2016

Issue Report

Bug

Outside flannel communications blocked after updating to 1097.0.0 on Alpha channel.
Related to: #1302

CoreOS Version

core@edge-1 ~ $ cat /etc/os-release 
NAME=CoreOS
ID=coreos
VERSION=1097.0.0
VERSION_ID=1097.0.0
BUILD_ID=2016-07-02-0145
PRETTY_NAME="CoreOS 1097.0.0 (MoreOS)"
ANSI_COLOR="1;32"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://github.com/coreos/bugs/issues"

Environment

Running on EC2:

core@edge-1 ~ $ etcdctl get /coreos.com/network/config
{ "Network": "10.136.128.0/18","SubnetLen":26 ,"SubnetMin": "10.136.128.0","SubnetMax": "10.136.191.192","Backend": {"Type": "vxlan"} }

core@edge-1 ~ $ systemctl cat flanneld
# /usr/lib64/systemd/system/flanneld.service
[Unit]
Description=Network fabric for containers
Documentation=https://github.com/coreos/flannel
After=etcd.service etcd2.service
Before=docker.service

[Service]
Type=notify
Restart=always
RestartSec=5
Environment="TMPDIR=/var/tmp/"
Environment="FLANNEL_VER=0.5.5"
Environment="FLANNEL_IMG=quay.io/coreos/flannel"
Environment="ETCD_SSL_DIR=/etc/ssl/etcd"
EnvironmentFile=-/run/flannel/options.env
LimitNOFILE=40000
LimitNPROC=1048576
ExecStartPre=/sbin/modprobe ip_tables
ExecStartPre=/usr/bin/mkdir -p /run/flannel
ExecStartPre=/usr/bin/mkdir -p ${ETCD_SSL_DIR}

ExecStart=/usr/bin/rkt run --net=host \
   --stage1-path=/usr/lib/rkt/stage1-images/stage1-fly.aci \
   --insecure-options=image \
   --set-env=NOTIFY_SOCKET=/run/systemd/notify \
   --inherit-env=true \
   --volume runsystemd,kind=host,source=/run/systemd,readOnly=false \
   --volume runflannel,kind=host,source=/run/flannel,readOnly=false \
   --volume ssl,kind=host,source=${ETCD_SSL_DIR},readOnly=true \
   --mount volume=runsystemd,target=/run/systemd \
   --mount volume=runflannel,target=/run/flannel \
   --mount volume=ssl,target=${ETCD_SSL_DIR} \
   ${FLANNEL_IMG}:${FLANNEL_VER} \
   -- --ip-masq=true

# Update docker options
ExecStartPost=/usr/bin/rkt run --net=host \
   --stage1-path=/usr/lib/rkt/stage1-images/stage1-fly.aci \
   --insecure-options=image \
   --volume runvol,kind=host,source=/run,readOnly=false \
   --mount volume=runvol,target=/run \
   ${FLANNEL_IMG}:${FLANNEL_VER} \
   --exec /opt/bin/mk-docker-opts.sh -- -d /run/flannel_docker_opts.env -i

ExecStopPost=/usr/bin/rkt gc --mark-only

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/flanneld.service.d/50-network-config.conf
[Service]
ExecStartPre=/usr/bin/etcdctl set /coreos.com/network/config '{ "Network": "10.136.128.0/18","SubnetLen":26 ,"SubnetMin": "10.136.128.0","SubnetMax": "10.136.191.192","Backend": {"Type": "vxlan"} }'

Expected Behavior

IP masquerade for traffic destined for outside the flannel network.

Actual Behavior

No IP masqerade for traffic destined for outside the flannel network.

Reproduction Steps

Setup flannel with --ip-masq=true
Ping to/from outside.

Other Information

core@edge-1 ~ $ loopssh cat /run/flannel/subnet.env
Warning: Permanently added '10.136.77.70' (ECDSA) to the list of known hosts.
FLANNEL_NETWORK=10.136.128.0/18
FLANNEL_SUBNET=10.136.137.193/26
FLANNEL_MTU=8951
FLANNEL_IPMASQ=false
Warning: Permanently added '10.136.90.69' (ECDSA) to the list of known hosts.
FLANNEL_NETWORK=10.136.128.0/18
FLANNEL_SUBNET=10.136.145.1/26
FLANNEL_MTU=8951
FLANNEL_IPMASQ=false
Warning: Permanently added '10.136.117.15' (ECDSA) to the list of known hosts.
FLANNEL_NETWORK=10.136.128.0/18
FLANNEL_SUBNET=10.136.140.65/26
FLANNEL_MTU=8951
FLANNEL_IPMASQ=false
Warning: Permanently added '10.136.0.12' (ECDSA) to the list of known hosts.
FLANNEL_NETWORK=10.136.128.0/18
FLANNEL_SUBNET=10.136.129.1/26
FLANNEL_MTU=8951
FLANNEL_IPMASQ=false
Warning: Permanently added '10.136.0.13' (ECDSA) to the list of known hosts.
FLANNEL_NETWORK=10.136.128.0/18
FLANNEL_SUBNET=10.136.139.65/26
FLANNEL_MTU=8951
FLANNEL_IPMASQ=false
Warning: Permanently added '10.136.87.143' (ECDSA) to the list of known hosts.
FLANNEL_NETWORK=10.136.128.0/18
FLANNEL_SUBNET=10.136.149.129/26
FLANNEL_MTU=8951
FLANNEL_IPMASQ=false
Warning: Permanently added '10.136.0.11' (ECDSA) to the list of known hosts.
FLANNEL_NETWORK=10.136.128.0/18
FLANNEL_SUBNET=10.136.143.193/26
FLANNEL_MTU=8951
FLANNEL_IPMASQ=false

core@edge-1 ~ $ sudo iptables -nvL -t nat
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

core@edge-1 ~ $ loopssh cat /run/flannel_docker_opts.env
Warning: Permanently added '10.136.77.70' (ECDSA) to the list of known hosts.
DOCKER_OPT_BIP="--bip=10.136.137.193/26"
DOCKER_OPT_IPMASQ="--ip-masq=true"
DOCKER_OPT_MTU="--mtu=8951"
Warning: Permanently added '10.136.90.69' (ECDSA) to the list of known hosts.
DOCKER_OPT_BIP="--bip=10.136.145.1/26"
DOCKER_OPT_IPMASQ="--ip-masq=true"
DOCKER_OPT_MTU="--mtu=8951"
Warning: Permanently added '10.136.117.15' (ECDSA) to the list of known hosts.
DOCKER_OPT_BIP="--bip=10.136.140.65/26"
DOCKER_OPT_IPMASQ="--ip-masq=true"
DOCKER_OPT_MTU="--mtu=8951"
Warning: Permanently added '10.136.0.12' (ECDSA) to the list of known hosts.
DOCKER_OPT_BIP="--bip=10.136.129.1/26"
DOCKER_OPT_IPMASQ="--ip-masq=true"
DOCKER_OPT_MTU="--mtu=8951"
Warning: Permanently added '10.136.0.13' (ECDSA) to the list of known hosts.
DOCKER_OPT_BIP="--bip=10.136.139.65/26"
DOCKER_OPT_IPMASQ="--ip-masq=true"
DOCKER_OPT_MTU="--mtu=8951"
Warning: Permanently added '10.136.87.143' (ECDSA) to the list of known hosts.
DOCKER_OPT_BIP="--bip=10.136.149.129/26"
DOCKER_OPT_IPMASQ="--ip-masq=true"
DOCKER_OPT_MTU="--mtu=8951"
Warning: Permanently added '10.136.0.11' (ECDSA) to the list of known hosts.
DOCKER_OPT_BIP="--bip=10.136.143.193/26"
DOCKER_OPT_IPMASQ="--ip-masq=true"
DOCKER_OPT_MTU="--mtu=8951"

core@edge-1 ~ $ rkt cat-manifest 13acfb2b
{
  "acVersion": "1.9.1",
  "acKind": "PodManifest",
  "apps": [
    {
      "name": "flannel",
      "image": {
        "name": "quay.io/coreos/flannel",
        "id": "sha512-edc1edfc41281a9fe58697b2b29d9ffb6a4e23667e47f2bc5a61064500209815",
        "labels": [
          {
            "name": "version",
            "value": "0.5.5"
          },
          {
            "name": "arch",
            "value": "amd64"
          },
          {
            "name": "os",
            "value": "linux"
          }
        ]
      },
      "app": {
        "exec": [
          "/bin/sh",
          "-c",
          "/opt/bin/flanneld",
          "--ip-masq=true"
        ],
        "user": "root",
        "group": "root",
        "workingDirectory": "/",
        "environment": [
          {
            "name": "PATH",
            "value": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
          },
          {
            "name": "NOTIFY_SOCKET",
            "value": "/run/systemd/notify"
          },
          {
            "name": "TMPDIR",
            "value": "/var/tmp/"
          },
          {
            "name": "FLANNEL_VER",
            "value": "0.5.5"
          },
          {
            "name": "FLANNEL_IMG",
            "value": "quay.io/coreos/flannel"
          },
          {
            "name": "ETCD_SSL_DIR",
            "value": "/etc/ssl/etcd"
          },
          {
            "name": "FLANNELD_IFACE",
            "value": "10.136.77.70"
          }
        ]
      },
      "mounts": [
        {
          "volume": "runsystemd",
          "path": "/run/systemd"
        },
        {
          "volume": "runflannel",
          "path": "/run/flannel"
        },
        {
          "volume": "ssl",
          "path": "/etc/ssl/etcd"
        }
      ]
    }
  ],
  "volumes": [
    {
      "name": "runsystemd",
      "kind": "host",
      "source": "/run/systemd",
      "readOnly": false
    },
    {
      "name": "runflannel",
      "kind": "host",
      "source": "/run/flannel",
      "readOnly": false
    },
    {
      "name": "ssl",
      "kind": "host",
      "source": "/etc/ssl/etcd",
      "readOnly": true
    }
  ],
  "isolators": null,
  "annotations": null,
  "ports": []
}

core@edge-1 ~ $ journalctl -u flanneld
-- Logs begin at Tue 2016-07-05 12:22:19 UTC, end at Tue 2016-07-05 12:32:43 UTC. --
Jul 05 12:22:55 edge-1.cell-1.dub.xnood.com systemd[1]: Starting Network fabric for containers...
Jul 05 12:22:57 edge-1.cell-1.dub.xnood.com etcdctl[1064]: { "Network": "10.136.128.0/18","SubnetLen":26 ,"SubnetMin": "10.136.128.0","SubnetMax": "10.136.191.192","Backend": {"Type": "vxlan"} }
Jul 05 12:22:57 edge-1.cell-1.dub.xnood.com rkt[1081]: image: using image from file /usr/lib/rkt/stage1-images/stage1-fly.aci
Jul 05 12:22:59 edge-1.cell-1.dub.xnood.com rkt[1081]: image: searching for app image quay.io/coreos/flannel
Jul 05 12:23:00 edge-1.cell-1.dub.xnood.com rkt[1081]: image: remote fetching from URL "https://quay.io/c1/aci/quay.io/coreos/flannel/0.5.5/aci/linux/amd64/"
Jul 05 12:23:01 edge-1.cell-1.dub.xnood.com rkt[1081]: Downloading ACI:  0 B/8.86 MB
Jul 05 12:23:01 edge-1.cell-1.dub.xnood.com rkt[1081]: Downloading ACI:  16.4 KB/8.86 MB
Jul 05 12:23:02 edge-1.cell-1.dub.xnood.com rkt[1081]: Downloading ACI:  503 KB/8.86 MB
Jul 05 12:23:03 edge-1.cell-1.dub.xnood.com rkt[1081]: Downloading ACI:  1.65 MB/8.86 MB
Jul 05 12:23:04 edge-1.cell-1.dub.xnood.com rkt[1081]: Downloading ACI:  3.29 MB/8.86 MB
Jul 05 12:23:05 edge-1.cell-1.dub.xnood.com rkt[1081]: Downloading ACI:  5.4 MB/8.86 MB
Jul 05 12:23:06 edge-1.cell-1.dub.xnood.com rkt[1081]: Downloading ACI:  8.27 MB/8.86 MB
Jul 05 12:23:06 edge-1.cell-1.dub.xnood.com rkt[1081]: Downloading ACI:  8.86 MB/8.86 MB
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1081]: I0705 12:23:12.314834 01081 main.go:275] Installing signal handlers
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1081]: I0705 12:23:12.315332 01081 main.go:188] Using 10.136.77.70 as external interface
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1081]: I0705 12:23:12.315541 01081 main.go:189] Using 10.136.77.70 as external endpoint
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1081]: I0705 12:23:12.413956 01081 etcd.go:204] Picking subnet in range 10.136.128.0 ... 10.136.191.192
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1081]: I0705 12:23:12.418480 01081 etcd.go:84] Subnet lease acquired: 10.136.137.192/26
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1081]: I0705 12:23:12.464946 01081 vxlan.go:153] Watching for L3 misses
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1081]: I0705 12:23:12.464967 01081 vxlan.go:159] Watching for new subnet leases
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1081]: I0705 12:23:12.466727 01081 vxlan.go:273] Handling initial subnet events
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1081]: I0705 12:23:12.466739 01081 device.go:159] calling GetL2List() dev.link.Index: 3
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1081]: I0705 12:23:12.466814 01081 device.go:164] calling NeighAdd: 10.136.87.143, ce:f1:af:2c:58:ea
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1081]: I0705 12:23:12.466859 01081 device.go:164] calling NeighAdd: 10.136.0.12, e2:a8:1c:d3:d7:f9
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1081]: I0705 12:23:12.466899 01081 device.go:164] calling NeighAdd: 10.136.117.15, 52:47:12:6a:71:0e
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1081]: I0705 12:23:12.467254 01081 device.go:164] calling NeighAdd: 10.136.0.13, aa:34:69:8e:4f:79
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1081]: I0705 12:23:12.467302 01081 device.go:164] calling NeighAdd: 10.136.0.11, e2:ab:8e:82:fb:bc
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1081]: I0705 12:23:12.467333 01081 device.go:164] calling NeighAdd: 10.136.90.69, b2:7c:fa:dd:50:4b
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1125]: image: using image from file /usr/lib/rkt/stage1-images/stage1-fly.aci
Jul 05 12:23:13 edge-1.cell-1.dub.xnood.com rkt[1125]: image: using image from local store for image name quay.io/coreos/flannel:0.5.5
Jul 05 12:23:14 edge-1.cell-1.dub.xnood.com systemd[1]: Started Network fabric for containers.

ajeddeloh · Jul 5, 2016

Looks like this is caused by /opt/bin/flanneld being run via /bin/sh -c and not getting passed the --ip-masq=true argument. Should be fixed here: coreos/coreos-overlay#2043

crawford added kind/regression component/flannel priority/P0 area/usability team/os labels Jul 5, 2016

crawford added this to the CoreOS 1109.0.0 milestone Jul 5, 2016

ajeddeloh referenced this issue Jul 5, 2016
Merged
Fix flannel issues #2043

ajeddeloh referenced this issue Jul 6, 2016
Closed
flanneld unable to talk to etcd following a restart #1445

ajeddeloh closed this in coreos/coreos-overlay#2043 Jul 6, 2016

coreos/bugs

No IP masquerade for traffic destined for outside the flannel network #1439

h0tbird commented Jul 5, 2016 •

edited

crawford added kind/regression component/flannel priority/P0 area/usability team/os labels Jul 5, 2016

crawford added this to the CoreOS 1109.0.0 milestone Jul 5, 2016

ajeddeloh referenced this issue Jul 5, 2016

Fix flannel issues #2043

This comment has been minimized.

ajeddeloh commented Jul 5, 2016

ajeddeloh referenced this issue Jul 6, 2016

flanneld unable to talk to etcd following a restart #1445

ajeddeloh closed this in coreos/coreos-overlay#2043 Jul 6, 2016

coreos/bugs

Join GitHub today

No IP masquerade for traffic destined for outside the flannel network #1439

Comments

h0tbird commented Jul 5, 2016 • edited

Issue Report

Bug

CoreOS Version

Environment

Expected Behavior

Actual Behavior

Reproduction Steps

Other Information

crawford added kind/regression component/flannel priority/P0 area/usability team/os labels Jul 5, 2016

crawford added this to the CoreOS 1109.0.0 milestone Jul 5, 2016

ajeddeloh referenced this issue Jul 5, 2016

Fix flannel issues #2043

This comment has been minimized.

ajeddeloh commented Jul 5, 2016

ajeddeloh referenced this issue Jul 6, 2016

flanneld unable to talk to etcd following a restart #1445

ajeddeloh closed this in coreos/coreos-overlay#2043 Jul 6, 2016

h0tbird commented Jul 5, 2016 •

edited