New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No IP masquerade for traffic destined for outside the flannel network #1439

Closed
h0tbird opened this Issue Jul 5, 2016 · 1 comment

Comments

Projects
None yet
3 participants
@h0tbird

h0tbird commented Jul 5, 2016

Issue Report

Bug

Outside flannel communications blocked after updating to 1097.0.0 on Alpha channel.
Related to: #1302

CoreOS Version

core@edge-1 ~ $ cat /etc/os-release 
NAME=CoreOS
ID=coreos
VERSION=1097.0.0
VERSION_ID=1097.0.0
BUILD_ID=2016-07-02-0145
PRETTY_NAME="CoreOS 1097.0.0 (MoreOS)"
ANSI_COLOR="1;32"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://github.com/coreos/bugs/issues"

Environment

Running on EC2:

core@edge-1 ~ $ etcdctl get /coreos.com/network/config
{ "Network": "10.136.128.0/18","SubnetLen":26 ,"SubnetMin": "10.136.128.0","SubnetMax": "10.136.191.192","Backend": {"Type": "vxlan"} }
core@edge-1 ~ $ systemctl cat flanneld
# /usr/lib64/systemd/system/flanneld.service
[Unit]
Description=Network fabric for containers
Documentation=https://github.com/coreos/flannel
After=etcd.service etcd2.service
Before=docker.service

[Service]
Type=notify
Restart=always
RestartSec=5
Environment="TMPDIR=/var/tmp/"
Environment="FLANNEL_VER=0.5.5"
Environment="FLANNEL_IMG=quay.io/coreos/flannel"
Environment="ETCD_SSL_DIR=/etc/ssl/etcd"
EnvironmentFile=-/run/flannel/options.env
LimitNOFILE=40000
LimitNPROC=1048576
ExecStartPre=/sbin/modprobe ip_tables
ExecStartPre=/usr/bin/mkdir -p /run/flannel
ExecStartPre=/usr/bin/mkdir -p ${ETCD_SSL_DIR}

ExecStart=/usr/bin/rkt run --net=host \
   --stage1-path=/usr/lib/rkt/stage1-images/stage1-fly.aci \
   --insecure-options=image \
   --set-env=NOTIFY_SOCKET=/run/systemd/notify \
   --inherit-env=true \
   --volume runsystemd,kind=host,source=/run/systemd,readOnly=false \
   --volume runflannel,kind=host,source=/run/flannel,readOnly=false \
   --volume ssl,kind=host,source=${ETCD_SSL_DIR},readOnly=true \
   --mount volume=runsystemd,target=/run/systemd \
   --mount volume=runflannel,target=/run/flannel \
   --mount volume=ssl,target=${ETCD_SSL_DIR} \
   ${FLANNEL_IMG}:${FLANNEL_VER} \
   -- --ip-masq=true

# Update docker options
ExecStartPost=/usr/bin/rkt run --net=host \
   --stage1-path=/usr/lib/rkt/stage1-images/stage1-fly.aci \
   --insecure-options=image \
   --volume runvol,kind=host,source=/run,readOnly=false \
   --mount volume=runvol,target=/run \
   ${FLANNEL_IMG}:${FLANNEL_VER} \
   --exec /opt/bin/mk-docker-opts.sh -- -d /run/flannel_docker_opts.env -i

ExecStopPost=/usr/bin/rkt gc --mark-only

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/flanneld.service.d/50-network-config.conf
[Service]
ExecStartPre=/usr/bin/etcdctl set /coreos.com/network/config '{ "Network": "10.136.128.0/18","SubnetLen":26 ,"SubnetMin": "10.136.128.0","SubnetMax": "10.136.191.192","Backend": {"Type": "vxlan"} }'

Expected Behavior

IP masquerade for traffic destined for outside the flannel network.

Actual Behavior

No IP masqerade for traffic destined for outside the flannel network.

Reproduction Steps

  1. Setup flannel with --ip-masq=true
  2. Ping to/from outside.

Other Information

core@edge-1 ~ $ loopssh cat /run/flannel/subnet.env
Warning: Permanently added '10.136.77.70' (ECDSA) to the list of known hosts.
FLANNEL_NETWORK=10.136.128.0/18
FLANNEL_SUBNET=10.136.137.193/26
FLANNEL_MTU=8951
FLANNEL_IPMASQ=false
Warning: Permanently added '10.136.90.69' (ECDSA) to the list of known hosts.
FLANNEL_NETWORK=10.136.128.0/18
FLANNEL_SUBNET=10.136.145.1/26
FLANNEL_MTU=8951
FLANNEL_IPMASQ=false
Warning: Permanently added '10.136.117.15' (ECDSA) to the list of known hosts.
FLANNEL_NETWORK=10.136.128.0/18
FLANNEL_SUBNET=10.136.140.65/26
FLANNEL_MTU=8951
FLANNEL_IPMASQ=false
Warning: Permanently added '10.136.0.12' (ECDSA) to the list of known hosts.
FLANNEL_NETWORK=10.136.128.0/18
FLANNEL_SUBNET=10.136.129.1/26
FLANNEL_MTU=8951
FLANNEL_IPMASQ=false
Warning: Permanently added '10.136.0.13' (ECDSA) to the list of known hosts.
FLANNEL_NETWORK=10.136.128.0/18
FLANNEL_SUBNET=10.136.139.65/26
FLANNEL_MTU=8951
FLANNEL_IPMASQ=false
Warning: Permanently added '10.136.87.143' (ECDSA) to the list of known hosts.
FLANNEL_NETWORK=10.136.128.0/18
FLANNEL_SUBNET=10.136.149.129/26
FLANNEL_MTU=8951
FLANNEL_IPMASQ=false
Warning: Permanently added '10.136.0.11' (ECDSA) to the list of known hosts.
FLANNEL_NETWORK=10.136.128.0/18
FLANNEL_SUBNET=10.136.143.193/26
FLANNEL_MTU=8951
FLANNEL_IPMASQ=false
core@edge-1 ~ $ sudo iptables -nvL -t nat
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
core@edge-1 ~ $ loopssh cat /run/flannel_docker_opts.env
Warning: Permanently added '10.136.77.70' (ECDSA) to the list of known hosts.
DOCKER_OPT_BIP="--bip=10.136.137.193/26"
DOCKER_OPT_IPMASQ="--ip-masq=true"
DOCKER_OPT_MTU="--mtu=8951"
Warning: Permanently added '10.136.90.69' (ECDSA) to the list of known hosts.
DOCKER_OPT_BIP="--bip=10.136.145.1/26"
DOCKER_OPT_IPMASQ="--ip-masq=true"
DOCKER_OPT_MTU="--mtu=8951"
Warning: Permanently added '10.136.117.15' (ECDSA) to the list of known hosts.
DOCKER_OPT_BIP="--bip=10.136.140.65/26"
DOCKER_OPT_IPMASQ="--ip-masq=true"
DOCKER_OPT_MTU="--mtu=8951"
Warning: Permanently added '10.136.0.12' (ECDSA) to the list of known hosts.
DOCKER_OPT_BIP="--bip=10.136.129.1/26"
DOCKER_OPT_IPMASQ="--ip-masq=true"
DOCKER_OPT_MTU="--mtu=8951"
Warning: Permanently added '10.136.0.13' (ECDSA) to the list of known hosts.
DOCKER_OPT_BIP="--bip=10.136.139.65/26"
DOCKER_OPT_IPMASQ="--ip-masq=true"
DOCKER_OPT_MTU="--mtu=8951"
Warning: Permanently added '10.136.87.143' (ECDSA) to the list of known hosts.
DOCKER_OPT_BIP="--bip=10.136.149.129/26"
DOCKER_OPT_IPMASQ="--ip-masq=true"
DOCKER_OPT_MTU="--mtu=8951"
Warning: Permanently added '10.136.0.11' (ECDSA) to the list of known hosts.
DOCKER_OPT_BIP="--bip=10.136.143.193/26"
DOCKER_OPT_IPMASQ="--ip-masq=true"
DOCKER_OPT_MTU="--mtu=8951"
core@edge-1 ~ $ rkt cat-manifest 13acfb2b
{
  "acVersion": "1.9.1",
  "acKind": "PodManifest",
  "apps": [
    {
      "name": "flannel",
      "image": {
        "name": "quay.io/coreos/flannel",
        "id": "sha512-edc1edfc41281a9fe58697b2b29d9ffb6a4e23667e47f2bc5a61064500209815",
        "labels": [
          {
            "name": "version",
            "value": "0.5.5"
          },
          {
            "name": "arch",
            "value": "amd64"
          },
          {
            "name": "os",
            "value": "linux"
          }
        ]
      },
      "app": {
        "exec": [
          "/bin/sh",
          "-c",
          "/opt/bin/flanneld",
          "--ip-masq=true"
        ],
        "user": "root",
        "group": "root",
        "workingDirectory": "/",
        "environment": [
          {
            "name": "PATH",
            "value": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
          },
          {
            "name": "NOTIFY_SOCKET",
            "value": "/run/systemd/notify"
          },
          {
            "name": "TMPDIR",
            "value": "/var/tmp/"
          },
          {
            "name": "FLANNEL_VER",
            "value": "0.5.5"
          },
          {
            "name": "FLANNEL_IMG",
            "value": "quay.io/coreos/flannel"
          },
          {
            "name": "ETCD_SSL_DIR",
            "value": "/etc/ssl/etcd"
          },
          {
            "name": "FLANNELD_IFACE",
            "value": "10.136.77.70"
          }
        ]
      },
      "mounts": [
        {
          "volume": "runsystemd",
          "path": "/run/systemd"
        },
        {
          "volume": "runflannel",
          "path": "/run/flannel"
        },
        {
          "volume": "ssl",
          "path": "/etc/ssl/etcd"
        }
      ]
    }
  ],
  "volumes": [
    {
      "name": "runsystemd",
      "kind": "host",
      "source": "/run/systemd",
      "readOnly": false
    },
    {
      "name": "runflannel",
      "kind": "host",
      "source": "/run/flannel",
      "readOnly": false
    },
    {
      "name": "ssl",
      "kind": "host",
      "source": "/etc/ssl/etcd",
      "readOnly": true
    }
  ],
  "isolators": null,
  "annotations": null,
  "ports": []
}
core@edge-1 ~ $ journalctl -u flanneld
-- Logs begin at Tue 2016-07-05 12:22:19 UTC, end at Tue 2016-07-05 12:32:43 UTC. --
Jul 05 12:22:55 edge-1.cell-1.dub.xnood.com systemd[1]: Starting Network fabric for containers...
Jul 05 12:22:57 edge-1.cell-1.dub.xnood.com etcdctl[1064]: { "Network": "10.136.128.0/18","SubnetLen":26 ,"SubnetMin": "10.136.128.0","SubnetMax": "10.136.191.192","Backend": {"Type": "vxlan"} }
Jul 05 12:22:57 edge-1.cell-1.dub.xnood.com rkt[1081]: image: using image from file /usr/lib/rkt/stage1-images/stage1-fly.aci
Jul 05 12:22:59 edge-1.cell-1.dub.xnood.com rkt[1081]: image: searching for app image quay.io/coreos/flannel
Jul 05 12:23:00 edge-1.cell-1.dub.xnood.com rkt[1081]: image: remote fetching from URL "https://quay.io/c1/aci/quay.io/coreos/flannel/0.5.5/aci/linux/amd64/"
Jul 05 12:23:01 edge-1.cell-1.dub.xnood.com rkt[1081]: Downloading ACI:  0 B/8.86 MB
Jul 05 12:23:01 edge-1.cell-1.dub.xnood.com rkt[1081]: Downloading ACI:  16.4 KB/8.86 MB
Jul 05 12:23:02 edge-1.cell-1.dub.xnood.com rkt[1081]: Downloading ACI:  503 KB/8.86 MB
Jul 05 12:23:03 edge-1.cell-1.dub.xnood.com rkt[1081]: Downloading ACI:  1.65 MB/8.86 MB
Jul 05 12:23:04 edge-1.cell-1.dub.xnood.com rkt[1081]: Downloading ACI:  3.29 MB/8.86 MB
Jul 05 12:23:05 edge-1.cell-1.dub.xnood.com rkt[1081]: Downloading ACI:  5.4 MB/8.86 MB
Jul 05 12:23:06 edge-1.cell-1.dub.xnood.com rkt[1081]: Downloading ACI:  8.27 MB/8.86 MB
Jul 05 12:23:06 edge-1.cell-1.dub.xnood.com rkt[1081]: Downloading ACI:  8.86 MB/8.86 MB
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1081]: I0705 12:23:12.314834 01081 main.go:275] Installing signal handlers
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1081]: I0705 12:23:12.315332 01081 main.go:188] Using 10.136.77.70 as external interface
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1081]: I0705 12:23:12.315541 01081 main.go:189] Using 10.136.77.70 as external endpoint
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1081]: I0705 12:23:12.413956 01081 etcd.go:204] Picking subnet in range 10.136.128.0 ... 10.136.191.192
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1081]: I0705 12:23:12.418480 01081 etcd.go:84] Subnet lease acquired: 10.136.137.192/26
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1081]: I0705 12:23:12.464946 01081 vxlan.go:153] Watching for L3 misses
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1081]: I0705 12:23:12.464967 01081 vxlan.go:159] Watching for new subnet leases
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1081]: I0705 12:23:12.466727 01081 vxlan.go:273] Handling initial subnet events
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1081]: I0705 12:23:12.466739 01081 device.go:159] calling GetL2List() dev.link.Index: 3
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1081]: I0705 12:23:12.466814 01081 device.go:164] calling NeighAdd: 10.136.87.143, ce:f1:af:2c:58:ea
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1081]: I0705 12:23:12.466859 01081 device.go:164] calling NeighAdd: 10.136.0.12, e2:a8:1c:d3:d7:f9
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1081]: I0705 12:23:12.466899 01081 device.go:164] calling NeighAdd: 10.136.117.15, 52:47:12:6a:71:0e
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1081]: I0705 12:23:12.467254 01081 device.go:164] calling NeighAdd: 10.136.0.13, aa:34:69:8e:4f:79
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1081]: I0705 12:23:12.467302 01081 device.go:164] calling NeighAdd: 10.136.0.11, e2:ab:8e:82:fb:bc
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1081]: I0705 12:23:12.467333 01081 device.go:164] calling NeighAdd: 10.136.90.69, b2:7c:fa:dd:50:4b
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1125]: image: using image from file /usr/lib/rkt/stage1-images/stage1-fly.aci
Jul 05 12:23:13 edge-1.cell-1.dub.xnood.com rkt[1125]: image: using image from local store for image name quay.io/coreos/flannel:0.5.5
Jul 05 12:23:14 edge-1.cell-1.dub.xnood.com systemd[1]: Started Network fabric for containers.
@ajeddeloh

This comment has been minimized.

ajeddeloh commented Jul 5, 2016

Looks like this is caused by /opt/bin/flanneld being run via /bin/sh -c and not getting passed the --ip-masq=true argument. Should be fixed here: coreos/coreos-overlay#2043

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment