Skip to content
This repository has been archived by the owner. It is now read-only.

/ bugs Archived

No IP masquerade for traffic destined for outside the flannel network #1439

Closed
h0tbird opened this issue Jul 5, 2016 · 1 comment
Closed

No IP masquerade for traffic destined for outside the flannel network #1439

h0tbird opened this issue Jul 5, 2016 · 1 comment
Labels
area/usability component/flannel kind/regression priority/P0 team/os
Milestone
CoreOS 1109.1.0

Comments

@h0tbird
Copy link

@h0tbird h0tbird commented Jul 5, 2016 •
edited

Issue Report

Bug

Outside flannel communications blocked after updating to 1097.0.0 on Alpha channel.
Related to: #1302

CoreOS Version

core@edge-1 ~ $ cat /etc/os-release 
NAME=CoreOS
ID=coreos
VERSION=1097.0.0
VERSION_ID=1097.0.0
BUILD_ID=2016-07-02-0145
PRETTY_NAME="CoreOS 1097.0.0 (MoreOS)"
ANSI_COLOR="1;32"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://github.com/coreos/bugs/issues"

Environment

Running on EC2:

core@edge-1 ~ $ etcdctl get /coreos.com/network/config
{ "Network": "10.136.128.0/18","SubnetLen":26 ,"SubnetMin": "10.136.128.0","SubnetMax": "10.136.191.192","Backend": {"Type": "vxlan"} }

core@edge-1 ~ $ systemctl cat flanneld
# /usr/lib64/systemd/system/flanneld.service
[Unit]
Description=Network fabric for containers
Documentation=https://github.com/coreos/flannel
After=etcd.service etcd2.service
Before=docker.service

[Service]
Type=notify
Restart=always
RestartSec=5
Environment="TMPDIR=/var/tmp/"
Environment="FLANNEL_VER=0.5.5"
Environment="FLANNEL_IMG=quay.io/coreos/flannel"
Environment="ETCD_SSL_DIR=/etc/ssl/etcd"
EnvironmentFile=-/run/flannel/options.env
LimitNOFILE=40000
LimitNPROC=1048576
ExecStartPre=/sbin/modprobe ip_tables
ExecStartPre=/usr/bin/mkdir -p /run/flannel
ExecStartPre=/usr/bin/mkdir -p ${ETCD_SSL_DIR}

ExecStart=/usr/bin/rkt run --net=host \
   --stage1-path=/usr/lib/rkt/stage1-images/stage1-fly.aci \
   --insecure-options=image \
   --set-env=NOTIFY_SOCKET=/run/systemd/notify \
   --inherit-env=true \
   --volume runsystemd,kind=host,source=/run/systemd,readOnly=false \
   --volume runflannel,kind=host,source=/run/flannel,readOnly=false \
   --volume ssl,kind=host,source=${ETCD_SSL_DIR},readOnly=true \
   --mount volume=runsystemd,target=/run/systemd \
   --mount volume=runflannel,target=/run/flannel \
   --mount volume=ssl,target=${ETCD_SSL_DIR} \
   ${FLANNEL_IMG}:${FLANNEL_VER} \
   -- --ip-masq=true

# Update docker options
ExecStartPost=/usr/bin/rkt run --net=host \
   --stage1-path=/usr/lib/rkt/stage1-images/stage1-fly.aci \
   --insecure-options=image \
   --volume runvol,kind=host,source=/run,readOnly=false \
   --mount volume=runvol,target=/run \
   ${FLANNEL_IMG}:${FLANNEL_VER} \
   --exec /opt/bin/mk-docker-opts.sh -- -d /run/flannel_docker_opts.env -i

ExecStopPost=/usr/bin/rkt gc --mark-only

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/flanneld.service.d/50-network-config.conf
[Service]
ExecStartPre=/usr/bin/etcdctl set /coreos.com/network/config '{ "Network": "10.136.128.0/18","SubnetLen":26 ,"SubnetMin": "10.136.128.0","SubnetMax": "10.136.191.192","Backend": {"Type": "vxlan"} }'

Expected Behavior

IP masquerade for traffic destined for outside the flannel network.

Actual Behavior

No IP masqerade for traffic destined for outside the flannel network.

Reproduction Steps

  1. Setup flannel with --ip-masq=true
  2. Ping to/from outside.

Other Information

core@edge-1 ~ $ loopssh cat /run/flannel/subnet.env
Warning: Permanently added '10.136.77.70' (ECDSA) to the list of known hosts.
FLANNEL_NETWORK=10.136.128.0/18
FLANNEL_SUBNET=10.136.137.193/26
FLANNEL_MTU=8951
FLANNEL_IPMASQ=false
Warning: Permanently added '10.136.90.69' (ECDSA) to the list of known hosts.
FLANNEL_NETWORK=10.136.128.0/18
FLANNEL_SUBNET=10.136.145.1/26
FLANNEL_MTU=8951
FLANNEL_IPMASQ=false
Warning: Permanently added '10.136.117.15' (ECDSA) to the list of known hosts.
FLANNEL_NETWORK=10.136.128.0/18
FLANNEL_SUBNET=10.136.140.65/26
FLANNEL_MTU=8951
FLANNEL_IPMASQ=false
Warning: Permanently added '10.136.0.12' (ECDSA) to the list of known hosts.
FLANNEL_NETWORK=10.136.128.0/18
FLANNEL_SUBNET=10.136.129.1/26
FLANNEL_MTU=8951
FLANNEL_IPMASQ=false
Warning: Permanently added '10.136.0.13' (ECDSA) to the list of known hosts.
FLANNEL_NETWORK=10.136.128.0/18
FLANNEL_SUBNET=10.136.139.65/26
FLANNEL_MTU=8951
FLANNEL_IPMASQ=false
Warning: Permanently added '10.136.87.143' (ECDSA) to the list of known hosts.
FLANNEL_NETWORK=10.136.128.0/18
FLANNEL_SUBNET=10.136.149.129/26
FLANNEL_MTU=8951
FLANNEL_IPMASQ=false
Warning: Permanently added '10.136.0.11' (ECDSA) to the list of known hosts.
FLANNEL_NETWORK=10.136.128.0/18
FLANNEL_SUBNET=10.136.143.193/26
FLANNEL_MTU=8951
FLANNEL_IPMASQ=false

core@edge-1 ~ $ sudo iptables -nvL -t nat
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

core@edge-1 ~ $ loopssh cat /run/flannel_docker_opts.env
Warning: Permanently added '10.136.77.70' (ECDSA) to the list of known hosts.
DOCKER_OPT_BIP="--bip=10.136.137.193/26"
DOCKER_OPT_IPMASQ="--ip-masq=true"
DOCKER_OPT_MTU="--mtu=8951"
Warning: Permanently added '10.136.90.69' (ECDSA) to the list of known hosts.
DOCKER_OPT_BIP="--bip=10.136.145.1/26"
DOCKER_OPT_IPMASQ="--ip-masq=true"
DOCKER_OPT_MTU="--mtu=8951"
Warning: Permanently added '10.136.117.15' (ECDSA) to the list of known hosts.
DOCKER_OPT_BIP="--bip=10.136.140.65/26"
DOCKER_OPT_IPMASQ="--ip-masq=true"
DOCKER_OPT_MTU="--mtu=8951"
Warning: Permanently added '10.136.0.12' (ECDSA) to the list of known hosts.
DOCKER_OPT_BIP="--bip=10.136.129.1/26"
DOCKER_OPT_IPMASQ="--ip-masq=true"
DOCKER_OPT_MTU="--mtu=8951"
Warning: Permanently added '10.136.0.13' (ECDSA) to the list of known hosts.
DOCKER_OPT_BIP="--bip=10.136.139.65/26"
DOCKER_OPT_IPMASQ="--ip-masq=true"
DOCKER_OPT_MTU="--mtu=8951"
Warning: Permanently added '10.136.87.143' (ECDSA) to the list of known hosts.
DOCKER_OPT_BIP="--bip=10.136.149.129/26"
DOCKER_OPT_IPMASQ="--ip-masq=true"
DOCKER_OPT_MTU="--mtu=8951"
Warning: Permanently added '10.136.0.11' (ECDSA) to the list of known hosts.
DOCKER_OPT_BIP="--bip=10.136.143.193/26"
DOCKER_OPT_IPMASQ="--ip-masq=true"
DOCKER_OPT_MTU="--mtu=8951"

core@edge-1 ~ $ rkt cat-manifest 13acfb2b
{
  "acVersion": "1.9.1",
  "acKind": "PodManifest",
  "apps": [
    {
      "name": "flannel",
      "image": {
        "name": "quay.io/coreos/flannel",
        "id": "sha512-edc1edfc41281a9fe58697b2b29d9ffb6a4e23667e47f2bc5a61064500209815",
        "labels": [
          {
            "name": "version",
            "value": "0.5.5"
          },
          {
            "name": "arch",
            "value": "amd64"
          },
          {
            "name": "os",
            "value": "linux"
          }
        ]
      },
      "app": {
        "exec": [
          "/bin/sh",
          "-c",
          "/opt/bin/flanneld",
          "--ip-masq=true"
        ],
        "user": "root",
        "group": "root",
        "workingDirectory": "/",
        "environment": [
          {
            "name": "PATH",
            "value": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
          },
          {
            "name": "NOTIFY_SOCKET",
            "value": "/run/systemd/notify"
          },
          {
            "name": "TMPDIR",
            "value": "/var/tmp/"
          },
          {
            "name": "FLANNEL_VER",
            "value": "0.5.5"
          },
          {
            "name": "FLANNEL_IMG",
            "value": "quay.io/coreos/flannel"
          },
          {
            "name": "ETCD_SSL_DIR",
            "value": "/etc/ssl/etcd"
          },
          {
            "name": "FLANNELD_IFACE",
            "value": "10.136.77.70"
          }
        ]
      },
      "mounts": [
        {
          "volume": "runsystemd",
          "path": "/run/systemd"
        },
        {
          "volume": "runflannel",
          "path": "/run/flannel"
        },
        {
          "volume": "ssl",
          "path": "/etc/ssl/etcd"
        }
      ]
    }
  ],
  "volumes": [
    {
      "name": "runsystemd",
      "kind": "host",
      "source": "/run/systemd",
      "readOnly": false
    },
    {
      "name": "runflannel",
      "kind": "host",
      "source": "/run/flannel",
      "readOnly": false
    },
    {
      "name": "ssl",
      "kind": "host",
      "source": "/etc/ssl/etcd",
      "readOnly": true
    }
  ],
  "isolators": null,
  "annotations": null,
  "ports": []
}

core@edge-1 ~ $ journalctl -u flanneld
-- Logs begin at Tue 2016-07-05 12:22:19 UTC, end at Tue 2016-07-05 12:32:43 UTC. --
Jul 05 12:22:55 edge-1.cell-1.dub.xnood.com systemd[1]: Starting Network fabric for containers...
Jul 05 12:22:57 edge-1.cell-1.dub.xnood.com etcdctl[1064]: { "Network": "10.136.128.0/18","SubnetLen":26 ,"SubnetMin": "10.136.128.0","SubnetMax": "10.136.191.192","Backend": {"Type": "vxlan"} }
Jul 05 12:22:57 edge-1.cell-1.dub.xnood.com rkt[1081]: image: using image from file /usr/lib/rkt/stage1-images/stage1-fly.aci
Jul 05 12:22:59 edge-1.cell-1.dub.xnood.com rkt[1081]: image: searching for app image quay.io/coreos/flannel
Jul 05 12:23:00 edge-1.cell-1.dub.xnood.com rkt[1081]: image: remote fetching from URL "https://quay.io/c1/aci/quay.io/coreos/flannel/0.5.5/aci/linux/amd64/"
Jul 05 12:23:01 edge-1.cell-1.dub.xnood.com rkt[1081]: Downloading ACI:  0 B/8.86 MB
Jul 05 12:23:01 edge-1.cell-1.dub.xnood.com rkt[1081]: Downloading ACI:  16.4 KB/8.86 MB
Jul 05 12:23:02 edge-1.cell-1.dub.xnood.com rkt[1081]: Downloading ACI:  503 KB/8.86 MB
Jul 05 12:23:03 edge-1.cell-1.dub.xnood.com rkt[1081]: Downloading ACI:  1.65 MB/8.86 MB
Jul 05 12:23:04 edge-1.cell-1.dub.xnood.com rkt[1081]: Downloading ACI:  3.29 MB/8.86 MB
Jul 05 12:23:05 edge-1.cell-1.dub.xnood.com rkt[1081]: Downloading ACI:  5.4 MB/8.86 MB
Jul 05 12:23:06 edge-1.cell-1.dub.xnood.com rkt[1081]: Downloading ACI:  8.27 MB/8.86 MB
Jul 05 12:23:06 edge-1.cell-1.dub.xnood.com rkt[1081]: Downloading ACI:  8.86 MB/8.86 MB
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1081]: I0705 12:23:12.314834 01081 main.go:275] Installing signal handlers
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1081]: I0705 12:23:12.315332 01081 main.go:188] Using 10.136.77.70 as external interface
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1081]: I0705 12:23:12.315541 01081 main.go:189] Using 10.136.77.70 as external endpoint
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1081]: I0705 12:23:12.413956 01081 etcd.go:204] Picking subnet in range 10.136.128.0 ... 10.136.191.192
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1081]: I0705 12:23:12.418480 01081 etcd.go:84] Subnet lease acquired: 10.136.137.192/26
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1081]: I0705 12:23:12.464946 01081 vxlan.go:153] Watching for L3 misses
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1081]: I0705 12:23:12.464967 01081 vxlan.go:159] Watching for new subnet leases
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1081]: I0705 12:23:12.466727 01081 vxlan.go:273] Handling initial subnet events
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1081]: I0705 12:23:12.466739 01081 device.go:159] calling GetL2List() dev.link.Index: 3
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1081]: I0705 12:23:12.466814 01081 device.go:164] calling NeighAdd: 10.136.87.143, ce:f1:af:2c:58:ea
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1081]: I0705 12:23:12.466859 01081 device.go:164] calling NeighAdd: 10.136.0.12, e2:a8:1c:d3:d7:f9
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1081]: I0705 12:23:12.466899 01081 device.go:164] calling NeighAdd: 10.136.117.15, 52:47:12:6a:71:0e
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1081]: I0705 12:23:12.467254 01081 device.go:164] calling NeighAdd: 10.136.0.13, aa:34:69:8e:4f:79
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1081]: I0705 12:23:12.467302 01081 device.go:164] calling NeighAdd: 10.136.0.11, e2:ab:8e:82:fb:bc
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1081]: I0705 12:23:12.467333 01081 device.go:164] calling NeighAdd: 10.136.90.69, b2:7c:fa:dd:50:4b
Jul 05 12:23:12 edge-1.cell-1.dub.xnood.com rkt[1125]: image: using image from file /usr/lib/rkt/stage1-images/stage1-fly.aci
Jul 05 12:23:13 edge-1.cell-1.dub.xnood.com rkt[1125]: image: using image from local store for image name quay.io/coreos/flannel:0.5.5
Jul 05 12:23:14 edge-1.cell-1.dub.xnood.com systemd[1]: Started Network fabric for containers.
@crawford crawford added this to the CoreOS 1109.0.0 milestone Jul 5, 2016
@ajeddeloh ajeddeloh mentioned this issue Jul 5, 2016
Merged
@ajeddeloh
Copy link

@ajeddeloh ajeddeloh commented Jul 5, 2016

Looks like this is caused by /opt/bin/flanneld being run via /bin/sh -c and not getting passed the --ip-masq=true argument. Should be fixed here: coreos/coreos-overlay#2043
@ajeddeloh ajeddeloh mentioned this issue Jul 6, 2016
Closed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
area/usability component/flannel kind/regression priority/P0 team/os
Projects
None yet
Milestone
CoreOS 1109.1.0
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
@crawford @h0tbird @ajeddeloh
You can’t perform that action at this time.