New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docker0 VLANs failed: Operation not permitted after updateing to 1185.3.0 stable #1642

Closed
pizzarabe opened this Issue Nov 2, 2016 · 5 comments

Comments

Projects
None yet
3 participants
@pizzarabe

pizzarabe commented Nov 2, 2016

Issue Report

After (auto-updating) to the latest stable 1185.3.0 I get the following error msg:

Nov 02 10:57:58 alien1 systemd-networkd[1474]: docker0: Could not append VLANs: Operation not permitted
Nov 02 10:57:58 alien1 systemd-networkd[1474]: docker0: Failed to assign VLANs to bridge port: Operation not permitted
Nov 02 10:57:58 alien1 systemd-networkd[1474]: docker0: Could not set bridge vlan: Operation not permitted

docker is running with flannel

$ systemctl cat flanneld | cat 
# /usr/lib/systemd/system/flanneld.service
[Unit]
Description=Network fabric for containers
Documentation=https://github.com/coreos/flannel
After=etcd.service etcd2.service
Before=docker.service

[Service]
Type=notify
Restart=always
RestartSec=5
Environment="TMPDIR=/var/tmp/"
Environment="FLANNEL_VER=v0.6.2-amd64"
Environment="FLANNEL_IMG=quay.io/coreos/flannel"
Environment="ETCD_SSL_DIR=/etc/ssl/etcd"
EnvironmentFile=-/run/flannel/options.env
LimitNOFILE=40000
LimitNPROC=1048576
ExecStartPre=/sbin/modprobe ip_tables
ExecStartPre=/usr/bin/mkdir -p /run/flannel
ExecStartPre=/usr/bin/mkdir -p ${ETCD_SSL_DIR}

ExecStart=/usr/bin/rkt run --net=host \
   --stage1-path=/usr/lib/rkt/stage1-images/stage1-fly.aci \
   --insecure-options=image \
   --set-env=NOTIFY_SOCKET=/run/systemd/notify \
   --inherit-env=true \
   --volume runsystemd,kind=host,source=/run/systemd,readOnly=false \
   --volume runflannel,kind=host,source=/run/flannel,readOnly=false \
   --volume ssl,kind=host,source=${ETCD_SSL_DIR},readOnly=true \
   --volume certs,kind=host,source=/usr/share/ca-certificates,readOnly=true \
   --volume resolv,kind=host,source=/etc/resolv.conf,readOnly=true \
   --volume hosts,kind=host,source=/etc/hosts,readOnly=true \
   --mount volume=runsystemd,target=/run/systemd \
   --mount volume=runflannel,target=/run/flannel \
   --mount volume=ssl,target=${ETCD_SSL_DIR} \
   --mount volume=certs,target=/etc/ssl/certs \
   --mount volume=resolv,target=/etc/resolv.conf \
   --mount volume=hosts,target=/etc/hosts \
   ${FLANNEL_IMG}:${FLANNEL_VER} \
   --exec /opt/bin/flanneld \
   -- --ip-masq=true

# Update docker options
ExecStartPost=/usr/bin/rkt run --net=host \
   --stage1-path=/usr/lib/rkt/stage1-images/stage1-fly.aci \
   --insecure-options=image \
   --volume runvol,kind=host,source=/run,readOnly=false \
   --mount volume=runvol,target=/run \
   ${FLANNEL_IMG}:${FLANNEL_VER} \
   --exec /opt/bin/mk-docker-opts.sh -- -d /run/flannel_docker_opts.env -i

ExecStopPost=/usr/bin/rkt gc --mark-only

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/flanneld.service.d/10-change-etcd-ssl-dir.conf
[Service]
Environment="ETCD_SSL_DIR=/etc/ssl/alien"
# /etc/systemd/system/flanneld.service.d/20-env-config.conf
[Service]
EnvironmentFile=/run/flannel/options.env
# /etc/systemd/system/flanneld.service.d/50-network-config.conf
[Unit]
Description=flanneld
After=etcd2.service
Requires=etcd2.service
[Service]
ExecStartPre=/usr/bin/etcdctl --ca-file=/etc/ssl/alien/alienca.cert.pem --cert-file=/etc/ssl/alien/alien1.cert.pem --key-file=/etc/ssl/alien/alien1.key.pem --endpoints=https://10.169.1.131:2379 set /coreos.com/network/config '{ "Network": "10.1.0.0/16", "SubnetLen": 24, "Backend": { "Type": "vxlan"}}'

CoreOS Version

1185.3.0

$ cat /etc/os-release
NAME=CoreOS
ID=coreos
VERSION=1185.3.0
VERSION_ID=1185.3.0
BUILD_ID=2016-11-01-0605
PRETTY_NAME="CoreOS 1185.3.0 (MoreOS)"
ANSI_COLOR="1;32"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://github.com/coreos/bugs/issues"

Environment

VMWare ESXi

Other Information

$ docker info

Containers: 22
 Running: 11
 Paused: 0
 Stopped: 11
Images: 4
Server Version: 1.11.2
Storage Driver: overlay
 Backing Filesystem: extfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge null host
Kernel Version: 4.7.3-coreos-r2
Operating System: CoreOS 1185.3.0 (MoreOS)
OSType: linux
Architecture: x86_64
CPUs: 6
Total Memory: 15.68 GiB
Name: alien1
ID: ODJ6:LWCO:HHLG:MXWM:NKJC:KRRT:HZZN:XWFL:MLIH:IZKN:LTME:KI4K
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/

$ ps aux | grep docker 

docker daemon --host=fd:// --bip=10.1.72.1/24 --mtu=1450 --ip-masq=false --selinux-enabled
@mischief

This comment has been minimized.

mischief commented Nov 2, 2016

@pizzarabe are these messages from systemd-networkd affecting operation of docker or flannel?

@pizzarabe

This comment has been minimized.

pizzarabe commented Nov 3, 2016

@mischief I am sorry but I don't know what is affected... I had some problems with rkt after the update (rkt list does not response) but I guess this is not related.
The systemd issue systemd/systemd#3876 seems to be related.

Its not a critical issue rather annoying, because of the printed message.

Its merged but there is no release yet.

@crawford

This comment has been minimized.

Member

crawford commented Nov 4, 2016

We can backport that fix into the next release. It sounds like this error doesn't have any detrimental side effects. I'm curious if others are seeing this error as well or if there is something special about your environment. How are you configuring your network interfaces?

@pizzarabe

This comment has been minimized.

pizzarabe commented Nov 7, 2016

I use the following systemd drop in (per cloud-config) to configure my network interface

       - name: 10-static.network
            content: |
              [Match]
              MACAddress=00:50:56:b7:3f:bf
              [Network]
              Address=10.169.1.131/25
              Gateway=10.169.1.254
              DNS=10.169.9.151
              DNS=10.169.9.152
              DHCP=no
              LinkLocalAddressing=no
              IPv6AcceptRA=no

The flannel config is in the first post and docker does not have any special configuration.

systemd 232 was released some days ago. I guess we should test this if the alpha is updated

@crawford

This comment has been minimized.

Member

crawford commented Nov 7, 2016

I found a machine with a similar network setup and it started seeing this error when it jumped from systemd 229 to 231. This machine has been running for months without any adverse effects, so it should be safe to ignore the error. We'll go ahead and backport the fix for it.

@pizzarabe pizzarabe added this to the CoreOS Alpha 1235.0.0 milestone Nov 7, 2016

@crawford crawford self-assigned this Nov 16, 2016

@crawford crawford closed this Nov 16, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment