docker daemon occasionally returns error: failed to set IPv6 gateway while updating gateway: file exists

[yugokato]

I have a python application that runs on a docker host (Ubuntu). The application basically allows users to concurrently start multiple containers on IPv4/IPv6 dualstack Macvlan network.
For example, if a user requests 3 containers, then the application tries to create and start 3 containers concurrently using 3 threads.

thread1: create -> start
thread2: create -> start
thread3: create -> start

Each container gets both IPv4 and IPv6 addresses assigned from the Macvlan network's IPv4/IPv6 pools.

Issue

The issue I encountered is the docker daemon occasionally returns below error when starting containers, and one of containers sometimes fails to start. It took about 250 times of attempts (each attempt = starting 2 containers concurrently and delete them) to see the error for 10 times.

docker: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:430: container init caused "process_linux.go:413: running prestart hook 0 caused \"error running hook: exit status 1, stdout: , stderr: time=\\\"2019-09-25T16:14:12-07:00\\\" level=fatal msg=\\\"failed to set IPv6 gateway while updating gateway: file exists\\\"\\n\""": unknown.

The issue does not happen when IPv6 is not enabled, or when IPv6 is enabled but starting just one container at a time.
This makes me assume that there is an issue where IPv6 network is not properly handled for concurrent access.

System information

lsb_release -a

Distributor ID: Ubuntu
Description: Ubuntu 18.04.2 LTS
Release: 18.04
Codename: bionic

uname -a

Linux hostname 4.4.0-159-generic #187-Ubuntu SMP Thu Aug 1 16:28:06 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

docker version

Client: Docker Engine - Community
Version: 19.03.2
API version: 1.40
Go version: go1.12.8
Git commit: 6a30dfc
Built: Thu Aug 29 05:28:19 2019
OS/Arch: linux/amd64
Experimental: false

Server: Docker Engine - Community
Engine:
Version: 19.03.2
API version: 1.40 (minimum version 1.12)
Go version: go1.12.8
Git commit: 6a30dfc
Built: Thu Aug 29 05:26:54 2019
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.2.6
GitCommit: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
runc:
Version: 1.0.0-rc8
GitCommit: 425e105d5a03fabd737a126ad93d62a9eeede87f
docker-init:
Version: 0.18.0
GitCommit: fec3683

Note

I've seen this issue on Ubuntu 16.04, docker version 17.09.0-ce too.

[skaliann]

I have also encountered the same issue (I think). When starting dockers concurrently w/ macvlan driver and IpV6, the api "SetGatewayIPv6" fails. We are starting about 20 dockers at the same time and almost always few of the dockers failed to start.

After adding more debug, this is what i found,
As part of the SetGatewayIPv6() -> programGateway() -> RouteAdd() is done to set the gw. But in the failure case the Gw is already set and its set to link-local ipv6 address..

Here's what i got from RouteList()

[
{Ifindex: 14777 Dst: xxxx:xxxx:xxxx:xxxx::/64 Src: Gw: Flags: [] Table: 254}
{Ifindex: 14777 Dst: fe80::/64 Src: Gw: Flags: [] Table: 254}
{Ifindex: 14777 Dst: Src: Gw: fe80::f60f:1bff:fecf:313f Flags: [] Table: 254}
{Ifindex: 14777 Dst: Src: Gw: fe80::f60f:1bff:fecf:207f Flags: [] Table: 254}
]

Thats the reason for RouteAdd to fail with EEXIST
What is not clear is how these 2 routes got added to some of the namespaces

Further debugging revealed that, these routes are getting added as part IPv6 Router Advertisement (RA). Looks like newly created network namespace has this enabled "/proc/sys/net/ipv6/conf/eth2/accept_ra"

Don't we have to set accept_ra=0 in macvlan driver??, otherwise SetGatewayIPv6() is going to randomly fail

Docker Version

Server Version: 18.05.0-ce
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871-dirty (expected: 4fc53a81fb7c994640722ac585fa9ca548971871)
init version: 949e6fa
Kernel Version: 4.8.28-WR9.0.0.20_cgl
OSType: linux
Architecture: x86_64

[rgsilva]

I can confirm this is an issue. Currently running a macvlan network for two containers and I'm randomly getting this error when starting them too fast (such as in docker-compose). Waiting for a bit (usually one or two minutes is enough) and then trying to restart the container seems to work just fine.

Docker Version

Server Version: 19.03.5-ce
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: d50db0a42053864a270f648048f9a8b4f24eced3.m
runc version: d736ef14f0288d6993a1845745d6756cfc9ddd5a
init version: fec3683
Kernel Version: 4.19.88-1-ARCH
OSType: linux
Architecture: armv7l