New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

vxlan id leak #2614

Open
mmorhun opened this Issue Jan 25, 2017 · 9 comments

Comments

Projects
None yet
5 participants
@mmorhun
Copy link

mmorhun commented Jan 25, 2017

Short description

Sometimes after network deletion vxlan id isn't released back.

Description

Our system uses overlay networks in swarm cluster. Workflow procedure:

  • create an overlay network
  • add some containers into it
  • do some operations with that containers
  • remove containers
  • and finally remove network

It is normally, that such iteration are performed in parallel.
After some time our system running, we found leak of vxlanid. When all ids is consumed, our system failed to create new network (Swarm response: failed to allocate vxlan id : no bit available) and becomes useless. At the end, we have no overlay networks (According to output of docker network ls), and we cannot use new one even through cli:

# docker -H localhost:23750 network create -d overlay ssss3
2fc7f3e7346092a150592fe246eb84fcfd3e4e9508de41618a34c50acb033498
# docker -H localhost:23750 run -ti --network=ssss3 busybox sh
docker: Error response from daemon: Error response from daemon: couldn't get vxlan id for "10.0.8.0/24": failed to allocate vxlan id: no bit available.
# docker -H localhost:23750 network inspect ssss3
[
    {
        "Name": "ssss3",
        "Id": "2fc7f3e7346092a150592fe246eb84fcfd3e4e9508de41618a34c50acb033498",
        "Scope": "global",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "10.0.8.0/24",
                    "Gateway": "10.0.8.1/24"
                }
            ]
        },
        "Internal": false,
        "Containers": {},
        "Options": {},
        "Labels": {}
    }
]

Environment

AWS EC2 Centos 7.3
Master: c4.2xlarge
3 Nodes: r3.xlarge
Swarm 1.2.5
Zookeeper 3.4.9

Output of # docker info

Containers: 8
 Running: 4
 Paused: 0
 Stopped: 4
Images: 39
Server Version: 1.12.5
Storage Driver: devicemapper
 Pool Name: docker-202:0-1708215-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: /dev/vg-docker/data
 Metadata file: /dev/vg-docker/metadata
 Data Space Used: 29.52 GB
 Data Space Total: 69.79 GB
 Data Space Available: 40.27 GB
 Metadata Space Used: 40.24 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.107 GB
 Thin Pool Minimum Free Space: 6.979 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Library Version: 1.02.107-RHEL7 (2016-06-09)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null overlay bridge host
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-327.36.3.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 29.55 GiB
Name: node2.c5.codenvy-stg.com
ID: EMDA:WZ57:VTC2:KNIC:TDWE:I5AO:L2B2:MNS5:C3W7:KSWE:M4L2:ZIFH
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-ip6tables is disabled
Labels:
Cluster Store: zk://some-host.com:some-port
Cluster Advertise: 172.30.12.84:2375
Insecure Registries:
 some-host.com:5000
 127.0.0.0/8

Output of # docker version

Client:
 Version:      1.12.5
 API version:  1.24
 Go version:   go1.6.4
 Git commit:   7392c3b
 Built:        Fri Dec 16 02:23:59 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.5
 API version:  1.24
 Go version:   go1.6.4
 Git commit:   7392c3b
 Built:        Fri Dec 16 02:23:59 2016
 OS/Arch:      linux/amd64

Steps to reproduce

I've written bash script to reproduce our workflow without our system: vxlanid_leak.zip
Script requirements: curl, perl (perl-Archive-Tar lib), jq

vxlan id leak is reproduced with it as well. Just get free ids number, run ./launcher.sh 10 200, wait (~ an hour), and get free ids again. (1st parameter - number of threads, 2nd - number of iterations in each thread)
After launch the script (with parameters above) on our infrastructure 54 ids were leaked.

Additional info

We tried etcd 2.3.1, etcd 2.5.1, now zookeeper 3.4.9 but the result is the same.
KV storage is used by docker only.

After update docker to 1.12 the total number of available ids become much bigger, but leak is still remain.

We failed to reproduce it on pure docker.
We failed to reproduce it on pure docker which was connected to swarm.

To get number of free vxlanid we used this utility.

Related issue in docker: moby/moby#29756

@dongluochen

This comment has been minimized.

Copy link
Contributor

dongluochen commented Jan 25, 2017

cc @docker/core-libnetwork-maintainers.

@aboch

This comment has been minimized.

Copy link

aboch commented Jan 25, 2017

Thanks for the detailed explanation.
So looks like concurrent requests is what is needed to reproduce this.
Did not take this into account when I tried to reproduce the related docker issue.

I will take a look at it.

@mmorhun

This comment has been minimized.

Copy link

mmorhun commented Jan 26, 2017

Thank you for quick response.
I thought so, but we've just ran the script in 1 thread for 500 iteration and 23 ids were leaked.

@mmorhun

This comment has been minimized.

Copy link

mmorhun commented Jan 26, 2017

@aboch feel free to ping me if you fail to reproduce it. So far we can reproduce the bug only with pure swarm.

@aboch

This comment has been minimized.

Copy link

aboch commented Jan 28, 2017

I think I have found a way to reproduce the issue with a parallel go test.

@skryzhny

This comment has been minimized.

Copy link

skryzhny commented Feb 17, 2017

We moved to docker 1.12.5 and bits limit increased from 745 to 16776960 so that problem not such critical for us at the moment.

We still have that problem and after two days of running we got
Free 16776130 from 16776960

@nishanttotla

This comment has been minimized.

Copy link
Contributor

nishanttotla commented May 4, 2017

@aboch was this fixed in Docker? If it was, could you link to the fix?

@nishanttotla nishanttotla modified the milestones: 1.2.9, 1.2.7 Jun 15, 2017

@skryzhny

This comment has been minimized.

Copy link

skryzhny commented Sep 7, 2017

Any news?
We switched to 17.04.0-ce and issue is still here.

@nishanttotla

This comment has been minimized.

Copy link
Contributor

nishanttotla commented Sep 7, 2017

cc @mavenugo do you know if this issue was fixed in Docker recently?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment