weave-proxy does not remove bridged interfaces

[aksentyev]

user@host
❯ ip l|grep veth|wc -l
72

user@host
❯ DOCKER_HOST=unix:///var/run/weave/weave.sock docker run -it --rm  ruby:2.3 bash
root@5319411c63e3:/# exit

user@host 6s
❯ ip l|grep veth|wc -l
73

user@host
❯ DOCKER_HOST=unix:///var/run/weave/weave.sock docker run -it --rm  ruby:2.3 bash
root@4414eba69b10:/# exit

user@host
❯ ip l|grep veth|wc -l
74

Weave 1.6

docker exec weaveplugin ip l |grep veth|wc -l
74

[aksentyev]

This problem appears only when proxy is used. Maybe problem is in docker, but I don't know how to debug it. Maybe you have any idea. Thanks in advance.

[rade]

works for me

$ weave launch
$ ip l|grep veth|wc -l
4
$ docker $(weave config) run -it --rm  ruby:2.3 bash
root@a438b6cc16e1:/# exit
exit
$ ip l|grep veth|wc -l
4

I suspect this is a kernel/OS issue. And quite rare at that - if this was widespread we'd have heard about it. Are you running some exotic/ancient kernel?

[rade]

Also, what docker version are you running?

[rade]

Also, check the names of the leaking veths to determine whether they are docker of weave veths - weave veths have a "vethwe" prefix.

[aksentyev]

Ubuntu 15.10 \n \l

Linux host 4.2.0-16-generic #19-Ubuntu SMP Thu Oct 8 15:35:06 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Docker version 1.11.2, build b9f10c9

All veths have vethwe prefix. The problem exists on several nodes.

[rade]

Could you try a different kernel?

[rade]

Also, what other containers have you got running, besides weave? We've seen some issues in the past with the likes of cadvisor bind-mounting /var/run and preventing cleanup of network namespaces. See #1455.

[aksentyev]

Hello, I have tried the next configuration: docker 1.11.2, newer ubuntu 16.04, linux kernel 4.4.0-21-generic 37-Ubuntu SMP Mon Apr 18 18:33:37 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux. Problem is still exists.

$ ip link|grep ethwe|wc -l
3
$ DOCKER_HOST=unix:///var/run/weave/weave.sock docker run --rm -it ruby:2.3 bash
root@71ea4a2f3378:/# exit
$ DOCKER_HOST=unix:///var/run/weave/weave.sock docker run --rm -it ruby:2.3 bash
root@26b3d0d560e9:/# exit
$ ip link|grep ethwe|wc -l
5

I have only run:

registrator:     registry.service.consul:5000/registrator:latest   Up 22 hours
consul:            gliderlabs/consul-agent:0.6                                 Up 22 hours
weaveproxy:   weaveworks/weaveexec:1.6.0                              Up 22 hours
weaveplugin:  weaveworks/plugin:1.6.0                                      Up 22 hours
weave:            weaveworks/weave:1.6.0                                      Up 22 hours

[rade]

do you see the same problem when the registratror and consul containers aren't running?

[aksentyev]

@rade , it looks like registrator affects interface deletion. When it's not running everything works fine. I'll research this.

[aksentyev]

I see the following error in proxy logs:

Error forwarding chunked response body: write unix /var/run/weave/weave.sock->@: write: broken pipe

[rvandegrift]

I hit this in ECS with just weave, weave-proxy & ecs-agent running. Was able to narrow it down to ecs-agent running with -v /var/run/weave/weave.sock:/var/run/weave/weave.sock:rw. Clearly can't just nuke that!

Upgrading to weave 1.8.0 seems to have fixed it. Looking through the merges, I don't see quite see why.

System is Ubuntu xenial, Linux 4.4.0-45-generic, docker 1.12.2, weave 1.6.0 originally.

[brb]

I'm closing this.

@aksentyev @rvandegrift could you re-open the issue if the problem persists? thanks.