docker container unable to attach to calico network (using mesos containerizer works fine)

[lanejlanej]

Firstly, apologies if I am missing something obvious.

Running up tasks with calico networking under mesos using the mesos containerizer works fine.
But, running up tasks using the docker containerizer of directly in docker on a host with calico networking configured fails. The error suggests that there is a problem connecting with the unix docker socket, but using curl to interrogate the socket indicates that traffic is hitting it, and the container is created but when it tries to attach to the calico network (by the look of it) it dies.

This is with calico/cni v1.10.0, calicoctl v1.6.1, calico/node docker image v2.6.2 (calico/cni v1.11.0 doesn't work with our mesos 1.2 setup as it doesn't manage to find/use calico-ipam plugin)

Current Behavior

1. create a docker network
   docker network create --driver=calico --ipam-driver=calico-ipam calico-net-1-docker

as reflected by

curl --unix-socket /var/run/docker.sock http://localhost/events

{"Type":"network","Action":"create","Actor":{"ID":"1a1fbda4d066c7020c4e90c55899a498dce24a24f03708adfc675436dd91f7bd","Attributes":{"name":"calico-net-1-docker","type":"calico"}},"time":1509639826,"timeNano":1509639826180717680}

2. docker run --net calico-net-1-docker --name workload-A -tid busybox
   f4a18aa1d69aa25511c00b658c2b7379394d1663e518b3409bce126bc9870d19
   docker: Error response from daemon: failed to create endpoint workload-A on network calico-net-1-docker: NetworkDriver.CreateEndpoint: {"Err":"Network 1a1fbda4d066c7020c4e90c55899a498dce24a24f03708adfc675436dd91f7bd inspection error: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"}
   null.

docker socket matching events

{"status":"create","id":"f4a18aa1d69aa25511c00b658c2b7379394d1663e518b3409bce126bc9870d19","from":"busybox","Type":"container","Action":"create","Actor":{"ID":"f4a18aa1d69aa25511c00b658c2b7379394d1663e518b3409bce126bc9870d19","Attributes":{"image":"busybox","name":"workload-A"}},"time":1509640076,"timeNano":1509640076928211335}

and /var/log/calico/libnetwork/current:

time="2017-11-02T16:23:46Z" level=info msg=CreateNetwork JSON="{"NetworkID":"1a1fbda4d066c7020c4e90c55899a498dce24a24f03708adfc675436dd91f7bd","Options":{"com.docker.network.enable_ipv6":false,"com.docker.network.generic":{}},"IPv4Data":[{"AddressSpace":"CalicoGlobalAddressSpace","Pool":"0.0.0.0/0","Gateway":"0.0.0.0/0","AuxAddresses":null}],"IPv6Data":[]}"
time="2017-11-02T16:23:46Z" level=info msg="CreateNetwork response" JSON="{}"
time="2017-11-02T16:27:56Z" level=info msg=RequestAddress JSON="{"PoolID":"CalicoPoolIPv4","Address":"","Options":null}"
time="2017-11-02T16:27:56Z" level=info msg="Auto assigning IP from Calico pools"
time="2017-11-02T16:27:56Z" level=info msg="Auto-assign 1 ipv4, 0 ipv6 addrs for host 'calico-test-1-node-0'"
time="2017-11-02T16:27:56Z" level=info msg="Auto-assigned 1 out of 1 IPv4s: [10.250.50.47]"
time="2017-11-02T16:27:56Z" level=info msg="RequestAddress response" JSON="{"Address":"10.250.50.47/32","Data":null}"
time="2017-11-02T16:27:56Z" level=info msg=CreateEndpoint JSON="{"NetworkID":"1a1fbda4d066c7020c4e90c55899a498dce24a24f03708adfc675436dd91f7bd","EndpointID":"8972198a8ef0b973d56a65f38c3ed3dc8e6dd593adc5be5fb5e57d777e08d1be","Interface":{"Address":"10.250.50.47/32","AddressIPv6":"","MacAddress":""},"Options":{"com.docker.network.endpoint.exposedports":[],"com.docker.network.portmap":[]}}"
time="2017-11-02T16:27:56Z" level=error msg="Network 1a1fbda4d066c7020c4e90c55899a498dce24a24f03708adfc675436dd91f7bd inspection error: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
time="2017-11-02T16:27:56Z" level=info msg=ReleaseAddress JSON="{"PoolID":"CalicoPoolIPv4","Address":"10.250.50.47"}"
time="2017-11-02T16:27:56Z" level=info msg="Releasing IP addresses: [10.250.50.47]"

Environment

• Calico version: calico/cni v1.10.0, calico/node v2.6.2, calicoctl v1.6.1
• Orchestrator version (e.g. kubernetes, mesos, rkt): mesos 1.2.0
• Operating System and version: Ubuntu 16.04

[lanejlanej]

I'm wondering if this might be a profile/policy issue. Don't see any default profile or policy created for the docker networks.

[ozdanborne]

@lanejlanej sorry so much time has passed. This slipped through my notifications and I lost track of it.

Calico's libnetwork plugin is run inside the calico/node container, and needs to be able to access the docker socket. As such, calico/node must be run as --privileged, and must get the following volume mounts: -v /run/docker/plugins:/run/docker/plugins -v /var/run/docker.sock:/var/run/docker.sock. Are you volume mounting the docker socket in?

[ozdanborne]

@lanejlanej It's been some time since we heard from you, so I'm going to close this issue.

Again, apologies for not responding to your initial request sooner. Please reopen if you'd like to explore further.