linked containers: can´t access target container

[busla]

This is my situation (see system output at the bottom):

I´ve got two containers that are linked with docker-options.

Container1 is my app that needs to post data to an API in Container2 exposed on port 8080. Expected env variables are fine on both containers so the linking seems to work.

On the other hand, the node app in Container1 times out when trying to send data to Container2 through port 8080. When I enter Container1 and try to ping Container2 I get no response. Note that posting to Container2´s port 8080 outside of my container works properly. Accessing Container1 outside Docker also works fine. I´m guessing it´s a networking issue between the containers.

Any ideas?

Containers:

c00841747475        dokku/earthquakes:latest            "/start web"             42 hours ago        Up 42 hours                                                                                  serene_ritchie
bda0d5c37ad1        sverhoeven/docker-cartodb           "/bin/bash /opt/start"   4 days ago          Up 3 days           0.0.0.0:3000->3000/tcp, 0.0.0.0:8080->8080/tcp, 0.0.0.0:8181->8181/tcp   angry_shockley

uname -a

Linux projects 3.13.0-63-generic #103-Ubuntu SMP Fri Aug 14 21:42:59 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

docker version

Client:
 Version:      1.8.2
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   0a8c2e3
 Built:        Thu Sep 10 19:19:00 UTC 2015
 OS/Arch:      linux/amd64
Get http:///var/run/docker.sock/v1.20/version: dial unix /var/run/docker.sock: permission denied.
* Are you trying to connect to a TLS-enabled daemon without TLS?
* Is your docker daemon up and running?
levy@projects:~$ sudo docker version
Client:
 Version:      1.8.2
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   0a8c2e3
 Built:        Thu Sep 10 19:19:00 UTC 2015
 OS/Arch:      linux/amd64


Server:
 Version:      1.8.2
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   0a8c2e3
 Built:        Thu Sep 10 19:19:00 UTC 2015
 OS/Arch:      linux/amd64

docker -D info

Containers: 23
Images: 230
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 284
 Dirperm1 Supported: false
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.13.0-63-generic
Operating System: Ubuntu 14.04.3 LTS
CPUs: 2
Total Memory: 1.947 GiB
Name: projects
ID: A4PD:HSRN:M4ZQ:2EM4:42UN:XIJI:II7X:Z2LV:RTZP:LI66:O7QA:64XD
WARNING: No swap limit support

dokku version
0.4.0

dokku plugin

 00_dokku-standard    0.4.0 enabled    dokku core standard plugin
  20_events            0.4.0 enabled    dokku core events logging plugin
  apps                 0.4.0 enabled    dokku core apps plugin
  backup               0.4.0 enabled    dokku core backup plugin
  build-env            0.4.0 enabled    dokku core build-env plugin
  certs                0.4.0 enabled    dokku core certificate management plugin
  checks               0.4.0 enabled    dokku core checks plugin
  common               0.4.0 enabled    dokku core common plugin
  config               0.4.0 enabled    dokku core config plugin
  docker-options       0.4.0 enabled    dokku core docker-options plugin
  dokku-mongo          1.0.0 enabled    dokku mongo service plugin
  domains              0.4.0 enabled    dokku core domains plugin
  enter                0.4.0 enabled    dokku core enter plugin
  git                  0.4.0 enabled    dokku core git plugin
  nginx-vhosts         0.4.0 enabled    dokku core nginx-vhosts plugin
  plugin               0.4.0 enabled    dokku core plugin plugin
  ps                   0.4.0 enabled    dokku core ps plugin
  shell                0.4.0 enabled    dokku core shell plugin
  tags                 0.4.0 enabled    dokku core tags plugin
  tar                  0.4.0 enabled    dokku core tar plugin

[josegonzalez]

The containers use internal ip addresses, so you'd need to also use that.

[busla]

Thanks for a quick reply.

Yes, my app is using the alias added when I linked the containers:

Deploy options:
    --link angry_shockley:earthquakes_cartodb
Run options:
    --link angry_shockley:earthquakes_cartodb

My app is using earthquakes_cartodb to communicate with the container. When I enter Container1 I can´t ping Container2 using it´s internal ip address or earthquakes_cartodb which is what the ip address resolves to. The alias is added properly to /etc/hosts in Container1.

[josegonzalez]

Can you include logs from a deploy after running dokku trace on?

[busla]

Deleted previous comment since I forgot to remove API keys :-/ now replaced with [removed]

ADMIN EDIT: output replaced with https://gist.github.com/michaelshobbs/d5339b8d20451161bff0

[michaelshobbs]

@busla it's almost always better to use https://gist.github.com for this sort of lengthy debug info.

This also looks incomplete. Can you use gist to post the full output?

[busla]

@michaelshobbs thanks for the tip 👍 will do from now on.

Here is the gist:
https://gist.github.com/busla/bbba5f39f65792e07d33

[michaelshobbs]

@josegonzalez i see where the deployed container is linked. what were you looking for in the trace output?

https://gist.github.com/busla/bbba5f39f65792e07d33#file-gistfile1-txt-L3660

[josegonzalez]

@busla what interface is that container process listening on?

[busla]

$ ip routes returns this:
172.17.0.0/16 dev eth0 proto kernel scope link src 172.17.0.225

Any other way that I can check which interface the container uses?

[josegonzalez]

Run netstat -plant | grep LISTEN inside the container and find your process.

[busla]

can´t apt-get install netstat even, probably network-related. I can only use port 80 it seems.

[josegonzalez]

Can you run that command outside of the container?

[busla]

yes.

tcp        0      0 10.0.3.1:53             0.0.0.0:*               LISTEN      -               
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -               
tcp        0      0 127.0.0.1:27017         0.0.0.0:*               LISTEN      -               
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      -               
tcp6       0      0 :::8181                 :::*                    LISTEN      -               
tcp6       0      0 :::22                   :::*                    LISTEN      -               
tcp6       0      0 :::3000                 :::*                    LISTEN      -               
tcp6       0      0 :::80                   :::*                    LISTEN      -               
tcp6       0      0 :::8080                 :::*                    LISTEN      -  

[busla]

Does this help?

[busla]

When I do docker inspect on the container I see that there is no bridge value:

   "NetworkSettings": {
        "Bridge": "",
        "EndpointID": "8a5c3ce86a536ae272146e4c021ab075895ed0a29c7194648e6db197d17354e0",
        "Gateway": "172.17.42.1",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "HairpinMode": false,
        "IPAddress": "172.17.2.199",
        "IPPrefixLen": 16,
        "IPv6Gateway": "",
        "LinkLocalIPv6Address": "",
        "LinkLocalIPv6PrefixLen": 0,
        "MacAddress": "02:42:ac:11:02:c7",
        "NetworkID": "690ce426a8fa8e07785353e69c5f312f5c22470100900296d5da8c895fc2630d",
        "PortMapping": null,
        "Ports": {},
        "SandboxKey": "/var/run/docker/netns/925962a9bdff",
        "SecondaryIPAddresses": null,
        "SecondaryIPv6Addresses": null
    },

Should that value be docker0 ?

[josegonzalez]

Maybe. How did you install dokku? On what sort of host?

[busla]

Ubuntu 12.04 (Linode).

[josegonzalez]

We absolutely do not support 12.04 Ubuntu. You need to use Ubuntu 14.04, and you may also be having issues with the kernel that is provided by linode. Please use 14.04 and see if you can replicate the issue.

[busla]

oops, my bad. I´m on 14.04 and I installed the new kernel properly.

[busla]

This issue happens when I restart my VPS. All custom Docker containers also disappear so I guess it has something to do with running non-dokku containers on a Dokku install.

[josegonzalez]

Dokku only restarts those containers, and runs dokku cleanup to stop all unstarted containers. We've mentioned this before, but essentially Dokku should be regarded as the sole arbiter of containers that exist on a server, and we cannot guarantee that other containers will not be removed etc.

[josegonzalez]

We should now be handling this properly. On instance reboot, we only start formerly running containers that are hosted by dokku.

We don't currently do anything about any other containers, so whatever docker does there is what docker does. Linked containers should start up again, and we won't clean up stopped containers until the following deploy (the start of which triggers a dokku cleanup call).