New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for using a custom docker network #2085

Open
brugnara opened this Issue Apr 6, 2016 · 42 comments

Comments

Projects
None yet
6 participants
@brugnara
Copy link

brugnara commented Apr 6, 2016

High-Level Spec:

The following is a high-level specification for anyone who wishes to complete the work here.

# the default is to default to do nothing, so bridge mode in effect
# setting this would add `--network=NETWORK_NAME` to the docker options
dokku network:set APP network NETWORK_NAME

# setting to empty string makes the network report as null
# and doesn't add a docker option
dokku network:set APP network

# modify the network.go#GetContainerIpaddress() function
# to either inspect each docker network, starting with the defaut/configured
# network. we inspect all in case the network setting was changed but the application
# was not redeployed

# optional aliases for managing networks should support all of the current network
# flags. These are mostly passthru commands.

# alias for docker network create
dokku network:create network-name

# alias for docker network rm
dokku network:destroy network-name

# alias for docker network ls
dokku network:list network-name

# alias for docker network inspect
dokku network:info network-name

As far as attaching a container to a network, there are a few ways to tackle it, with pros and cons for each:

  1. Add two new triggers to the network plugin, docker-args-deploy and docker-args-run. This will handle the "deployed" containers, as well as any one-off containers. We can leave the build step as is. We'll then need some method of having nginx talk to the container, which may be difficult at best.
  2. Add a new plugin trigger that allows us to run a command against a container after it has started. This will let us connect an application to a network. Note that this will break any applications that expect database access at boot if the database is not accessible on the default network, but it will ease our usage of nginx. As datastores are not currently aware of networks, this is probably the better initial bet.
  3. Create the app container, attach it to the network, and then start it. This will require changes in a few different places, as well as a new plugin hook - scheduler-container-start - that we'll need to figure out where to implement. My gut says it should be shared responsibility between the scheduler-docker-local plugin to actually connect it to the network, and the network plugin for configuration.

Original Comment

Hi there.

I have a dokku on digitalocean (v0.5.4) and having some issue with overlay network.

This seems related to #1795 but my problem is nginx listening to the wrong ip.

I have 504 bad gateway on my webpage and dokku nginx:error-logs says:

2016/04/06 14:44:59 [error] 6181#0: *1 upstream timed out (110: Connection timed out) while connecting to upstream, client: 78.134.12.34, server: pre-login.mysite.it, request: "GET / HTTP/1.1", upstream: "http://10.0.0.6:5000", host: "pre-login.mysite.it", referrer: "http://pre-login.mysite.it/"

Please notice the ip: 10.0.0.6 which is the overlay ip for the container:

root@pre:~# docker inspect pre-login.web.1 | grep IPAddress
            "SecondaryIPAddresses": null,
            "IPAddress": "",
                    "IPAddress": "10.0.0.6",

but as I learnt from docker docs, overlay is a closed network, accessible only via the docker_gwbridge from the host. I am not understanding if the merged issue #1814 should solve this because the very same output of the command documented here gives the very same and wrong (I suppose) ip (10.0.0.6). The correct IP in my case should be 172.18.0.2 which is shown with this command in my dokku-app-container (docker exec -it pre-login.web.1 bash)

root@pre-login:/# ip addr | grep inet | grep eth
    inet 10.0.0.6/24 scope global eth0
    inet 172.18.0.2/16 scope global eth1

The problem here is that nginx points to the wrong ip, so from the dokku host:

curl 10.0.0.6:5000
Timeout error

curl 172.18.0.2:5000
<html here..>

Am I doing something wrong or missing something? Any help will be apreciated. Thank you.

@josegonzalez

This comment has been minimized.

Copy link
Member

josegonzalez commented Apr 6, 2016

Can you include all of the information we ask for in our issue template? It is helpful for debugging problems. Thanks.

@brugnara

This comment has been minimized.

Copy link
Author

brugnara commented Apr 6, 2016

Sure. I'll try to include all I can.

# uname -a
Linux pre.services.futur3.it 3.13.0-77-generic #121-Ubuntu SMP Wed Jan 20 10:50:42 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

# free -mh
             total       used       free     shared    buffers     cached
Mem:          2.0G       964M       1.0G       1.1M       110M       532M
-/+ buffers/cache:       321M       1.6G
Swap:           0B         0B         0B

# docker version
Client:
 Version:      1.10.3
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   20f81dd
 Built:        Thu Mar 10 15:54:52 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.10.3
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   20f81dd
 Built:        Thu Mar 10 15:54:52 2016
 OS/Arch:      linux/amd64

# docker -D info
Containers: 17
 Running: 1
 Paused: 0
 Stopped: 16
Images: 48
Server Version: 1.10.3
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 94
 Dirperm1 Supported: false
Execution Driver: native-0.2
Logging Driver: json-file
Plugins:
 Volume: local
 Network: overlay host bridge null
Kernel Version: 3.13.0-77-generic
Operating System: Ubuntu 14.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.955 GiB
Name: pre.services.mysite.it
ID: AUJP:B2EQ:QVV4:RFCN:5DDS:2TTA:3UJG:PIXR:BZT3:J4XQ:OEBJ:2YSL
WARNING: No swap limit support
Cluster store: consul://10.135.20.127:8500/network
Cluster advertise: 10.135.4.248:2375

# sigil -v
0.4.0

# docker run -ti gliderlabs/herokuish:latest herokuish version
herokuish: 0.3.9
buildpacks:
  heroku-buildpack-clojure   v75
  heroku-buildpack-erlang    fa17af9
  heroku-buildpack-go        v31
  heroku-buildpack-gradle    v17
  heroku-buildpack-grails    v20
  heroku-buildpack-java      v44
  heroku-buildpack-multi     v1.0.0
  heroku-buildpack-nodejs    v88
  heroku-buildpack-php       v95
  heroku-buildpack-play      v26
  heroku-buildpack-python    v77
  heroku-buildpack-ruby      v145
  heroku-buildpack-scala     v66
  buildpack-nginx            v5

# dokku version
sudo: unable to resolve host pre.services.mysite.it
0.5.4

# dokku plugin
plugn: dev
  00_dokku-standard    0.5.4 enabled    dokku core standard plugin
  20_events            0.5.4 enabled    dokku core events logging plugin
  apps                 0.5.4 enabled    dokku core apps plugin
  build-env            0.5.4 enabled    dokku core build-env plugin
  certs                0.5.4 enabled    dokku core certificate management plugin
  checks               0.5.4 enabled    dokku core checks plugin
  common               0.5.4 enabled    dokku core common plugin
  config               0.5.4 enabled    dokku core config plugin
  deployment-keys      0.1.0 enabled    Manage SSH deployment keys that should get injected into your containers on-build
  docker-options       0.5.4 enabled    dokku core docker-options plugin
  domains              0.5.4 enabled    dokku core domains plugin
  enter                0.5.4 enabled    dokku core enter plugin
  git                  0.5.4 enabled    dokku core git plugin
  hostkeys-plugin      0.1.0 enabled    Manage hostkeys for your container environment
  logs                 0.5.4 enabled    dokku core logs plugin
  named-containers     0.5.4 enabled    dokku core named containers plugin
  nginx-vhosts         0.5.4 enabled    dokku core nginx-vhosts plugin
  plugin               0.5.4 enabled    dokku core plugin plugin
  proxy                0.5.4 enabled    dokku core proxy plugin
  ps                   0.5.4 enabled    dokku core ps plugin
  shell                0.5.4 enabled    dokku core shell plugin
  storage              0.5.4 enabled    dokku core storage plugin
  tags                 0.5.4 enabled    dokku core tags plugin
  tar                  0.5.4 enabled    dokku core tar plugin

# docker inspect pre-login.web.1
[
    {
        "Id": "ed7464815632ce0a2d738b802d9fb43cc09c035537c7bf204fb4fa00f499e915",
        "Created": "2016-04-06T14:38:44.097333596Z",
        "Path": "/start",
        "Args": [
            "web"
        ],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 1375,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2016-04-06T18:41:23.306432019Z",
            "FinishedAt": "2016-04-06T17:49:37.304163604Z"
        },
        "Image": "sha256:464651447cb5b5cbddf3f50bc056b0833920e0fd5535420d98460f722cb15140",
        "ResolvConfPath": "/var/lib/docker/containers/ed7464815632ce0a2d738b802d9fb43cc09c035537c7bf204fb4fa00f499e915/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/ed7464815632ce0a2d738b802d9fb43cc09c035537c7bf204fb4fa00f499e915/hostname",
        "HostsPath": "/var/lib/docker/containers/ed7464815632ce0a2d738b802d9fb43cc09c035537c7bf204fb4fa00f499e915/hosts",
        "LogPath": "/var/lib/docker/containers/ed7464815632ce0a2d738b802d9fb43cc09c035537c7bf204fb4fa00f499e915/ed7464815632ce0a2d738b802d9fb43cc09c035537c7bf204fb4fa00f499e915-json.log",
        "Name": "/pre-login.web.1",
        "RestartCount": 0,
        "Driver": "aufs",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": null,
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "futur3.preprod",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "always",
                "MaximumRetryCount": 0
            },
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "ShmSize": 67108864,
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "KernelMemory": 0,
            "Memory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": -1,
            "OomKillDisable": false,
            "PidsLimit": 0,
            "Ulimits": null
        },
        "GraphDriver": {
            "Name": "aufs",
            "Data": null
        },
        "Mounts": [],
        "Config": {
            "Hostname": "pre-login",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PORT=5000",
                "DYNO='web.1'",
                "CACHE_PATH=/cache",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": [
                "/start",
                "web"
            ],
            "Image": "dokku/pre-login:latest",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {
                "dokku": ""
            },
            "StopSignal": "SIGTERM"
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "c22730071b09ddfadbe1145e02d6288d0953d1d0b10d96935c6f511e91ba12e0",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/c22730071b09",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "futur3.preprod": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "15599d0add7eace5a35c089cb88b5fdcf0d496f8e4b0372babe959dbddb19081",
                    "EndpointID": "ccad28da7c0e9d59252a12567a23668e992c903080200b6dce0468d4315f92c7",
                    "Gateway": "",
                    "IPAddress": "10.0.0.6",
                    "IPPrefixLen": 24,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:0a:00:00:06"
                }
            }
        }
    }
]

# cat /home/dokku/pre-login/nginx.conf

server {
  listen      [::]:80;
  listen      80;
  server_name pre-login.futur3.it;
  access_log  /var/log/nginx/pre-login-access.log;
  error_log   /var/log/nginx/pre-login-error.log;


  location    / {

    gzip on;
    gzip_min_length  1100;
    gzip_buffers  4 32k;
    gzip_types    text/css text/javascript text/xml text/plain text/x-component application/javascript application/x-javascript application/json application/xml  application/rss+xml font/truetype application/x-font-ttf font/opentype application/vnd.ms-fontobject image/svg+xml;
    gzip_vary on;
    gzip_comp_level  6;

    proxy_pass  http://pre-login;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $http_host;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-For $remote_addr;
    proxy_set_header X-Forwarded-Port $server_port;
    proxy_set_header X-Request-Start $msec;
  }
  include /home/dokku/pre-login/nginx.conf.d/*.conf;


}


upstream pre-login {
  server 10.0.0.6:5000;
}

# cat DOCKER_OPTIONS_DEPLOY
--restart=always
--net=futur3.preprod
--hostname=pre-login

To reproduce, create an overlay network:

dokku apps:create test-bug
docker network create -d overlay ovl1
echo "--net=ovl1" >> /home/dokku/test-bug/DOCKER_OPTIONS_DEPLOY

# DEPLOY THE APP, a simple node app, like this will do the job:

const http = require('http');
const hostname = '0.0.0.0';
const port = process.env.PORT;

http.createServer((req, res) => {
  res.writeHead(200, { 'Content-Type': 'text/plain' });
  res.end('Hello World\n');
}).listen(port, hostname, () => {
  console.log(`Server running at http://${hostname}:${port}/`);
});

# after this, a
cat /home/dokku/test-bug/nginx.conf 
# will report
upstream test-bug {
  server 10.0.0.6:5000;
}
# and a curl will complete the circle:

curl 10.0.0.6:5000
Timeout

docker exec -it CONTAINER_NAME bash
> ip addr
curl 172.18.0.2:5000
Hello World

If I can add something more, please ask. Thank you for your time!

@josegonzalez

This comment has been minimized.

Copy link
Member

josegonzalez commented Apr 6, 2016

I don't see how we're supposed to figure out that the ip address should be 172.18.0.2 from any of the info you posted. How would you suggest we do this?

@brugnara

This comment has been minimized.

Copy link
Author

brugnara commented Apr 6, 2016

here:

docker exec -it CONTAINER_NAME bash
> ip addr

or better
> ip addr | grep inet | grep eth
    inet 10.0.0.6/24 scope global eth0
    inet 172.18.0.2/16 scope global eth1
@michaelshobbs

This comment has been minimized.

Copy link
Member

michaelshobbs commented Apr 6, 2016

It would seem we do not support docker's overlay networking at this point. We use the output from docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$id" to populate the nginx upstream.

Not sure why the container is not available on the IP that it reports. Perhaps the container needs to be started with additional args to make that happen? Unfortunately, I have no experience with docker overlay networks.

If it is the case that the deployed container needs additional args to know to listen on that IP, then you could use the docker-options plugin to add that argument to the deploy phase.

http://dokku.viewdocs.io/dokku/docker-options/

@brugnara

This comment has been minimized.

Copy link
Author

brugnara commented Apr 6, 2016

@michaelshobbs I am doing with the file DOCKER_OPTIONS_DEPLOY that tells dokku to create a container with --net option. I think the problem here is nginx is configured to listen to the wrong IP in case of an overlay network. The command you suggest is wrong in this specific case and it should be possible to specify or find in some way the eth1 ip. Eth1 ip is the one connected on the docker_gwbridge, the only way for the host to connect to a container in an overlay network..

@michaelshobbs

This comment has been minimized.

Copy link
Member

michaelshobbs commented Apr 6, 2016

That's clearly the issue but what I'm saying is if the container tells us it's IP is 10.0.0.6 then we're doing the right thing in my opinion. I don't think parsing ip addr output is the right way to solve this.

@josegonzalez

This comment has been minimized.

Copy link
Member

josegonzalez commented Apr 6, 2016

@brugnara thats all well and good, but how should we detect that overlay ip address? If there is a bit in docker inspect that can tell us our initial check is going to be wrong, we can adjust however necessary, but atm we're using what docker tells us so....

@michaelshobbs

This comment has been minimized.

Copy link
Member

michaelshobbs commented Apr 6, 2016

Furthermore, how do we know which IP your app is actually listening on if there are two choices? Is it always the second one?

@brugnara

This comment has been minimized.

Copy link
Author

brugnara commented Apr 6, 2016

I know. Docker overlay is done this way and the ip 10.0.0.6 is private and accessible only from other container in that overlay network. I was asking for help because I don't know how to solve this :)
Any hints or workaround will be appreciated.

@josegonzalez

This comment has been minimized.

Copy link
Member

josegonzalez commented Apr 6, 2016

Do you mind asking in the ##docker irc room as to how we're supposed to detect the correct ip for a docker container? I'm willing to concede that our current method is crude, but it's the one we have, so we'll need more info to do anything more.

@brugnara

This comment has been minimized.

Copy link
Author

brugnara commented Apr 6, 2016

this issue says eth1 is the interface connected to that bridge, I will try to ask on this thread for more informations..

@michaelshobbs

This comment has been minimized.

Copy link
Member

michaelshobbs commented Apr 6, 2016

@brugnara on this host what is the output from docker network inspect docker_gwbridge?

@brugnara

This comment has been minimized.

Copy link
Author

brugnara commented Apr 6, 2016

@michaelshobbs

docker network inspect docker_gwbridge
[
    {
        "Name": "docker_gwbridge",
        "Id": "9faf389e2b15f7ef65b4acb367652ba9fb0ba98b0d20678d5e9046b57e902e2a",
        "Scope": "local",
        "Driver": "bridge",
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.18.0.0/16",
                    "Gateway": "172.18.0.1/16"
                }
            ]
        },
        "Containers": {
            "211029630a89104931026320c39caa7a67c4c3693cd366118daf5695bad15d06": {
                "Name": "gateway_211029630a89",
                "EndpointID": "e45a090fde0c2cd0cdeeeab8516582527ea23302b0e4cf9d9cf80e41ad67a22b",
                "MacAddress": "02:42:ac:12:00:03",
                "IPv4Address": "172.18.0.3/16",
                "IPv6Address": ""
            },
            "fb7193b44317276f5a879951a3a3970427e336d87085522c97d7496b4f5c1844": {
                "Name": "gateway_fb7193b44317",
                "EndpointID": "5c03f6b40f76af7a3ed2595d21ac9d3f03ee87c77ccff50092b70f06a87b0106",
                "MacAddress": "02:42:ac:12:00:04",
                "IPv4Address": "172.18.0.4/16",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.bridge.enable_icc": "false",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.name": "docker_gwbridge"
        }
    }
]
@michaelshobbs

This comment has been minimized.

Copy link
Member

michaelshobbs commented Apr 6, 2016

@josegonzalez if we're to implement this today, that command is our solution. However there is discussion on that docker issue mentioned about exposing that data in the container metadata.

@brugnara bottom line is dokku doesn't support this today

@michaelshobbs

This comment has been minimized.

Copy link
Member

michaelshobbs commented Apr 6, 2016

@brugnara also thank you for providing all this information. Overlay networking hasn't been brought up until now and I definitely appreciate the assistance in data gathering.

@michaelshobbs

This comment has been minimized.

Copy link
Member

michaelshobbs commented Apr 6, 2016

@josegonzalez thoughts on order of override? overlay up first, then follow the rest of the current logic? Probably? ¯_(ツ)_/¯

@josegonzalez

This comment has been minimized.

Copy link
Member

josegonzalez commented Apr 6, 2016

Commented in the docker issue but:

Would be great if we had a canonical way of deciding how to retrieve the ip/port(s) of a container. For instance, we (dokku) currently handle the following case:

- single server, no overlay

In this case, the user wanted to do:

- single server, overlay

And I am currently investigating official docker swarm support, which requires:

- multiple servers, overlay

Ideally we have a `docker inspect` command we can run that will:

- tell us if it's an overlay network or not, and then either:
  - retrieve the non-overlay ip/port(s)
  - retrieve the overlay ip/port(s)

Sounds like we're going to maybe have to punt on docker-swarm support until upstream gets this into master...

@josegonzalez

This comment has been minimized.

Copy link
Member

josegonzalez commented Apr 6, 2016

Oddly enough, the inspect of that bridge doesn't include the inspected container that @brugnara output. @brugnara can you do the following:

docker ps -a

And gist the output?

@brugnara

This comment has been minimized.

Copy link
Author

brugnara commented Apr 6, 2016

@josegonzalez

docker ps -a
CONTAINER ID        IMAGE                         COMMAND                  CREATED             STATUS                         PORTS               NAMES
fb7193b44317        dokku/test-bug:latest         "/start web"             51 minutes ago      Up 51 minutes                                      test-bug.web.1
ce86fbcc453b        6a6887b87c86                  "/bin/bash -c 'mkdir "   51 minutes ago      Exited (0) 51 minutes ago                          happy_feynman
8ffaa1399db6        d6b7c8817794                  "/bin/bash -c 'mkdir "   51 minutes ago      Exited (0) 51 minutes ago                          evil_carson
2b6d3ae1e332        9387b0807123                  "/build"                 51 minutes ago      Exited (0) 51 minutes ago                          gigantic_brattain
4830b580921f        2ff35a53a6a4                  "/bin/bash -c 'cat >>"   51 minutes ago      Exited (0) 51 minutes ago                          agitated_bose
441c64e45471        3f811fb8218e                  "/bin/bash -c 'cat >>"   51 minutes ago      Exited (0) 51 minutes ago                          goofy_shirley
e18f016a894c        8d68efc582d1                  "/bin/bash -c 'mkdir "   51 minutes ago      Exited (0) 51 minutes ago                          romantic_leakey
630c7c3fb3eb        9e8a90093a4a                  "/bin/bash -c 'cat >>"   51 minutes ago      Exited (0) 51 minutes ago                          big_pasteur
62013ca4a8a3        33f084b21c8b                  "/bin/bash -c 'cat >>"   51 minutes ago      Exited (0) 51 minutes ago                          gloomy_turing
b21f3fb02925        d5c43cc6df39                  "/bin/bash -c 'cat >>"   51 minutes ago      Exited (0) 51 minutes ago                          compassionate_hoover
a6526a04839d        61054d9d8cbc                  "/bin/bash -c 'mkdir "   51 minutes ago      Exited (0) 51 minutes ago                          silly_goldberg
bc49186591d6        83a6efa94014                  "/bin/bash -c 'cat >>"   51 minutes ago      Exited (0) 51 minutes ago                          hopeful_joliot
f836f1585e52        68b69afe697a                  "/bin/bash -c 'for EN"   51 minutes ago      Exited (0) 51 minutes ago                          hopeful_brattain
8b45b7981466        gliderlabs/herokuish          "/bin/bash -c 'mkdir "   51 minutes ago      Exited (0) 51 minutes ago                          evil_swartz
85d1014fa11a        ad7b9da3d988                  "/start web"             About an hour ago   Exited (255) 49 minutes ago                        test-bug.web.1.1459976177
211029630a89        dokku/pre-login:latest        "/start web"             About an hour ago   Up About an hour                                   pre-login.web.1
083cbcf50d84        gliderlabs/herokuish:latest   "herokuish version"      About an hour ago   Exited (0) About an hour ago                       big_leakey
root@pre:~#
@josegonzalez

This comment has been minimized.

Copy link
Member

josegonzalez commented Apr 6, 2016

Okay so it does have the containers listed. Is there a way to enumerate all network overlays?

@brugnara

This comment has been minimized.

Copy link
Author

brugnara commented Apr 6, 2016

@josegonzalez

docker network ls
NETWORK ID          NAME                DRIVER
15599d0add7e        mysite.preprod      overlay
34ad8be617a6        ovl2                overlay
5cccd03cf05c        overlay1            overlay
1c739974ecf5        network.local       bridge
9faf389e2b15        docker_gwbridge     bridge
509a752a83d1        bridge              bridge
10d91c8c295b        none                null
e9ed93dcc19d        host                host
@brugnara

This comment has been minimized.

Copy link
Author

brugnara commented Apr 6, 2016

but as you can see, an inspect of an overlay network does not provide such information:

docker network inspect ovl2
[
    {
        "Name": "ovl2",
        "Id": "34ad8be617a674c0e24cf36e79008f6e754bebd478c74ea328cba4d742bd42d6",
        "Scope": "global",
        "Driver": "overlay",
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "192.168.0.0/16"
                }
            ]
        },
        "Containers": {
            "fb7193b44317276f5a879951a3a3970427e336d87085522c97d7496b4f5c1844": {
                "Name": "test-bug.web.1",
                "EndpointID": "7b8e65fe3192109bf808facfec5d6c8bfd20818fa37e01a43e6724b7293cbeea",
                "MacAddress": "02:42:c0:a8:00:04",
                "IPv4Address": "192.168.0.4/16",
                "IPv6Address": ""
            }
        },
        "Options": {}
    }
]
@josegonzalez

This comment has been minimized.

Copy link
Member

josegonzalez commented Apr 6, 2016

We could probably do something like:

  • enumerate all overlay networks
    • check if container id in overlay network
      • use IPv4Address from overlay network if not empty (parse out the network mask of course)
      • use IPv6Address from overlay network if not empty (parse out the network mask of course)
      • fallback to our current ip checking
    • if not in an overlay network, use current ip checking

Thoughts? This definitely won't be supported soon by us, but I would accept a patch that does the above, assuming @michaelshobbs is okay with it.

@josegonzalez

This comment has been minimized.

Copy link
Member

josegonzalez commented Apr 6, 2016

Aside: how has no one else ever had this issue/complained about it?

@michaelshobbs

This comment has been minimized.

Copy link
Member

michaelshobbs commented Apr 6, 2016

It would seem our primary user base doesn't use overlay networking...

@michaelshobbs

This comment has been minimized.

Copy link
Member

michaelshobbs commented Apr 6, 2016

@josegonzalez Seems like we would just run something like this to extract the IPv4Address. Similar for IPv6Address. No?

$ docker network inspect -f '{{((index .Containers "$CONTAINER_ID") 0).IPv4Address}}' docker_gwbridge
@michaelshobbs

This comment has been minimized.

Copy link
Member

michaelshobbs commented Apr 7, 2016

@brugnara oh yeah totally makes sense. I'm sure this will be implemented in the near future.

@josegonzalez

This comment has been minimized.

Copy link
Member

josegonzalez commented Apr 9, 2016

@michaelshobbs I think you can name overlay networks, so we can't just assume that name, right?

@josegonzalez

This comment has been minimized.

Copy link
Member

josegonzalez commented Jun 25, 2016

@michaelshobbs Can this be rolled into #1795, or is it a completely new feature? I think yes, but leaving the final say up to you since you have a better feel for whats coming in the new docker swarm/networking world.

@josegonzalez

This comment has been minimized.

Copy link
Member

josegonzalez commented Sep 1, 2016

Closing in favor of building this into a new network plugin.

@crisward

This comment has been minimized.

Copy link
Contributor

crisward commented Jan 20, 2018

Just looking into this issue as I'm currently looking for a neater solution to hosting a database on another server. I currently use ambassador containers and a ssh tunnel.

It seems one solution may be to write an overlay network plugin. If a network is setup with

docker network create -d overlay mynet --attachable

Then networks can be added to containers with

docker network connect mynet sample.web.1  

This would need to be done after each push. However this should be possible inside a plugin by running this after each push / rebuild with the post-deploy trigger. A list of network names would just need to be stored for each app. A file in the app folder called OVERLAY_NETWORKS may make sense.

I realise dokku would like to be a single host solution, but my small company has 100'ish sites using dokku and only one app needs a database on another server. I think this functionality adds a bit of wriggle room in the scope of how dokku can be used. What do you think?

@josegonzalez

This comment has been minimized.

Copy link
Member

josegonzalez commented Jan 20, 2018

@crisward would your company be interested in sponsoring some some work to enable proper network handling in Dokku?

@crisward

This comment has been minimized.

Copy link
Contributor

crisward commented Jan 20, 2018

It'd probably be easier for us to contribute a plugin. We're a small web company based in the north of England, which unfortunately doesn't equate with big profits 😞

@josegonzalez

This comment has been minimized.

Copy link
Member

josegonzalez commented Jan 20, 2018

Then feel free to contribute the PR to do this.

Note that you should probably just setup iptables on the other server and only allow access to the database from your app server/dev environments.

@josegonzalez

This comment has been minimized.

Copy link
Member

josegonzalez commented Jan 21, 2018

@michaelshobbs just a note: that inspect call you mentioned works.


@crisward We would probably want to add this detection to the network plugin like so:

# the default is to default to do nothing, so bridge mode in effect
# setting this would add `--network=NETWORK_NAME` to the docker options
dokku network:set APP network NETWORK_NAME

# setting to empty string makes the network report as null
# and doesn't add a docker option
dokku network:set APP network

# we'll want to add two new triggers to the network plugin, `docker-args-deploy`
# and `docker-args-run`. This will handle the "deployed" containers, as well as 
# any one-off containers. We can leave the `build` step as is. 

# modify the network.go#GetContainerIpaddress() function
# to either inspect each docker network, starting with the defaut/configured
# network. we inspect all in case the network setting was changed but the application
# was not redeployed

# optional aliases for managing networks should support all of the current network
# flags. These are mostly passthru commands.
# alias for docker network create
dokku network:create network-name
# alias for docker network rm
dokku network:destroy network-name
# alias for docker network ls
dokku network:list network-name
# alias for docker network inspect
dokku network:info network-name

We'd also want documentation caveats around linking plugins (which currently only support bridge mode) and how this might affect the proxy plugin (can host nginx talk to something not on the bridge network?).

@josegonzalez josegonzalez reopened this Jan 21, 2018

@crisward

This comment has been minimized.

Copy link
Contributor

crisward commented Jan 21, 2018

If you add --network=NETWORK_NAME it'll be isolated from the bridge network, so nginx won't be able to see it. If you leave that option off and use docker network connect NETWORK_NAME APP it'll be on both networks, so nginx does work. But the latter needs running after the container has started. The command also works with plugins. I've tried this with the mariadb plugin and a small node app. However the network needs to be created with the --attachable flag mentioned above. I believe you also need a fairly recent version of docker for this to work.

@crisward

This comment has been minimized.

Copy link
Contributor

crisward commented Jan 21, 2018

@josegonzalez

This comment has been minimized.

Copy link
Member

josegonzalez commented Jan 21, 2018

Thats cool, but might interfere with applications that make database connections on boot. We might need to do a docker create to create the container, docker network connect to attach it to the network, and then docker run. We can use the network-get-property trigger to see if there is a configured network so we know whether or not to do this.

I believe all the logic for starting containers is in the common/functions bash file.

@josegonzalez josegonzalez changed the title overlay network, nginx listening to the wrong container ip Add support for using a custom network Apr 27, 2018

@josegonzalez josegonzalez changed the title Add support for using a custom network Add support for using a custom docker network Apr 27, 2018

@josegonzalez josegonzalez added this to the v1.0.0 milestone Jan 13, 2019

@aivils

This comment has been minimized.

Copy link

aivils commented Feb 18, 2019

@crisward , @josegonzalez Thank You for explaining the trouble. I released a small single liner plugin: https://github.com/ithouse/dokku-connect-network solves my troubles. It attach second network to the containers. I don't know it uses right trigger. At least it works for me. Containers can communicate each with other.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment