First time deploys port mappings mess up after reboot

[eugeneware]

When I deploy a new application and reboot the docker container restarts on another port so all the nginx mappings fail and the app can't be accessed.

Steps to reproduce:

1. Push a brand new repo to dokku
2. Reboot the machine
3. Try to access the app.

You'll either the a 502 bad gateway error, or the wrong app (if the port got mapped to another app).

Here's an audit trail after I pushed a new app nodetest2.

After I initially pushed it, here's what the docker ps returned:

ID                  IMAGE                       COMMAND                CREATED             STATUS              PORTS               SIZE
3c6858a2eab4        app/nodetest2:latest        /bin/bash -c /start    38 seconds ago      Up 23 seconds       49160->5000         16.39 kB (virtual 985.9 MB)

Notice that it was listening on port 49160.

And cating the CONTAINER and PORT entries in the /home/git/nodetest2 did:

root@euge:/home/git/nodetest2# cat CONTAINER
3c6858a2eab4
root@euge:/home/git/nodetest2# cat PORT
49160
root@euge:/home/git/nodetest2# cat nginx.conf
upstream nodetest2 { server 127.0.0.1:49160; }
server {
  listen      80;
  server_name nodetest2.euge.co;
  location    / {
    proxy_pass  http://nodetest2;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $http_host;
  }
}

Looking at the config.json for the docker container gives:

{"ID":"3c6858a2eab44bac6bcd17e09d2749fd2fbe59289efcb981ac289308721851fb","Created":"2013-07-07T10:58:45.651908068Z","Path":"/bin/bash","Args":["-c","/start web"],"Config":{"Hostname":"3c6858a2eab4","User":"","Memory":0,"MemorySwap":0,"CpuShares":0,"AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"PortSpecs":["5000"],"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":["PORT=5000"],"Cmd":["/bin/bash","-c","/start web"],"Dns":null,"Image":"app/nodetest2","Volumes":{},"VolumesFrom":""},"State":{"Running":true,"Pid":3111,"ExitCode":0,"StartedAt":"2013-07-07T10:58:45.733151021Z","Ghost":false},"Image":"cb67ce2218ffc55c9e9ba52c7413246b0697edebcba01f63fdab3123a868ee74","NetworkSettings":{"IPAddress":"172.16.42.12","IPPrefixLen":24,"Gateway":"172.16.42.1","Bridge":"docker0","PortMapping":{"5000":"49160"}},"SysInitPath":"/usr/bin/docker","ResolvConfPath":"/etc/resolv.conf","Volumes":{}}

Everything is working and set to map to port 49160.

When I reboot, however, this is what docker ps shows:

ID                  IMAGE                       COMMAND                CREATED             STATUS              PORTS               SIZE
3c6858a2eab4        app/nodetest2:latest        /bin/bash -c /start    11 minutes ago      Up 11 seconds       49153->5000         16.39 kB (virtual 985.9 MB)

See how the port has now been mapped to 49153. Therefore the nginx port forwarding won't work and the app is inaccessible.

Checking the docker container file once again:

{"ID":"3c6858a2eab44bac6bcd17e09d2749fd2fbe59289efcb981ac289308721851fb","Created":"2013-07-07T10:58:45.651908068Z","Path":"/bin/bash","Args":["-c","/start web"],"Config":{"Hostname":"3c6858a2eab4","User":"","Memory":0,"MemorySwap":0,"CpuShares":0,"AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"PortSpecs":["5000"],"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":["PORT=5000"],"Cmd":["/bin/bash","-c","/start web"],"Dns":null,"Image":"app/nodetest2","Volumes":{},"VolumesFrom":""},"State":{"Running":true,"Pid":1029,"ExitCode":0,"StartedAt":"2013-07-07T11:09:05.82098963Z","Ghost":false},"Image":"cb67ce2218ffc55c9e9ba52c7413246b0697edebcba01f63fdab3123a868ee74","NetworkSettings":{"IPAddress":"172.16.42.3","IPPrefixLen":24,"Gateway":"172.16.42.1","Bridge":"docker0","PortMapping":{"5000":"49153"}},"SysInitPath":"/usr/bin/docker","ResolvConfPath":"/etc/resolv.conf","Volumes":{}}

And the port has changed to 49153.

I'm not sure what the solution is as my knowledge of docker is limited. You could rewrite the nginx conf file after reboot...

Strangely enough, however, when you do a SECOND push, then the whole thing works as it's not mapping to port 5000, and the port mappings stay correct the second push.

Not sure what the answer is.

[progrium]

Good catch, great info. Turns out Docker will not use the same ports when it's told to automatically restart containers when it restarts. It works on your second deploy because we basically manually restart the container with the right port. One option, as you said, is to rebuild the Nginx configs when docker restarts. Another option is to turn off automatic restart of containers when docker restarts and manually start them. The latter sounds more appropriate, but to me that makes it sound like a deficiency in docker (If we have to turn a useful feature off to do it right).

I'll poke around the Docker community and see how they feel.

[eugeneware]

Yeah. I've been scratching my head about this too. Seems kind of silly. Let me know what you find. I've been asking around IRC and the mailing list too as it seems to be a pretty fundamental problem.

If docker is for building a PaaS then having the ports stay after a reboot would have to be pretty fundamental I would have thought?

[fujin]

Have you guys considered using the IP addresses of the containers directly instead of using the docker port mapping feature? I have found the pattern to work quite well in situations like this: reverse proxy on the host (in another container, even, potentially), talking to pre-determined ports on the IP of containers -- easily exposed with docker inspect on the container ID. Since the containers are on the same subnet in the same bridge, they can easily talk between themselves. You only really have to re-map the reverse proxy container, if any!

[eugeneware]

@fujin That's a great idea. Seems a lot simpler. However, the same issue occurs. The IP addresses of the containers change on reboot unfortunately. Seems like the only way to go is to set all this stuff up on reboot unless the docker guys weigh in with a better alternative.

[eugeneware]

I've created a PR #88 that should address this issue that takes the nginx rewriting approach.

[eugeneware]

I've created an alternative solution relying on a central port registry that fixes this issue too. It's in PR #94

[giefferre]

There's another problem with the nginx configuration template file which is imho the cause of the error.

Solved w/ pull request #95

[amaltson]

👍 just ran into the exact same problem here.

[eugeneware]

Yeah. I've put in 2 PRs which I've been using in production now for about a month that solve the problem.

• Rewrite nginx.conf files based on latest Docker dynamic port mappings on server reboot #88 rewrites the ports on reboot
• Central Port registry to preserve docker container ports on reboot #94 creates a central port registry and does static port assignments upon container creation to avoid the dynamic rewrite.

Either PR will fix the issue.

In a real PaaS there would be a routing layer such as hipache to manage the routing, so maintaining any port mappings is less important. @progrium has been saying that he'll chat to the docker guys to work out the best solution to this for dokku.

[amaltson]

Thanks @eugeneware, should I just apply one of those PRs and follow the standard upgrade procedure?

[eugeneware]

@amaltson Yes. Though some of the changes affect the upstart scripts that get generated. If there's not too many deploys then it might be simpler to just do a fresh install. The main file you need to create and run is the dokku.conf upstart script. See the diff to see what it changes. Yell out if you have any problems.

[asm89]

I posted about this on the docker mailing list after having a chat on irc with @shykes from the docker team. Feel free to chime in: https://groups.google.com/forum/#!topic/docker-user/Py3YHb1C8Jo.

[eugeneware]

Thanks @asm89 I've put in my $0.02!

It sounds like on the face of it, from Solomon's response, that the only real reliable way to deal with the reboot issue is to use the docker API and reallocate ports (e.g. the approach in #88). That even if they try to keep the same ports after reboot (which would be nice!), that there is still the possibility with reconfiguration, or another service coming up before docker can grab the port - that the port mapping may not stick.

[progrium]

Flynn is working on a service discovery component that should help with
this issue nicely. If there are short term solutions, we can use them for
now, as long as they're marked short term and relatively modular (so we can
remove it later)

On Thu, Aug 15, 2013 at 8:50 PM, Eugene Ware notifications@github.comwrote:

Thanks @asm89 https://github.com/asm89 I've put in my $0.02!

It sounds like on the face of it, from Solomon's response, that the only
real reliable way to deal with the reboot issue is to use the docker API
and reallocate ports (e.g. the approach in #88#88).
That even if they try to keep the same ports after reboot (which would be
nice!), that there is still the possibility with reconfiguration, or
another service coming up before docker can grab the port - that the port
mapping may not stick.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/82#issuecomment-22746239
.

Jeff Lindsay
http://progrium.com

[asm89]

Tagged the issue as "enhancement" for now. It should be improved when the service discovery component of flynn can be pulled in.