Support docker-swarm

[aisensiy]

I read the document in docker it said that it is possible to use dokku with swarm and I install dokku and a 3-node swarm cluster with vagrant and is there some method to let dokku's default docker host is the cluster?

[michaelshobbs]

Currently dokku is a single host solution. I think deis is the proposed multi host alternative, currently. I know some folks have been tinkering a bit with swarm and dokku but not sure where all that stands. I would definitely be interested in helping to spec out a swarm implementation, though.

Would you mind linking to that document from docker?

[aisensiy]

@michaelshobbs https://docs.docker.com/swarm/

[aisensiy]

@michaelshobbs We are trying to use deis to run a PaaS but it has no concept of addon or plugin in deis, which means I can just run a web application without cache, database.

[josegonzalez]

@aisensiy I don't think there is currently a way to script set the -H option on all docker commands. You could try the docker-options plugin, but that will break for the dokku logs command I think, among others.

I'll try and see if I can set this up this weekend, would be interesting.

[aisensiy]

@josegonzalez Thanks for your reply, I tried with -H option but still no work...

[blackxored]

I've played with this quite a bit, being able to have dokku running in a "swarmy", however most of the time gives me errors on not being able to find the image in the host, and I stopped there. I assume it has to do with the cleanup it runs on deploy. Anyways, I did have to modify dokku itself, so there's no easy way so far.

[josegonzalez]

@blackxored what did you do to modify dokku? I'm definitely into any changes necessary.

One thing I think that might be a PITA is that images are built on a single box, and I don't know if swarm knows how to locate "local" images. Might that be the reason why you can't easily scale up/down instances?

Seems related: docker-archive/classicswarm#806

Not sure of a good way to get around ^ if that is a real issue, other than creating a registry, pushing the image there, then scaling...

[blackxored]

@josegonzalez In 360 view I had to pass/obey/verify DOCKER_HOST, DOCKER_TLS_VERIFY, plus DOCKER_CERT_PATH, had to have it respect custom-made overlay net $DOCKER_NETWORK_NAME, and due to the nature of multi-host, disable the host's nginx altogether and run it in a container instead (mounting /etc/nginx and /home/dokku as volumes).

Anyways, between the volatile image errors on deployment and the fact that I was half-way through and clueless into having it send the right IP to nginx's configuration template I kinda gave up at the moment.

[blackxored]

The ghost images issues might have to do with the cleanup dokku does everytime, can someone confirm?

dokku/plugins/common/functions

Line 355 in c561a8b

docker rmi $(docker images -f 'dangling=true' -q) &> /dev/null &

[josegonzalez]

Image cleanup happens before a deploy for previously used images (that aren't attached to containers), so I doubt it.

[u2mejc]

Because the cluster manager presents itself just like the standard docker cli (dokku just wraps the cli today) I was able to get dokku/swarm working with only minor work. The biggest issue for dokku is that we configure a local nginx server. Disabling nginx and moving to another routing mechanism that would allow routing to any backend server would be required. Moving forward, we also need to remove the assumption that you're always on the same machine from dokku plugins. But that might not be a blocker for core.

[josegonzalez]

Could we use --net=host for swarm installs, and then use the publicly exposed port in nginx?

[josegonzalez]

@u2mejc mind wrapping up your changes to dokku to get initial support going? Would be good since in 0.5.x, we're going to have the ability to disable the nginx proxy (and potentially support alternative proxies...).

[josegonzalez]

Okay, I looked more into swarm because I have a need for this at work. Some thoughts:

• Because swarm won't relocate images to where they need to be - boo - we'll need to handle it ourselves:
  • replicate the images across the cluster first via docker export or docker save
  • push the image to a registry, then pull it down somehow (done automatically if all nodes are auth'd on the registry via docker login and X-Registry-Auth).
• Attaching local disk won't work. I don't think this is necessarily bad, since anyone using multiple hosts should just chuck persistent files to a separate store (s3/cloudfiles?)
• In 0.5.x, we can probably create a custom proxy plugin that communicates with some central load balancer.
  • This will almost assuredly require not using internal docker ips. @michaelshobbs is this possible to do with your refactor?
• Dokku should be communicating with one of the central manager instances. Easy to do if dokku lives on the manager or if dokku has the env vars set in a ~/.dokkurc file

[matthub]

@josegonzalez maybe the "affinity" flag could help with the image location issue (at least partially for a single container instance)? See https://docs.docker.com/swarm/scheduler/filter/

[josegonzalez]

Yeah but that doesn't really gain you anything other than now you have swarm on a single instance.

[cweagans]

I'm hoping to take a look at this pretty soon. I think it'll be doable without too much modification to Dokku.

[josegonzalez]

I'm going to close this issue for the following reasons:

• Dokku's explicit goal is to be a solid, single-server PaaS. Swarm is multi-server, so...
• Adding Swarm support makes things a bit more complex and we'll need to deal with swarm bugs in the Dokku core. They just rebuilt Swarm in Docker 1.12, and lots of stuff is half-baked, so the interface could change at any time, necessitating more work to keep the functionality working.
• Docker Swarm is just one of many schedulers, and I don't want to tie Dokku to a single scheduler.
  • A better way forward would be to refactor the ps, run, and enter plugins - as well as associated stuff like app.json handling - to be generic enough to execute against a specified scheduler through a well-defined interface.

For the record, I have already opened issues that would get us partially there - the network plugin would allow us to do quite a bit for instance - and made pull requests to add functionality needed to get Dokku into a place where we could add Swarm support.

If you'd like to work on this feature, a pull request that aligns with the above thoughts would be great (feel free to pm me with questions!). Otherwise, I think it's in the project's best interest to remain simple in this area.