Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support docker-swarm #1795

Closed
aisensiy opened this issue Dec 22, 2015 · 18 comments
Closed

Support docker-swarm #1795

aisensiy opened this issue Dec 22, 2015 · 18 comments

Comments

@aisensiy
Copy link

@aisensiy aisensiy commented Dec 22, 2015

I read the document in docker it said that it is possible to use dokku with swarm and I install dokku and a 3-node swarm cluster with vagrant and is there some method to let dokku's default docker host is the cluster?

@michaelshobbs

This comment has been minimized.

Copy link
Member

@michaelshobbs michaelshobbs commented Dec 22, 2015

Currently dokku is a single host solution. I think deis is the proposed multi host alternative, currently. I know some folks have been tinkering a bit with swarm and dokku but not sure where all that stands. I would definitely be interested in helping to spec out a swarm implementation, though.

Would you mind linking to that document from docker?

@aisensiy

This comment has been minimized.

Copy link
Author

@aisensiy aisensiy commented Dec 23, 2015

@aisensiy

This comment has been minimized.

Copy link
Author

@aisensiy aisensiy commented Dec 23, 2015

@michaelshobbs We are trying to use deis to run a PaaS but it has no concept of addon or plugin in deis, which means I can just run a web application without cache, database.

@josegonzalez

This comment has been minimized.

Copy link
Member

@josegonzalez josegonzalez commented Dec 29, 2015

@aisensiy I don't think there is currently a way to script set the -H option on all docker commands. You could try the docker-options plugin, but that will break for the dokku logs command I think, among others.

I'll try and see if I can set this up this weekend, would be interesting.

@aisensiy

This comment has been minimized.

Copy link
Author

@aisensiy aisensiy commented Dec 31, 2015

@josegonzalez Thanks for your reply, I tried with -H option but still no work...

@blackxored

This comment has been minimized.

Copy link
Contributor

@blackxored blackxored commented Jan 2, 2016

I've played with this quite a bit, being able to have dokku running in a "swarmy", however most of the time gives me errors on not being able to find the image in the host, and I stopped there. I assume it has to do with the cleanup it runs on deploy. Anyways, I did have to modify dokku itself, so there's no easy way so far.

@josegonzalez

This comment has been minimized.

Copy link
Member

@josegonzalez josegonzalez commented Jan 2, 2016

@blackxored what did you do to modify dokku? I'm definitely into any changes necessary.

One thing I think that might be a PITA is that images are built on a single box, and I don't know if swarm knows how to locate "local" images. Might that be the reason why you can't easily scale up/down instances?

Seems related: docker/classicswarm#806

Not sure of a good way to get around ^ if that is a real issue, other than creating a registry, pushing the image there, then scaling...

@blackxored

This comment has been minimized.

Copy link
Contributor

@blackxored blackxored commented Jan 2, 2016

@josegonzalez In 360 view I had to pass/obey/verify DOCKER_HOST, DOCKER_TLS_VERIFY, plus DOCKER_CERT_PATH, had to have it respect custom-made overlay net $DOCKER_NETWORK_NAME, and due to the nature of multi-host, disable the host's nginx altogether and run it in a container instead (mounting /etc/nginx and /home/dokku as volumes).

Anyways, between the volatile image errors on deployment and the fact that I was half-way through and clueless into having it send the right IP to nginx's configuration template I kinda gave up at the moment.

blackxored added a commit to blackxored/dokku that referenced this issue Jan 2, 2016
I was running into issues with the IP addresses, and it seems
that it's using an incorrect/outdated syntax for retrieving
the container's IP Address. In multi-host networking
`NetworkSettings.IPAddress` is empty.

The docker documentation at https://docs.docker.com/engine/reference/commandline/inspect/
uses a different syntax which I've tested to work both on multi-host and
standard bridge networking.

If you're interested in the other issues and the work/hacks I've been doing related
to dokku#1795, let me know. In the meantime, I think this fix is needed.
@blackxored

This comment has been minimized.

Copy link
Contributor

@blackxored blackxored commented Jan 2, 2016

The ghost images issues might have to do with the cleanup dokku does everytime, can someone confirm?

docker rmi $(docker images -f 'dangling=true' -q) &> /dev/null &

@josegonzalez

This comment has been minimized.

Copy link
Member

@josegonzalez josegonzalez commented Jan 3, 2016

Image cleanup happens before a deploy for previously used images (that aren't attached to containers), so I doubt it.

@u2mejc

This comment has been minimized.

Copy link
Member

@u2mejc u2mejc commented Jan 4, 2016

Because the cluster manager presents itself just like the standard docker cli (dokku just wraps the cli today) I was able to get dokku/swarm working with only minor work. The biggest issue for dokku is that we configure a local nginx server. Disabling nginx and moving to another routing mechanism that would allow routing to any backend server would be required. Moving forward, we also need to remove the assumption that you're always on the same machine from dokku plugins. But that might not be a blocker for core.

@josegonzalez

This comment has been minimized.

Copy link
Member

@josegonzalez josegonzalez commented Jan 4, 2016

Could we use --net=host for swarm installs, and then use the publicly exposed port in nginx?

@josegonzalez

This comment has been minimized.

Copy link
Member

@josegonzalez josegonzalez commented Feb 14, 2016

@u2mejc mind wrapping up your changes to dokku to get initial support going? Would be good since in 0.5.x, we're going to have the ability to disable the nginx proxy (and potentially support alternative proxies...).

@josegonzalez

This comment has been minimized.

Copy link
Member

@josegonzalez josegonzalez commented Feb 28, 2016

Okay, I looked more into swarm because I have a need for this at work. Some thoughts:

  • Because swarm won't relocate images to where they need to be - boo - we'll need to handle it ourselves:
    • replicate the images across the cluster first via docker export or docker save
    • push the image to a registry, then pull it down somehow (done automatically if all nodes are auth'd on the registry via docker login and X-Registry-Auth).
  • Attaching local disk won't work. I don't think this is necessarily bad, since anyone using multiple hosts should just chuck persistent files to a separate store (s3/cloudfiles?)
  • In 0.5.x, we can probably create a custom proxy plugin that communicates with some central load balancer.
    • This will almost assuredly require not using internal docker ips. @michaelshobbs is this possible to do with your refactor?
  • Dokku should be communicating with one of the central manager instances. Easy to do if dokku lives on the manager or if dokku has the env vars set in a ~/.dokkurc file
@josegonzalez josegonzalez modified the milestone: v0.6.0 Mar 30, 2016
@matthub

This comment has been minimized.

Copy link

@matthub matthub commented Mar 30, 2016

@josegonzalez maybe the "affinity" flag could help with the image location issue (at least partially for a single container instance)? See https://docs.docker.com/swarm/scheduler/filter/

@josegonzalez

This comment has been minimized.

Copy link
Member

@josegonzalez josegonzalez commented Mar 30, 2016

Yeah but that doesn't really gain you anything other than now you have swarm on a single instance.

@cweagans

This comment has been minimized.

Copy link

@cweagans cweagans commented Jul 26, 2016

I'm hoping to take a look at this pretty soon. I think it'll be doable without too much modification to Dokku.

josegonzalez added a commit to josegonzalez/swarm that referenced this issue Jul 31, 2016
Fix the docker swarm documentation, which is better than misleading users while we investigate support.

Refs dokku/dokku#1795
@josegonzalez

This comment has been minimized.

Copy link
Member

@josegonzalez josegonzalez commented Sep 1, 2016

I'm going to close this issue for the following reasons:

  • Dokku's explicit goal is to be a solid, single-server PaaS. Swarm is multi-server, so...
  • Adding Swarm support makes things a bit more complex and we'll need to deal with swarm bugs in the Dokku core. They just rebuilt Swarm in Docker 1.12, and lots of stuff is half-baked, so the interface could change at any time, necessitating more work to keep the functionality working.
  • Docker Swarm is just one of many schedulers, and I don't want to tie Dokku to a single scheduler.
    • A better way forward would be to refactor the ps, run, and enter plugins - as well as associated stuff like app.json handling - to be generic enough to execute against a specified scheduler through a well-defined interface.

For the record, I have already opened issues that would get us partially there - the network plugin would allow us to do quite a bit for instance - and made pull requests to add functionality needed to get Dokku into a place where we could add Swarm support.

If you'd like to work on this feature, a pull request that aligns with the above thoughts would be great (feel free to pm me with questions!). Otherwise, I think it's in the project's best interest to remain simple in this area.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
7 participants
You can’t perform that action at this time.