Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support compose version > 2 #2054

Closed
dazinator opened this issue Jul 19, 2018 · 72 comments
Closed

support compose version > 2 #2054

dazinator opened this issue Jul 19, 2018 · 72 comments
Labels
area/stack-creation area/stacks dep/libcompose kind/enhancement

Comments

@dazinator
Copy link

@dazinator dazinator commented Jul 19, 2018

Is your feature request related to a problem? Please describe.

I tried to deploy a stack from the stacks tab and pasting in contents of a docker-compose.yml file.
Unfortunately the yml in question is using version 2.4 of the compose spec as it needs things added in that version.

I noted the message in the UI that says only compose version 2 is supported. True to that warning, I got an error when attempting to deploy my 2.4 version.

Would you consider adding supporting for the latest version?

@deviantony
Copy link
Member

@deviantony deviantony commented Jul 19, 2018

Yes, we'll consider this as an enhancement. It implies to upgrade libcompose to support compose > 2

@f3l1x
Copy link

@f3l1x f3l1x commented Jul 28, 2018

Thanks. I'm looking forward to it.

@flothemaker
Copy link

@flothemaker flothemaker commented Aug 11, 2018

If you upgrade libcompose, would it be possible to deploy compose version >3 from portainer?

@deviantony
Copy link
Member

@deviantony deviantony commented Aug 11, 2018

@braaccckkk I believe this is the way to go to support version > 2

@mariolopjr
Copy link

@mariolopjr mariolopjr commented Aug 12, 2018

@deviantony That would be amazing! I didn't realize just how many version 3 features I used (even version ~2.4 like HEALTHCHECK, etc.). Thanks to all for the wonderful work! :)

@acamilleri
Copy link

@acamilleri acamilleri commented Aug 20, 2018

Strange...
I'm using docker-compose 3.6 version with Portainer and it works very well.

@deviantony
Copy link
Member

@deviantony deviantony commented Aug 20, 2018

@krysennn You can deploy a Swarm stack (docker stack deploy equivalent) version > 3, but we only support Standalone stacks (docker-compose equivalent) version 2.

@acamilleri
Copy link

@acamilleri acamilleri commented Aug 20, 2018

Oh! You're right, sorry i'm confused :)

@MicahZoltu
Copy link

@MicahZoltu MicahZoltu commented Sep 1, 2018

Where can I get more information on the difference between Swarm Stacks and Standalone Stacks? I use docker-compose files to manage my deployments and make them portable across tools, but in Portainer UI I don't see a way to deploy a Swarm Stack, only the v2 stacks. I'm really struggling with the limitations of v2.0 docker-compose files (in particular, networking related) and would like to figure out how to use a newer version of docker-compose files (e.g., v3 or at least v2.4).

@ncresswell
Copy link
Member

@ncresswell ncresswell commented Sep 1, 2018

@MicahZoltu
Copy link

@MicahZoltu MicahZoltu commented Sep 1, 2018

Not that I know of. :) I have a host (Ubuntu), I installed Docker CE on it, I ran the Portainer container. If that doesn't result in me having a Swarm cluster then no, I don't have one.

Feel free to redirect me to some documentation if I'm just missing some critical piece of a puzzle here. I am using Portainer to manage a single docker host, not a complex multi-host orchestration system. I just want access to newer versions of docker-compose is all.

@deviantony
Copy link
Member

@deviantony deviantony commented Sep 1, 2018

Hi @MicahZoltu

A simple docker swarm init command will allow you to create a simple one node Swarm cluster and give you access to Swarm features.

Once your host is in Swarm mode, Portainer will auto-detect it and you'll be able to use Compose v3+ to deploy stacks.

@josephduchesne
Copy link

@josephduchesne josephduchesne commented Nov 19, 2018

These aren't exactly equivalent since there are a decent number of docker-compose options that docker swarm ignores (--device, --privileged, etc.). Support for docker-compose 3.X would be really nice!

@deviantony deviantony changed the title support compose 2.4 support compose version > 2 Jan 3, 2019
@lsascha
Copy link

@lsascha lsascha commented Jan 22, 2019

Just noticed this limitation for non-stack setups after preparing templates etc... really annoying :(

@ncresswell
Copy link
Member

@ncresswell ncresswell commented Jan 22, 2019

@joseluisq
Copy link

@joseluisq joseluisq commented Jan 22, 2019

More info Libcompose Unmaintained

@deviantony deviantony added the dep/libcompose label Jan 22, 2019
@lsascha
Copy link

@lsascha lsascha commented Jan 22, 2019

Thanks. i now just initialized a single node swarm to use newer compose file versions.
I was just baffled and unaware that this limitation existed because i used it successfully on a swarm setup before,

Is libcompose not used for swarm setups?

@deviantony
Copy link
Member

@deviantony deviantony commented Jan 22, 2019

@lsascha Portainer uses libcompose behind the scene to deploy stacks on standalone Docker hosts (as you would do when using the docker-compose tool) and uses the Docker CLI binary to deploy stacks on Swarm environments (equivalent of docker stack deploy ... command).

@lsascha
Copy link

@lsascha lsascha commented Jan 22, 2019

thx for the explanation. Now another stupid question. Why is the official docker-compose CLI tool not used to deploy to standalone Docker hosts? ;)

@deviantony
Copy link
Member

@deviantony deviantony commented Jan 22, 2019

Mainly because the Portainer image aims to be a lightweight multi-platform image that can be deployed on any OS/arch and that the compose binary is only available for linux and windows amd64.

We would either need to bundle python into our Portainer images or create our own unofficial build pipeline for compose on the different platforms we support (assuming it's possible).

Overall, interfacing with an API/library is much easier/has less constraints than interfacing with a binary. Although I'm aware that our possibilities are limited in this context.

If we decide to limit our os/arch target scope we would be able to switch to bundling/using the official compose binary. A potential way to do it might be to disable stack management on the platform where the binary is not available.

Another way of doing it would be to create an extension that would bring enhanced compose support for these platforms.

@doubled1
Copy link

@doubled1 doubled1 commented Feb 3, 2019

What about the other way around? Use the binary if available and fall back to what it uses right now if it isn't?

@deviantony
Copy link
Member

@deviantony deviantony commented Feb 3, 2019

That could be a potential way of working around this too.

@alphaDev23
Copy link

@alphaDev23 alphaDev23 commented Feb 24, 2019

Very frustrating. Deploying stacks via docker-compose 3.x version works well in Portainer 1.17.1. Furthermore, 3.x fully supports deploying swarm stacks per the docker-compose reference: "A good place to start is the Getting Started tutorial which uses version 3 Compose stack files to implement multi-container apps, service definitions, and swarm mode." See also https://docs.docker.com/compose/compose-file/

@HWiese1980
Copy link

@HWiese1980 HWiese1980 commented Aug 29, 2019

And on a single Swarm node, now that I have initialized it?

@KopfKrieg
Copy link

@KopfKrieg KopfKrieg commented Aug 30, 2019

Single node swarm cluster is massively beneficial.

Doesn't it also have drawbacks?

@MicahZoltu
Copy link

@MicahZoltu MicahZoltu commented Aug 30, 2019

Doesn't it also have drawbacks?

Not that I have run into now that I'm fully migrated. I think the reason docker didn't just automatically move everyone to swarm is that there are breaking changes between non-swarm and swarm docker. All of my single-node docker hosts are swarm clusters of 1.

@bjmi
Copy link

@bjmi bjmi commented Sep 9, 2019

Doesn't it also have drawbacks?

Privileged mode is not supported in docker swarm.

@ncresswell
Copy link
Member

@ncresswell ncresswell commented Sep 9, 2019

@Desnoo
Copy link

@Desnoo Desnoo commented Jan 15, 2020

There is another library docker/cli maintained by docker org. This library is referenced in the libcompose Library. The pitfall is that only docker-compose files with version >= 3 are supported.

Maybe this could be used in the future?

@tcurdt
Copy link

@tcurdt tcurdt commented Jan 24, 2020

It would be really great if the supported docker compose format version could be stated explicitly in the README.

@PackElend
Copy link

@PackElend PackElend commented Feb 2, 2020

Privileged mode is not supported in docker swarm.

there is hope: moby/moby#24862

This is just for my home lab. One physical machine, I don't deal with pods/clusters/nodes. Very simple setup. I just need to be able to manage my docker compose projects via the web application.

this my scenario too. It is properly a silly question but I would like to double-check.
Solutions given in

only applicable when using docker-compose V2, aren't they?

@itsconquest
Copy link
Member

@itsconquest itsconquest commented Feb 3, 2020

only applicable when using docker-compose V2, aren't they?

Portainer itself cannot deploy a stack from yaml > v2 on a standalone host, due to our use of the libcompose library which does not support it. You can still deploy Portainer and the agent via docker-compose up with yaml > v2

Hope this helps!

@PackElend
Copy link

@PackElend PackElend commented Feb 3, 2020

I'm a bit lost here.
docker-compose portainer up
docker-compose portainer-agent up
gonna work, regardless the yaml version, as docker-compose is called from the shell or any other tool doing so.

Portainer cannot do docker-compose my_remote_standalone_container up if yaml is >v2, what is nowadays the standard if you running multi-container Docker applications on a single docker engine.
Instead portainer can do docker stack deploy my_remote_stack.

If you want to want to switch from compose to swarm a good point to start is Docker Swarm – Single Node, inst' it?

btw, compose vs stack is well explained in The Difference Between Docker Compose And Docker Stack

@itsconquest
Copy link
Member

@itsconquest itsconquest commented Mar 11, 2020

@PackElend sorry for the late reply on this.

gonna work, regardless the yaml version

Indeed, docker-compose CLI supports v2 and v3 and I was recommending you use this.

Portainer cannot do docker-compose my_remote_standalone_container up if yaml is >v2

As explained previously we use a library called libcompose which doesn't support v3, so we aren't actually executing docker compose on the CLI or directly using the docker-compose binary. Portainer is able to deploy stacks with v3 because we use a different binary (docker binary) for this, which would execute the command docker stack deploy my_remote_stack so we aren't interacting with the CLI directly from Portainer.

I am not sure how easy it would be to switch the binary we use for compose to support v3, IIRC from internal discussions its likely to be a lot of work.

If you want to want to switch from compose to swarm a good point to start is Docker Swarm – Single Node, inst' it?

I will give this a read when I have more time and get back to you. But generally in my experience it is not too difficult to switch, you may have to modify your yml a small amount :)

@deviantony
Copy link
Member

@deviantony deviantony commented Mar 11, 2020

Now that Docker is actually working on rewriting docker-compose in Go, this is probably something that we'll be able to reconsider.

See https://github.com/docker/roadmap/projects/1

@itsconquest
Copy link
Member

@itsconquest itsconquest commented Mar 11, 2020

this is probably something that we'll be able to reconsider

Fingers crossed!! 🤞

@itsconquest
Copy link
Member

@itsconquest itsconquest commented Apr 24, 2020

I think its pretty clear that we need to drop libcompose, its missing a lot of compose features and there have been a lot of bugs with it that I have been opened against our repo. As such, I have opened a 'master issue' to link all of these together and I have pinned it.

I'm gonna close this and refer conversation to this issue -> #3750

@frakman1
Copy link

@frakman1 frakman1 commented Jan 31, 2021

A simple docker swarm init command will allow you to create a simple one node Swarm cluster and give you access to Swarm features.

Once your host is in Swarm mode, Portainer will auto-detect it and you'll be able to use Compose v3+ to deploy stacks.

This didn't work for me. I ran docker swarm init on the host and got this output:

# docker swarm init
Swarm initialized: current node (k1syhvup1fvg2jsdf9bjks7od) is now a manager.

To add a worker to this swarm, run the following command:

    docker swarm join --token SWMTKN-1-45gwdwo07hab28fabspascyc5sntncu55m275vjildpnh0dgea-4bdhlaqmb8tfw37ozz8so5vge 192.168.122.1:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

# docker swarm join --token SWMTKN-1-45gwdwo07hab28fabspascyc5sntncu55m275vjildpnh0dgea-4bdhlaqmb8tfw37ozz8so5vge 192.168.122.1:2377
Error response from daemon: This node is already part of a swarm. Use "docker swarm leave" to leave this swarm and join another one.

Weird that using their command to add a worker didn't work. Anyway, I went to Portainer and tried to run a simple wordpress example docker-compose stack that uses v 3.3 and it failed:

image

I am using the portainer docker from: https://registry.hub.docker.com/r/portainer/portainer/

UPDATE

OK, I followed the link to the other conversation and found the beta release that supports newer versions of docker-compose. I ended up using this tag for portainer: portainerci/portainer:2.1 and it works now

@MicahZoltu
Copy link

@MicahZoltu MicahZoltu commented Jan 31, 2021

Weird that using their command to add a worker didn't work.

When you did docker swarm init it created a new swarm with that host joined as the manager. Since you are already joined to the swarm as a manager, you cannot join again. The docker swarm join ... command should be run on other hosts in the cluster. Since you are running a single node cluster, you don't have to do any joining.

Anyway, I went to Portainer and tried to run a simple wordpress example docker-compose stack that uses v 3.3 and it failed:

I wonder if you need to restart portainer after doing the swarm init command before it notices that it is a swarm?

@frakman1
Copy link

@frakman1 frakman1 commented Jan 31, 2021

Thank you for explaining the swam/node thing. I get it now.

I restarted the container and got the same error.
I posted an update. I went to the beta release (portainerci/portainer:2.1) and that seems to work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/stack-creation area/stacks dep/libcompose kind/enhancement
Projects
None yet
Development

No branches or pull requests