docker-compose 2.0.0 is no longer written in Python, thus breaking the docker_compose module

[felixfontein]

SUMMARY

I guess we have to mention in the requirements of the module that it needs docker-compose < 2.0.0. Potentially we either have to adjust the module to also work with docker-compose 2.0.0 and newer, or we have to deprecate it.

ISSUE TYPE

• Bug Report

COMPONENT NAME

docker_compose

[felixfontein]

CC @aminvakil thanks for reporting this! (BTW the PR you mentioned is unrelated to this.)

[felixfontein]

Adding a warning and the < 2.0.0 requirement in #217.

[dmp1ce]

Should I expect that this module will be updated to support version 2.0.0 and greater of Compose? I'm trying to figure out if I should rewrite my docker-compose sections of my playbook or just wait for an update.

[felixfontein]

@dmp1ce that depends on whether someone wants to rewrite the docker_compose module in such a way that it will simply work with the CLI. Right now the module depends on importing the Python code of docker-compose itself, which only works for docker-compose < 2.0.0 since for 2.0.0 docker-compose was rewritten in Go.

[libricoleur]

This has become a problem on distros that already updated to compose v2, such as Arch Linux. Has anyone started working on a port of the module?

In the meantime I installed v1 from pip in a virtualenv and pointed the PYTHONPATH environment variable to it.

[felixfontein]

I'm not aware of anyone working on this.

[dmp1ce]

@libricoleur I created my own tasks which copy the docker-compose.yml to the machine and then run docker-compose. It looks something like this:

- name: Loop through starting Docker services (manually)
  block:
    - name: Copy docker-compose templates for each service
      ansible.builtin.template:
        src: files/docker_services/{{ item.project_name }}.yml.j2
        dest: /home/maintenance/docker_services/{{ item.project_name }}/docker-compose.yml
        backup: yes
      loop: "{{ docker_services }}"

    - name: "Update docker service image"
      command:
        chdir: /home/maintenance/docker_services/{{ item.project_name }}
        cmd: docker-compose pull
      loop: "{{ docker_services }}"
    - name: "Start docker service"
      command:
        chdir: /home/maintenance/docker_services/{{ item.project_name }}
        cmd: docker-compose up -d
      loop: "{{ docker_services }}"
  when: docker_services is defined

I have a feeling it might be awhile before this issue gets resolved, so I hope this work around helps some people.

[pgassmann]

related to Future of community.docker in community.docker #364

[craph]

Any updates on this subject ?

[felixfontein]

Unfortunately nobody wanted to work on this so far, so no, there are no news.

[Sid-Sun]

Hi, I am working on create a new compose v2 module by using python on whales (thanks to @gotmax23 for pointing out python-on-whales in #364) - is there any interest in implementing that in community.docker? Thanks

[felixfontein]

Since folks have been repeatedly asking for this: yes, there is an interest in that.

(I personally don't think that using python-on-whales is a good idea, since you can also simply run the docker-compose binary directly without using a third-party library for that and forcing all users of the module to install that library. But since I'm neither writing nor maintaining that module, I don't care that much :) )

[felixfontein]

For everone interested, there's now a PR: #540

[goldyfruit]

@felixfontein thanks for this PR, I hope to see it merge soon.
Just had a limitation due to docker-compose. c.f: docker/compose#9505

[septatrix]

Funnily enough podman-compose is what I have now been using with --podman-path docker and --podman-args compose such that in invokes docker compose instead of podman :D

[ptr727]

Apologies for being a bit off-topic, please redirect as appropriate.
I tried using compose, failed, found that v < 2.0, found this.

I just need rudimentary inter-container startup/restart behavior, that will be honored by portainer stacks and and watchtower updates.
Other than using compose, is there another way to specify inter-instance dependencies using the normal container module options?

[cdauth]

This is driving me crazy. Has anyone managed to get this to run?

I have installed docker-compose to a virtual env using pip, but Ansible keeps telling me that there is No module named 'compose'. I've tried adding the virtual env to my PATH using the environment property in my playbook configuration, but it does not help.

[felixfontein]

@cdauth you have to point ansible to the Python interpreter of that venv - then it should work. (At least for the docker_compose tasks - ansible_python_interpreter should help here.)

[cdauth]

Thanks! This is how I installed the pip module on the server:

- pip:
    name: docker-compose
    virtualenv: /usr/local/pip

And this is how I configured my task that uses the docker_compose module:

    vars:
        ansible_python_interpreter: /usr/local/pip/bin/python

You can also add that latter configuration to the playbook itself, but that would mean that the interpreter has to be commented out the first time the machine is bootstrapped (because the interpreter does not exist yet the first time the playbook is run).

[felixfontein]

@cdauth just to clarify - do you mean that it now works (with what you pasted)? Or do you still have trouble using the module even though you provide ansible_python_interpreter?

[felixfontein]

@cdauth great! :) Also thanks for documenting it here, this will surely help some others!

[maxkratz]

The default version of Docker compose on the latest stable Debian (12, bookworm) is also >=2. Thus, the docker_compos module is also broken on Debian since June.

[felixfontein]

You can always install docker-compose < 2 by using pip. And if someone produces a mergable version of a module which supports version 2 (i.e. it supports basic functionality and has tests that pass), I will happily merge that.

[maxkratz]

You can always install docker-compose < 2 by using pip.

Sure, but docker-compose v1 is EOL as of July this year:

From July 2023 Compose V1 stopped receiving updates. It’s also no longer available in new releases of Docker Desktop.

Reference: https://docs.docker.com/compose/install/linux/

[felixfontein]

It has already been effectively EOL since May 2021; the last release was 1.29.2 on May 10th, 2019: https://pypi.org/project/docker-compose/

[metajiji]

Hello everyone, what if we use the --format json feature? For example:

$ docker-compose ps --help

Usage:  docker compose ps [OPTIONS] [SERVICE...]

List containers

Options:
      --format string        Format the output. Values: [table | json] (default "table")

It would be nice to add these flags to the docker-compose project in the other commands to retrieve the state, for example, for the up command, like this

docker-compose up --format json

And docker-compose should return JSON with the state of all running containers.

[truongvanluan]

This is what I use:

`

• name: Stop current apps
  command:
  cmd: sudo docker compose down

• name: Start new apps
  command:
  cmd: sudo docker compose up -d
  `

[tiagogoncalves-7egend]

Just use the following and save your time:

• name: Start services
  shell: docker-compose up -d

[HeikoBoettger]

You can always install docker-compose < 2 by using pip. And if someone produces a mergable version of a module which supports version 2 (i.e. it supports basic functionality and has tests that pass), I will happily merge that.

I current have a qnap device which only come with docker compose v2. Installing docker-compose failed due to an issue with PyYAML6 and Cython 3 incombination with python 3.10 which seems to work in python 3.12. However the qnap has not yet released python3.12.

In other words docker-compose v1 is no longer supported on this device.

I don't want to complain but it means it's no longer always guaranteed you can install docker compose v1 since it's EOL.

[jakubgs]

@HeikoBoettger I believe you can get the old compose package to install by pinning Cython and PyYAML to older versions.

See:

• Error installing Pyyaml==5.4, Cython_sources yaml/pyyaml#724
• https://github.com/status-im/infra-role-bootstrap-linux/blob/4af21635f7b6edee139cbf26442ce6a680fd6eb3/tasks/packages/pyyaml_fix.yml#L7

[HeikoBoettger]

@jakubgs Thanks, I guess that will work. How can I get ansible to work with a specific python-env when executing each task? It rather complicated to do something like this on the preinstalled python3 package on a qnap device.

[jakubgs]

You don't need to use Ansible, that's just my use case, buy you can pin versions by calling pip directly. Also, I never used QNAP.

[rafalkk]

Funnily enough podman-compose is what I have now been using with --podman-path docker and --podman-args compose such that in invokes docker compose instead of podman :D

@septatrix Hi, could you please specify how to run podman-compose using Ansible? Are you using the Ansible shell module? I cannot find a podman-compose Ansible module.

[felixfontein]

I created a PR for adding (another instance of) a docker_compose_v2 module: #739. It is based on the Docker CLI tools I added for the new docker_image_build module, and re-uses the docker compose output parsing code from #586.

It does not implement all features from the docker_compose module, but sticks to the most important ones (IMO). It does support check mode (and needs --dry-run for that, i.e. docker compose's version must be 2.18.0 or later).

[felixfontein]

There's now a new release, 3.6.0-b1 (changelog), including the module. Assuming there aren't big problems showing up, I plan to release the final 3.6.0 version in the next 2-3 weeks (so it will appear in Ansible 9.2.0). Please try out the pre-release and provide feedback, especially if something doesn't work as expected, or if you're missing something important you really need. (For example right now, there's no way to explicitly pull or build.)

[HeikoBoettger]

Thank you very much for resolving this issue.

[felixfontein]

I created a second prerelease 3.6.0-b2 (changelog), including a new module docker_compose_v2_pull, and a new option pull to specify the pull policy.

[felixfontein]

I created a release candidata 3.6.0-rc1 (changelog - unfortunately due to a change on GitHub this no longer renders...); there are a few new features, like being able to specify which services to start/stop/..., and to pass the --build/--no-build flags to docker compose up. I plan to do the final release on Sunday or Monday.

[felixfontein]

Version 3.6.0 is finally out!

[dodgex]

Nice, I just came back to my ansible stuff and checked for this! Luckily i came to this issue after first checking the docs as they are still on a 3.4 release.

While I found docs on the Ansible Galaxy Site , I wonder when the other docs gets updated too. As they are what you find when you search for "ansible docker compose", no link to the galaxy site found on google.

[dodgex]

Nevermind. After posting this comment I found the notes about documentation in the readme.

For anyone as stupid as me, here is the proper docs for the 3.7 release on the Ansible Docs site: https://docs.ansible.com/ansible/devel/collections/community/docker/

[felixfontein]

The docs on https://docs.ansible.com/ansible/latest/ will be updated on the next Ansible release, which will likely be on next Tuesday.