Compose v2: Podman is unable to execute pods while running services via docker-compose

[kriansa]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

We can't run docker-compose run on a service which is currently running (via docker-compose start/up). It fails with the following error message:

ERROR: for test  Cannot create container for service test: bad parameter: Link is not supported
ERROR: Encountered errors while bringing up the project.

Steps to reproduce the issue:

1. Create a docker-compose.yml file with the following content:

services:
  test:
    image: docker.io/library/alpine:latest
    command: sleep 1000

2. Run docker-compose up -d

3. Run docker-compose run --rm test sh

Describe the results you received:

ERROR: for test  Cannot create container for service test: bad parameter: Link is not supported
ERROR: Encountered errors while bringing up the project.

Describe the results you expected:

I expect to have a running shell on the container

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

podman version 3.3.1

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.22.3
  cgroupControllers: []
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.0.29-1
    path: /usr/bin/conmon
    version: 'conmon version 2.0.29, commit: 7e6de6678f6ed8a18661e1d5721b81ccee293b9b'
  cpus: 12
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  hostname: Arch-Home.lan.cx
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 10000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 10000
      size: 65536
  kernel: 5.14.7-arch1-1
  linkmode: dynamic
  memFree: 9888661504
  memTotal: 16666849280
  ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 1.0-2
    path: /usr/bin/crun
    version: |-
      crun version 1.0
      commit: 139dc6971e2f1d931af520188763e984d6cdfbf8
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns is owned by slirp4netns 1.1.12-1
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 0
  swapTotal: 0
  uptime: 26m 49.71s
registries: {}
store:
  configFile: /home/dpereira/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: /usr/bin/fuse-overlayfs is owned by fuse-overlayfs 1.7.1-1
      Version: |-
        fusermount3 version: 3.10.5
        fuse-overlayfs: version 1.7.1
        FUSE library version 3.10.5
        using FUSE kernel interface version 7.31
  graphRoot: /home/dpereira/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 0
  runRoot: /run/user/1000/containers
  volumePath: /home/dpereira/.local/share/containers/storage/volumes
version:
  APIVersion: 3.3.1
  Built: 1630517266
  BuiltTime: Wed Sep  1 14:27:46 2021
  GitCommit: 4c5283fabff2de5145838f1847a5a7b2b1fbc0a5-dirty
  GoVersion: go1.17
  OsArch: linux/amd64
  Version: 3.3.1

Package info (e.g. output of rpm -q podman or apt list podman):

❯ pacman -Qi podman
Name            : podman
Version         : 3.3.1-1
Description     : Tool and library for running OCI-based containers in pods
Architecture    : x86_64
URL             : https://github.com/containers/podman
Licenses        : Apache
Groups          : None
Provides        : None
Depends On      : cni-plugins  conmon  containers-common  device-mapper  iptables
                  libseccomp  crun  slirp4netns  libsystemd  fuse-overlayfs
                  libgpgme.so=11-64
Optional Deps   : podman-docker: for Docker-compatible CLI [installed]
                  btrfs-progs: support btrfs backend devices [installed]
                  catatonit: --init flag support
                  crun: support for unified cgroupsv2 [installed]
Required By     : podman-docker
Optional For    : None
Conflicts With  : None
Replaces        : None
Installed Size  : 71.85 MiB
Packager        : Morten Linderud <foxboron@archlinux.org>
Build Date      : Wed 01 Sep 2021 02:27:46 PM -03
Install Date    : Mon 20 Sep 2021 03:30:04 PM -03
Install Reason  : Explicitly installed
Install Script  : No
Validated By    : Signature

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

Physical/my own computer

[Luap99]

Podman does not support the link feature. Docker deprecated this feature years ago, https://docs.docker.com/network/links/.

[kriansa]

@Luap99 I understand that, but as a docker-compose user I'm not specifically trying to link a container network, instead I'm simply trying to execute a container. It might do all sorts of stuff behind the scenes, even deprecated stuff like that one -- but I don't know that as I haven't checked docker-compose codebase.

My point is more of a docker-compose user trying to replace docker by podman: although recent efforts made this possible, it's not at a state that we can simply start suggesting podman as a drop-in replacement for even basic workflows IMHO.

If we diagnose that this is not a podman fault and a fix for that is not feasible, and there's no workaround for it, then I think we should document that behavior for newer users. What do you think?

[rhatdan]

We are not going to support Linux, so perhaps every docker-compose ever written will not work with Podman. Can you script be fixed to not rely on link?

[Luap99]

I think this is the first time someone brings this up, I never heard of docker-compose run before and I do not know what it is doing but apparently it is trying to use link.
Can you use docker-compose exec instead?

[mheon]

Might be worth a brief investigation of links, to see if we can "fake" them by sharing namespaces - I wouldn't want to expose this by the command line, but having something for Docker APIv2 might be worthwhile.

[kriansa]

@Luap99 Sure, I can use exec instead, I just need to adapt my scripts but it's perfectly doable - thanks for the tip!

I think this is the first time someone brings this up, I never heard of docker-compose run before and I do not know what it is doing but apparently it is trying to use link.

Tbh, that's news to me. I thought run was really common among docker-compose users, but now it makes sense how I couldn't find anything about this issue on the web for instance. Maybe my use case is too narrow and probably the best solution is a simple text entry on the shortcomings section of docker-compose compatibility?

[mheon]

From a brief investigation of linking, there is a large amount of environment-variable stuff that seems extraneous, but the core of it appears to be adding an entry to /etc/hosts pointing to the IP of the linked container from a given name... That seems absolutely trivial to implement for Docker APIv2? I wouldn't want to do anything else related, but we could stop returning an error.

[Luap99]

Yeah I also looked at this but the container create config from docker-compose looks weird. It sends "Links": ["", ":test", ":test_test_1"],, according to docker api documentation the format should be container_name:alias. So why does docker-compose does not set the container name?

[mheon]

Alright, that's honestly bizarre...

[tobley]

@kriansa might want to try running it with COMPOSE_INTERACTIVE_NO_CLI=1 docker-compose run ... to see if that helps.

I couldn't get docker-compose exec to work and found that the exec and run commands are invoked differently that other compose commands. The docker-compose source shows that the docker executable is invoked directly since variable use_cli defaults to true. You can override it with COMPOSE_INTERACTIVE_NO_CLI=1 described here.

I don't have docker installed in my environment (WSL2 with Ubuntu 20.04) and I think docker-compose is supposed to throw an error for the missing executable, but I never saw anything.

This could really get wonky if you have docker and podman installed. If you run DOCKER_HOST=unix:///var/run/podman/podman.sock docker-compose run ... and expect it to be talking to podman, you're wrong. What's happening under the covers is the docker binary getting invoked directly.

[kriansa]

@tobley Thanks for digging into that. I tried that option but it really doesn't help, and I get the same error. As I side note, I also only have podman installed and docker is actually the same podman binary (and that's being provided by podman-docker package on Arch Linux)

[kriansa]

Well, I've got some good and bad news.

It seems that the recently released Compose V2 (rebranded docker-compose) does work as expected (at least run's out of started services). 😃

However, I think that native support for podman is now broken by default. That's because `docker-compose` for Linux rely on a [deprecated feature of Docker](https://github.com/docker/cli/blob/master/docs/deprecated.md#cli-plugins-support) called `Docker CLI Plugin` and this is not supported by `podman` as of today (see: #5802). In practical terms, it means we cannot simply call `docker-compose` from the CLI anymore as there's no binary installed to the `/usr/bin`, instead it is shipped as a single binary and is supposed to be placed at `$HOME/.docker/cli-plugins/` folder.

This can be quickly worked around by a simple bash alias, and that's how I got docker-compose running after the upgrade:

alias docker-compose="/usr/lib/docker/cli-plugins/docker-compose compose"

Update: Update v2.0.1 fixed it and we can use docker-compose as a standalone binary.

Moreover, now I've got an issue with docker-compose build: it's simply not working at all. It fails with the following error message, and I wasn't able to find out exactly why:

failed to get status: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing unable to upgrade to h2c, received 404"

tl;dr: run issue is fixed, but now we've got new problems with both "native" integration as well as the build subcommand. Should we still pursue fixing it for compose v1, or should we focus on the newer issues?

[rhatdan]

Nice work, we could easily add support for the alias into podman-docker package.

[kriansa]

Newer compose release did add support for running docker-compose as a standalone package, so no need for adding that bash alias anymore.

[FallenWarrior2k]

Another issue I've found: If there's VOLUME declarations in an image, manually mounting over those to actually preserve the data in a named volume will throw an Error response from daemon: fill out specgen: /config: duplicate mount destination.

To reproduce (I did this in a fresh Arch VM to ensure no outside influence):

1. Install podman-docker and docker-compose
2. If running from Archiso or comparable live image, you probably have to change the storage driver to VFS in storage.conf because overlay on overlay doesn't work.
3. Set up docker.io in unqualified-seach-registries in registries.conf, otherwise Compose will barf even if you explicitly write docker.io/library/... in docker-compose.yml
4. Try to run this simple Caddy setup

version: "3.5"
services:
  caddy:
    image: caddy:2.4.5
    volumes:
      - config:/config
      - data:/data
volumes:
  config:
  data:

This does not happen if you try to podman run -v caddy_config:/config -v caddy_data:/data caddy:2.4.5.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[vrothberg]

I think this has been resolved in the meantime. Also note that images will resolve to docker.io (compat API) starting with Podman 4.0 (to be released early next year)

[dszakallas]

It seems I am running into the same issue reported by @FallenWarrior2k when using rootless podman with docker-compose. I am using the basic wordpress example from docker-compose's website:

$ podman --version
podman version 4.0.3

wp.yaml:

version: "3.9"
    
services:
  db:
    image: mysql:5.7
    volumes:
      - db_data:/var/lib/mysql
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: somewordpress
      MYSQL_DATABASE: wordpress
      MYSQL_USER: wordpress
      MYSQL_PASSWORD: wordpress
    
  wordpress:
    depends_on:
      - db
    image: wordpress:latest
    volumes:
      - wordpress_data:/var/www/html
    ports:
      - "8000:80"
    restart: always
    environment:
      WORDPRESS_DB_HOST: db
      WORDPRESS_DB_USER: wordpress
      WORDPRESS_DB_PASSWORD: wordpress
      WORDPRESS_DB_NAME: wordpress
volumes:
  db_data: {}
  wordpress_data: {}

This results in the following error:

docker-compose up
Error response from daemon: fill out specgen: /var/lib/mysql: duplicate mount destination

However, I don't think it has much to do with an existing volume declaration in this case. I tried with busybox and still got the same error:

version: "3.9"
    
services:
  busybox:
    image: busybox:latest
    volumes:
      - data:/data
    entrypoint:
      - sh
    command:
      - "-c"
      - "sleep 1000"
    
volumes:
  data: {}

Error response from daemon: fill out specgen: /data: duplicate mount destination

I also tried it out with podman-compose, that works fine.

[tdgroot]

Having the exact same problem as @dszakallas reports. I'm running Podman v4.3.1.

[tdgroot]

Can verify that upgrading from docker-compose 1.9.x to 2.14.x solved this problem for me.