Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docker-compose exec container error #11081

Closed
hholst80 opened this issue Jul 29, 2021 · 34 comments
Closed

docker-compose exec container error #11081

hholst80 opened this issue Jul 29, 2021 · 34 comments
Labels
docker-composev2 locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.

Comments

@hholst80
Copy link

Description

docker-compose exec won't work fails with

Error: cannot start container c50ad95f9a3986610e46a09cb0501b07bf742eb88faf79126278eb3b0db71462: fork/exec /usr/bin/conmon: permission denied

Steps to reproduce the issue:

  1. docker-compose up -d

  2. docker-compose exec container sh

Describe the results you received:

asdf $ cat docker-compose.yml
version: '2'
services:
  ubuntu:
    image: ubuntu:latest
    command: sleep inf
asdf $ docker-compose up -d 
...
asdf $ podman exec -it asdf_ubuntu_1 sh
#  exit
asdf $ docker-compose exec ubuntu sh
Error: cannot start container ef65d9bbc64c32a3fa40b794d5b246555e9e8e6825d09f6d3341bbb9d8b53c09: fork/exec /usr/bin/conmon: permission denied

Output of podman version:

Version:      3.2.3
API Version:  3.2.3
Go Version:   go1.16.5
Built:        Fri Jul 16 22:33:32 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.21.3
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.29-2.fc34.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.29, commit: '
  cpus: 8
  distribution:
    distribution: fedora
    version: "34"
  eventLogger: journald
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 10000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 10000
      size: 65536
  kernel: 5.12.13-300.fc34.x86_64
  linkmode: dynamic
  memFree: 2132680704
  memTotal: 33525682176
  ociRuntime:
    name: crun
    package: crun-0.20.1-1.fc34.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.20.1
      commit: 0d42f1109fd73548f44b01b3e84d04a279e99d2e
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.9-1.fc34.x86_64
    version: |-
      slirp4netns version 1.1.8+dev
      commit: 6dc0186e020232ae1a6fcc1f7afbc3ea02fd3876
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 25309732864
  swapTotal: 25425862656
  uptime: 626h 42m 29.25s (Approximately 26.08 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/om/.config/containers/storage.conf
  containerStore:
    number: 15
    paused: 0
    running: 14
    stopped: 1
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/om/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 168
  runRoot: /run/user/1000/containers
  volumePath: /home/om/.local/share/containers/storage/volumes
version:
  APIVersion: 3.2.3
  Built: 1626467612
  BuiltTime: Fri Jul 16 22:33:32 2021
  GitCommit: ""
  GoVersion: go1.16.5
  OsArch: linux/amd64
  Version: 3.2.3

Package info (e.g. output of rpm -q podman or apt list podman):

podman-3.2.3-1.fc34.x86_64

Minimal example docker-compose.yml

version: '2'
services:
  ubuntu:
    image: ubuntu:latest
    command: sleep inf
@mheon
Copy link
Member

mheon commented Jul 30, 2021

Can you provide more details on your environment - how is Podman being run (root or rootless, from a systemd service or directly, installed from a package or built from source?).

Can you also do an ls -al /usr/bin/conmon and include the output of that?

@hholst80
Copy link
Author

hholst80 commented Aug 1, 2021

Fedora 34 vanilla install; podman rootless via systemd as a user service; regular 0755 root:root flags.

@mheon
Copy link
Member

mheon commented Aug 2, 2021

Can you run containers with the Podman CLI (podman run -t -i alpine ls as an example)? Or does that error too?

@hholst80
Copy link
Author

hholst80 commented Aug 3, 2021

The stack was started with podman, yes. I just cannot connect using docker-compose exec to the running containers. You see in the debug output that podman is indeed running containers.

@mheon
Copy link
Member

mheon commented Aug 4, 2021

Did you try running containers without using compose? Can you try a podman run ...; podman exec... combination without Compose in the picture?

@hholst80
Copy link
Author

hholst80 commented Aug 5, 2021

Yes, and that works. I can also podman exec into a container started by compose, I just cannot exec from docker-compose.

@mheon
Copy link
Member

mheon commented Aug 5, 2021

OK. I can't replicate this locally, so this is probably environment-related.

Can you start the Podman service manually with debug-level logging enab;ed (podman system service -t 0 --log-level=debug) and then run Compose against it and provide the logs that Podman prints? It should tell us what commands Compose are sending that are proving problematic.

@hholst80
Copy link
Author

hholst80 commented Aug 6, 2021

What about cgroup kernel settings?

@mheon
Copy link
Member

mheon commented Aug 6, 2021 via email

@github-actions
Copy link

github-actions bot commented Sep 6, 2021

A friendly reminder that this issue had no activity for 30 days.

@rhatdan
Copy link
Member

rhatdan commented Sep 7, 2021

@hholst80 Did you figure out any additional information on this?

@vrothberg
Copy link
Member

Friendly ping

@hholst80
Copy link
Author

hholst80 commented Sep 23, 2021

"Did you find out any additional information about this". No, I shelved the podman project for now as Docker engine has working support for this although only for a single user and only for Linux.

@vrothberg
Copy link
Member

vrothberg commented Sep 23, 2021

Thanks, @hholst80! I am closing this issue for now since we'd need more data. Feel free to reopen or comment in the future.

@DrSensor
Copy link

DrSensor commented Nov 3, 2021

I also run the same issue. Seems like exec command is not yet implemented in podman-compose.

$ podman-compose --help
usage: podman-compose [-h] [-f FILE] [-p PROJECT_NAME]
                      [--podman-path PODMAN_PATH] [--no-ansi] [--no-cleanup]
                      [--dry-run]
                      [-t {1pod,1podfw,hostnet,cntnet,publishall,identity}]
                      {pull,push,build,up,down,run,start,stop,restart} ...

optional arguments:
  -h, --help            show this help message and exit
  -f FILE, --file FILE  Specify an alternate compose file (default: docker-
                        compose.yml)
  -p PROJECT_NAME, --project-name PROJECT_NAME
                        Specify an alternate project name (default: directory
                        name)
  --podman-path PODMAN_PATH
                        Specify an alternate path to podman (default: use
                        location in $PATH variable)
  --no-ansi             Do not print ANSI control characters
  --no-cleanup          Do not stop and remove existing pod & containers
  --dry-run             No action; perform a simulation of commands
  -t {1pod,1podfw,hostnet,cntnet,publishall,identity}, --transform_policy {1pod,1podfw,hostnet,cntnet,publishall,identity}
                        how to translate docker compose to podman
                        [1pod|hostnet|accurate]

command:
  {pull,push,build,up,down,run,start,stop,restart}
    pull                pull stack images
    push                push stack images
    build               build stack images
    up                  Create and start the entire stack or some of its
                        services
    down                tear down entire stack
    run                 create a container similar to a service to run a one-
                        off command
    start               start specific services
    stop                stop specific services
    restart             restart specific services
$ podman --version
podman version 3.4.0

related to containers/podman-compose#71

EDIT: I got it works by replacing podman-compose with docker-compose (without docker) then set the DOCKER_HOST to podman.sock.

After removing podman-compose then install docker-compose.

systemctl --user start podman.service
export DOCKER_HOST="unix://$XDG_RUNTIME_DIR/podman/podman.sock"

--user flag is needed only if you want to run it in rootless mode

@rhatdan
Copy link
Member

rhatdan commented Nov 3, 2021

podman-compose is not part of Podman, it is a separate project.
Podman supports both podman-compose and docker-compose, and perhaps docker-compose has exec.

@gdetrez
Copy link

gdetrez commented Dec 23, 2021

I have a very similar issue. Fresh install of Fedora 35 with both podman and docker-compose installed with dnf.

I tried to follow the steps recommended by @mheon, and here are the logs being printed when I do docker-compose exec ubuntu ls (I used the minimal compose file from the original issue):


DEBU[0043] IdleTracker:new 6m+1h/16t connection(s)       X-Reference-Id=0xc00013d5f8
DEBU[0043] IdleTracker:active 6m+1h/17t connection(s)    X-Reference-Id=0xc00013d5f8
@ - - [23/Dec/2021:15:26:32 +0100] "GET /version HTTP/1.1" 200 885 "" "docker-compose/1.29.2 docker-py/5.0.2 Linux/5.14.10-300.fc35.x86_64"
DEBU[0043] IdleTracker:idle 7m+1h/17t connection(s)      X-Reference-Id=0xc00013d5f8
DEBU[0043] IdleTracker:new 7m+1h/17t connection(s)       X-Reference-Id=0xc00013d618
DEBU[0043] IdleTracker:active 7m+1h/18t connection(s)    X-Reference-Id=0xc00013d618
INFO[0043] Request Failed(Not Found): unable to find network configuration for 20211223podmanexec_default: network not found
@ - - [23/Dec/2021:15:26:32 +0100] "GET /v1.40/networks/20211223podmanexec_default HTTP/1.1" 404 144 "" "docker-compose/1.29.2 docker-py/5.0.2 Linux/5.14.10-300.fc35.x86_64"
DEBU[0043] IdleTracker:idle 8m+1h/18t connection(s)      X-Reference-Id=0xc00013d618
DEBU[0043] IdleTracker:new 8m+1h/18t connection(s)       X-Reference-Id=0xc00013d780
DEBU[0043] IdleTracker:active 8m+1h/19t connection(s)    X-Reference-Id=0xc00013d780
DEBU[0043] Error parsing cgroup: expected 3 fields but got 1: /proc/120633/cgroup
@ - - [23/Dec/2021:15:26:32 +0100] "GET /v1.40/containers/json?limit=-1&all=0&size=0&trunc_cmd=0&filters=%7B%22label%22%3A+%5B%22com.docker.compose.project%3D20211223-podman-exec%22%2C+%22com.docker.compose.service%3Dubuntu%22%2C+%22com.docker.compose.oneoff%3DFalse%22%2C+%22com.docker.compose.container-number%3D1%22%5D%7D HTTP/1.1" 200 1282 "" "docker-compose/1.29.2 docker-py/5.0.2 Linux/5.14.10-300.fc35.x86_64"
DEBU[0043] IdleTracker:idle 9m+1h/19t connection(s)      X-Reference-Id=0xc00013d780
DEBU[0043] IdleTracker:closed 9m+1h/19t connection(s)    X-Reference-Id=0xc00013d618
DEBU[0043] IdleTracker:closed 8m+1h/19t connection(s)    X-Reference-Id=0xc00013d5f8
DEBU[0043] IdleTracker:closed 7m+1h/19t connection(s)    X-Reference-Id=0xc00013d780

Could this 👇 be the issue?

Error parsing cgroup: expected 3 fields but got 1: /proc/120633/cgroup

@mheon
Copy link
Member

mheon commented Dec 23, 2021

Potentially?

Is the error exactly the same as the one here (permission denied forking conmon)?

Anything else in the logs? Dmesg?

@gdetrez
Copy link

gdetrez commented Dec 23, 2021

I'm not sure. I don't get an error message when running docker-compose exec, it just exits with return code 1.
If I add --verbose --log-level debug to docker-compose, I get this:

❯❯❯ docker-compose --log-level DEBUG --verbose exec ubuntu ls
compose.config.config.find: Using configuration files: ./docker-compose.yml
compose.cli.docker_client.get_client: docker-compose version 1.29.2, build unknown
docker-py version: 5.0.2
CPython version: 3.10.1
OpenSSL version: OpenSSL 1.1.1l  FIPS 24 Aug 2021
compose.cli.docker_client.get_client: Docker base_url: http+docker://localhost
compose.cli.docker_client.get_client: Docker version: Platform={'Name': 'linux/amd64/fedora-35'}, Components=[{'Name': 'Podman Engine', 'Version': '3.4.4', 'Details': {'APIVersion': '3.4.4', 'Arch': 'amd64', 'BuildTime': '2021-12-08T22:45:07+01:00', 'Experimental': 'false', 'GitCommit': '', 'GoVersion': 'go1.16.8', 'KernelVersion': '5.14.10-300.fc35.x86_64', 'MinAPIVersion': '3.1.0', 'Os': 'linux'}}, {'Name': 'Conmon', 'Version': 'conmon version 2.0.30, commit: ', 'Details': {'Package': 'conmon-2.0.30-2.fc35.x86_64'}}, {'Name': 'OCI Runtime (crun)', 'Version': 'crun version 1.3\ncommit: 8e5757a4e68590326dafe8a8b1b4a584b10a1370\nspec: 1.0.0\n+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL', 'Details': {'Package': 'crun-1.3-1.fc35.x86_64'}}], Version=3.4.4, ApiVersion=1.40, MinAPIVersion=1.24, GitCommit=, GoVersion=go1.16.8, Os=linux, Arch=amd64, KernelVersion=5.14.10-300.fc35.x86_64, BuildTime=2021-12-08T22:45:07+01:00
compose.cli.verbose_proxy.proxy_callable: docker inspect_network <- ('20211223podmanexec_default')
compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=20211223-podman-exec', 'com.docker.compose.service=ubuntu', 'com.docker.compose.oneoff=False', 'com.docker.compose.container-number=1']})
compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 1 items)

@gdetrez
Copy link

gdetrez commented Dec 23, 2021

Log is empty (I assume you mean the output of docker-compose logs) and no message appear in the output of dmesg when running docker-compose exec ....

@gdetrez
Copy link

gdetrez commented Jan 12, 2022

@mheon Any tips on how I can further debug the issue? Or should I open a different issue to track this since I don't get the same error message?

@mheon
Copy link
Member

mheon commented Jan 12, 2022

Full logs from a Podman API server (podman system service --log-level=debug -t=0) with Compose attached to it would help a lot.

@gdetrez
Copy link

gdetrez commented Jan 12, 2022

@mheon here's a log for docker-compose up and docker-compose exec: https://gist.github.com/gdetrez/75ea01a1133cc905c33278030125ea44
Tell me if you need more.

@mheon
Copy link
Member

mheon commented Jan 12, 2022

What Podman version are you on? I swear we fixed this (the cgroup parsing issue) recently. If not, I think we can zero in on that as the most likely culprit, nothing else jumps out - the network configuration issues are a bit worrying but are tied to other requests that seem to succeed.

@gdetrez
Copy link

gdetrez commented Jan 13, 2022

I'm on 3.4.4, installed from fedora's repositories.

❯❯❯ podman --version
podman version 3.4.4

@mheon
Copy link
Member

mheon commented Jan 13, 2022

Alright, that's definitely new enough that it should have been fixed... I'm going to reopen this bug given all of this, we have a clear issue identified.

@mheon mheon reopened this Jan 13, 2022
@baude
Copy link
Member

baude commented Jan 17, 2022

@mheon couple of points here ... with docker-compose exec I now get a return code of 0 but also no output ... and now I only see the cgroups error when using podman exec.

@github-actions
Copy link

A friendly reminder that this issue had no activity for 30 days.

@github-actions
Copy link

A friendly reminder that this issue had no activity for 30 days.

@rhatdan
Copy link
Member

rhatdan commented Mar 23, 2022

@mheon @baude @gdetrez Is this fixed in podman 4.0?

@mheon
Copy link
Member

mheon commented Mar 23, 2022

Don't believe so

@github-actions
Copy link

A friendly reminder that this issue had no activity for 30 days.

@rhatdan
Copy link
Member

rhatdan commented Apr 25, 2022

@mheon was this one of the problems you fixed with docker-compose2?

@mheon
Copy link
Member

mheon commented Apr 25, 2022

Nope, but I think we can close - we only have one report of the original issue and it's been almost a year.

@mheon mheon closed this as completed Apr 25, 2022
@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 20, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 20, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
docker-composev2 locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Development

No branches or pull requests

7 participants