podman can not remove pod with multiple containers as non root user

[ikke-t]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

I run AWX containers into single pod to mimic docker compose. While I try removing pod, it fails with use having not permissions to do that. Reason being (discussed in IRC) that recursive killing of pods kills the pause container first, thus loosing namespace for the memcached container.

Steps to reproduce the issue:

1. run pod with containers:

- name: create awx_pod
  command: >
    podman pod create --name awx

- name: Activate postgres container
  command: >
    podman run -d --pod awx --name postgres
    -e POSTGRES_USER="{{ pg_username }}"
    -e POSTGRES_PASSWORD="{{ pg_password }}"
    -e POSTGRES_DB="{{ pg_database }}"
    -e PGDATA="/var/lib/postgresql/data/pgdata"
    "{{ postgresql_image }}"
  register: postgres_container_activate

- name: Activate rabbitmq container
  command: >
    podman run -d --pod awx --name rabbitmq
    -e RABBITMQ_DEFAULT_VHOST="{{ rabbitmq_default_vhost }}"
    -e RABBITMQ_ERLANG_COOKIE="{{ rabbitmq_erlang_cookie }}"
    -e RABBITMQ_DEFAULT_USER="{{ rabbitmq_default_username }}"
    -e RABBITMQ_DEFAULT_PASS="{{ rabbitmq_default_password }}"
    "{{ rabbitmq_image }}"
  register: rabbitmq_container_activate

- name: Activate memcached container
  command: podman run -d --pod awx --name memcached memcached:alpine


- name: Activate AWX Web Container
  tags: test
  command: >
    podman run -d --pod awx --name awx_web
    -u root
    --hostname "{{ awx_web_hostname }}"
    -e http_proxy="{{ http_proxy | default('') }}"
    -e https_proxy="{{ https_proxy | default('') }}"
    -e no_proxy="{{ no_proxy | default('') }}"
    -e SECRET_KEY="{{ secret_key }}"
    -e DATABASE_NAME="{{ pg_database }}"
    -e DATABASE_USER="{{ pg_username }}"
    -e DATABASE_PASSWORD="{{ pg_password }}"
    -e DATABASE_PORT="{{ pg_port }}"
    -e DATABASE_HOST="{{ pg_hostname_actual }}"
    -e RABBITMQ_USER="{{ rabbitmq_default_username }}"
    -e RABBITMQ_PASSWORD="{{ rabbitmq_default_password }}"
    -e RABBITMQ_HOST="rabbitmq"
    -e RABBITMQ_PORT="{{ rabbitmq_port }}"
    -e RABBITMQ_VHOST="{{ rabbitmq_default_vhost }}"
    -e MEMCACHED_HOST="memcached"
    -e MEMCACHED_PORT="11211"
    -e AWX_ADMIN_USER="{{ admin_user|default('admin') }}"
    -e AWX_ADMIN_PASSWORD="{{ admin_password|default('password') }}"
    ansible/awx_web
  register: awx_web_container

- name: Activate AWX Task Container
  command: >
    podman run -d --pod awx --name awx_task
    -u root
    --hostname "{{ awx_task_hostname }}"
    -e http_proxy="{{ http_proxy | default('') }}"
    -e https_proxy="{{ https_proxy | default('') }}"
    -e no_proxy="{{ no_proxy | default('') }}"
    -e SECRET_KEY="{{ secret_key }}"
    -e DATABASE_NAME="{{ pg_database }}"
    -e DATABASE_USER="{{ pg_username }}"
    -e DATABASE_PASSWORD="{{ pg_password }}"
    -e DATABASE_HOST="{{ pg_hostname_actual }}"
    -e DATABASE_PORT="{{ pg_port }}"
    -e RABBITMQ_USER="{{ rabbitmq_default_username }}"
    -e RABBITMQ_PASSWORD="{{ rabbitmq_default_password }}"
    -e RABBITMQ_HOST="rabbitmq"
    -e RABBITMQ_PORT="{{ rabbitmq_port }}"
    -e RABBITMQ_VHOST="{{ rabbitmq_default_vhost }}"
    -e MEMCACHED_HOST="memcached"
    -e MEMCACHED_PORT="11211"
    -e AWX_ADMIN_USER="{{ admin_user|default('admin') }}"
    -e AWX_ADMIN_PASSWORD="{{ admin_password|default('password') }}"
    ansible/awx_task

env variables for the above command in here (the above output with podman generate kube):

apiVersion: v1
kind: Pod
metadata:
  labels:
    app: awx
  name: awx
spec:
  volumes:
  - name: awx_data-volume
    hostPath:
      path: /tmp/awx_data
      type: Directory
  - name: db-volume
    hostPath:
      path: /tmp/pg_data
      type: Directory
  containers:
  #
  # postgres container
  #
  - command:
    - docker-entrypoint.sh
    - postgres
    env:
    - name: PATH
      value: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/postgresql/9.6/bin
    - name: TERM
      value: xterm
    - name: HOSTNAME
    - name: container
      value: podman
    - name: PG_MAJOR
      value: "9.6"
    - name: POSTGRES_USER
      value: awx
    - name: POSTGRES_DB
      value: awx
    - name: GOSU_VERSION
      value: "1.11"
    - name: LANG
      value: en_US.utf8
    - name: PG_VERSION
      value: 9.6.12-1.pgdg90+1
    - name: PGDATA
      value: /var/lib/postgresql/data/pgdata
    - name: POSTGRES_PASSWORD
      value: awxpass
    image: docker.io/library/postgres:9.6
    name: postgres
    resources: {}
    securityContext:
      allowPrivilegeEscalation: true
      capabilities: {}
      privileged: false
      readOnlyRootFilesystem: false
    workingDir: /
    volumeMounts:
    - mountPath: /var/lib/postgresql/data/pgdata:z
      name: db-volume
  #
  # memcached container
  #
  - command:
    - docker-entrypoint.sh
    - memcached
    env:
    - name: PATH
      value: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
    - name: TERM
      value: xterm
    - name: HOSTNAME
    - name: container
      value: podman
    - name: MEMCACHED_VERSION
      value: 1.5.12
    - name: MEMCACHED_SHA1
      value: f67096ba64b0c47668bcad5b680010c4f8987d4c
    image: docker.io/library/memcached:alpine
    name: memcached
    resources: {}
    securityContext:
      allowPrivilegeEscalation: true
      capabilities: {}
      privileged: false
      readOnlyRootFilesystem: false
      runAsUser: 11211
    workingDir: /
  #
  # awx-web container
  #
  - command:
    - /tini
    - --
    - /bin/sh
    - -c
    - /usr/bin/launch_awx.sh
    env:
    - name: PATH
      value: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
    - name: TERM
      value: xterm
    - name: HOSTNAME
      value: awxweb
    - name: container
      value: podman
    - name: DATABASE_NAME
      value: awx
    - name: DATABASE_USER
      value: awx
    - name: DATABASE_PORT
      value: "5432"
    - name: RABBITMQ_HOST
      value: rabbitmq
    - name: RABBITMQ_VHOST
      value: awx
    - name: AWX_ADMIN_PASSWORD
      value: password
    - name: http_proxy
    - name: no_proxy
    - name: DATABASE_PASSWORD
      value: awxpass
    - name: DATABASE_HOST
      value: postgres
    - name: RABBITMQ_PASSWORD
      value: guest
    - name: AWX_ADMIN_USER
      value: admin
    - name: https_proxy
    - name: SECRET_KEY
      value: awxsecret
    - name: RABBITMQ_USER
      value: guest
    - name: RABBITMQ_PORT
      value: "5672"
    - name: MEMCACHED_HOST
      value: memcached
    - name: MEMCACHED_PORT
      value: "11211"
    image: docker.io/ansible/awx_web:latest
    name: awxweb
    resources: {}
    securityContext:
      allowPrivilegeEscalation: true
      capabilities: {}
      privileged: false
      readOnlyRootFilesystem: false
      runAsUser: 0
    workingDir: /var/lib/awx
    volumeMounts:
    - mountPath: /var/lib/awx/projects:z
      name: awx_data-volume
  #
  # awx-task container
  #
  - command:
    - /tini
    - --
    - /bin/sh
    - -c
    - /usr/bin/launch_awx_task.sh
    env:
    - name: PATH
      value: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
    - name: TERM
      value: xterm
    - name: HOSTNAME
      value: awx
    - name: container
      value: podman
    - name: AWX_ADMIN_PASSWORD
      value: password
    - name: http_proxy
    - name: no_proxy
    - name: MEMCACHED_HOST
      value: memcached
    - name: https_proxy
    - name: RABBITMQ_USER
      value: guest
    - name: MEMCACHED_PORT
      value: "11211"
    - name: AWX_ADMIN_USER
      value: admin
    - name: RABBITMQ_PASSWORD
      value: guest
    - name: RABBITMQ_VHOST
      value: awx
    - name: SECRET_KEY
      value: awxsecret
    - name: DATABASE_NAME
      value: awx
    - name: DATABASE_USER
      value: awx
    - name: DATABASE_PASSWORD
      value: awxpass
    - name: RABBITMQ_PORT
      value: "5672"
    - name: DATABASE_HOST
      value: postgres
    - name: DATABASE_PORT
      value: "5432"
    - name: RABBITMQ_HOST
      value: rabbitmq
    image: docker.io/ansible/awx_task:latest
    name: awxtask
    resources: {}
    securityContext:
      allowPrivilegeEscalation: true
      capabilities: {}
      privileged: false
      readOnlyRootFilesystem: false
      runAsUser: 0
    workingDir: /var/lib/awx
    volumeMounts:
    - mountPath: /var/lib/awx/projects:z
      name: awx_data-volume
  - command:
    - docker-entrypoint.sh
    - /bin/sh
    - -c
    - /launch.sh
    env:
    - name: PATH
      value: /opt/rabbitmq/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
    - name: TERM
      value: xterm
    - name: HOSTNAME
    - name: container
      value: podman
    - name: RABBITMQ_LOGS
      value: '-'
    - name: RABBITMQ_SASL_LOGS
      value: '-'
    - name: RABBITMQ_GPG_KEY
      value: 0A9AF2115F4687BD29803A206B73A36E6026DFCA
    - name: RABBITMQ_VERSION
      value: 3.7.4
    - name: RABBITMQ_GITHUB_TAG
      value: v3.7.4
    - name: RABBITMQ_DEFAULT_VHOST
      value: awx
    - name: RABBITMQ_HOME
      value: /opt/rabbitmq
    - name: HOME
      value: /var/lib/rabbitmq
    - name: RABBITMQ_ERLANG_COOKIE
      value: cookiemonster
    - name: RABBITMQ_DEFAULT_USER
      value: guest
    - name: RABBITMQ_DEFAULT_PASS
      value: guest
    image: docker.io/ansible/awx_rabbitmq:3.7.4
    name: rabbitmq
    resources: {}
    securityContext:
      allowPrivilegeEscalation: true
      capabilities: {}
      privileged: false
      readOnlyRootFilesystem: false
    workingDir: /

2. I run the ansible as user, the same could be made pasting commands to command line

3. after pod is up, do podman pod rm awx

Describe the results you received:

[ikke@ohuska ~]$ podman pod rm -f awx                                                                                            
container_linux.go:387: signaling init process caused "operation not permitted"
Error: error stopping container 1cd2e9e5ed6c809ac9db5d5a1460f3b53f8680b5be8ee43e52cd5104795fdff1 to remove pod d930fb92661c4a05c3
9df8fbbdbebcf68f3c40b1dc81e8da3c130b582cef2187: operation not permitted                                                         

Describe the results you expected:

pod to disappear

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

https://kojipkgs.fedoraproject.org//packages/podman/1.2.0/5.dev.git9adcda7.fc31/x86_64/podman-1.2.0-5.dev.git9adcda7.fc31.x86_64.rpm

Output of podman info --debug:

foo

Additional environment details (AWS, VirtualBox, physical, etc.):

[haircommander]

@giuseppe PTAL :)

[ikke-t]

found even easier reproducal :)

podman run -d  --name memcached memcached
$ podman rm -f memcached
container_linux.go:387: signaling init process caused "operation not permitted"
Error: cannot remove container 89437b684522187ecdafc7c6687d838ce5273fbf210dfb50c8d66cae16455154 as it could not be stopped: operation not permitted

[ikke-t]

Dockerfile for the that "official" memcached image here: https://github.com/docker-library/memcached/blob/24ecf1cbeb76244031036eed161bb8bd00c99085/alpine/Dockerfile

This has something to do with setting the user.