feature: Stop pod when one of its containers dies

[fpoirotte]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind feature

Description
When a container belonging to a pod dies (exits successfully or with an error code, or due to a crash), I would like the whole pod to stop.

My use case is as follows:

• I start a pod using systemd and podman play kube (through podman-kube@.service)
• The pod contains several containers with complex interdependencies. When a container dies, it is not enough to just restart the faulty container (and so, using restartPolicy: always inside the pod's manifest is not an option).
• Instead, I would like to stop the whole pod (so that systemd can pick on that and restart the pod based on its own Restart= policy)

Ideally, this would also take into account results from health checks (hence an unhealthy container would also kill the pod).

Steps to reproduce the issue:

1. Create /tmp/example.yml with the following content:

apiVersion: v1
kind: Pod
metadata:
  name: example
spec:
  restartPolicy: Never

  containers:
  - name: faulty
    image: "alpine:3.15"
    command:
    - /bin/sleep
    - "3"

  - name: goodboy
    image: "alpine:3.15"
    command:
    - /bin/sh
    - "-c"
    - "mkfifo /tmp/fifo; read -r cmd < /tmp/fifo"

3. Start the pod with podman-kube@.service : systemctl start podman-kube@-tmp-example.yml.service

4. Wait 5 seconds then check the pod and service's status.

Describe the results you received:

# podman pod ls
POD ID        NAME        STATUS      CREATED             INFRA ID      # OF CONTAINERS
c650eca91928  example     Degraded    About a minute ago  37c09f5a7c81  3

# podman ps -a
CONTAINER ID  IMAGE                                    COMMAND     CREATED         STATUS                     PORTS       NAMES
819e07d1cb4e  localhost/podman-pause:4.3.1-1668180253              16 seconds ago  Up 13 seconds ago                      975e9878585a-service
37c09f5a7c81  localhost/podman-pause:4.3.1-1668180253              16 seconds ago  Up 13 seconds ago                      c650eca91928-infra
bb81f8866d7a  docker.io/library/alpine:3.15                        15 seconds ago  Exited (0) 10 seconds ago              example-faulty
31f4bfaf7472  docker.io/library/alpine:3.15                        13 seconds ago  Up 13 seconds ago                      example-goodboy

# systemctl status podman-kube@-tmp-example.yml.service
● podman-kube@-tmp-example.yml.service - A template for running K8s workloads via podman-play-kube
     Loaded: loaded (/usr/lib/systemd/system/podman-kube@.service; disabled; vendor preset: disabled)
     Active: active (running) since Wed 2022-12-14 12:32:01 CET; 10s ago
             ^^^^^^^^^^^^^^^^

Describe the results you expected:

The pod should have been stopped, resulting in the systemd service being marked as failed.

Additional information you deem important (e.g. issue happens only occasionally):

Happens every time.

Output of podman version:

Client:       Podman Engine
Version:      4.3.1
API Version:  4.3.1
Go Version:   go1.18.7
Built:        Fri Nov 11 16:24:13 2022
OS/Arch:      linux/amd64

Output of podman info:

  arch: amd64
  buildahVersion: 1.28.0
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.5-1.fc36.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.5, commit: '
  cpuUtilization:
    idlePercent: 61.33
    systemPercent: 10.57
    userPercent: 28.1
  cpus: 8
  distribution:
    distribution: fedora
    variant: workstation
    version: "36"
  eventLogger: file
  hostname: localhost
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.0.11-200.fc36.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 4358381568
  memTotal: 16622592000
  networkBackend: cni
  ociRuntime:
    name: crun
    package: crun-1.7-1.fc36.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.7
      commit: 40d996ea8a827981895ce22886a9bac367f87264
      rundir: /run/user/0/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +WASM:wasmedge +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-0.2.beta.0.fc36.x86_64
    version: |-
      slirp4netns version 1.2.0-beta.0
      commit: 477db14a24ff1a3de3a705e51ca2c4c1fe3dda64
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 14279786496
  swapTotal: 17179860992
  uptime: 147h 26m 59.00s (Approximately 6.12 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 4
    paused: 0
    running: 3
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 52521566208
  graphRootUsed: 33276317696
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 7
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.3.1
  Built: 1668180253
  BuiltTime: Fri Nov 11 16:24:13 2022
  GitCommit: ""
  GoVersion: go1.18.7
  Os: linux
  OsArch: linux/amd64
  Version: 4.3.1

Package info (e.g. output of rpm -q podman or apt list podman or brew info podman):

podman-4.3.1-1.fc36.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

Physical box

[fpoirotte]

The following issues are somewhat related to this feature request but do not fully address it: #14505, #15806.

[vrothberg]

Thanks for reaching out. I understand the use case and think we can support it fairly easily.

podman kube play is making use of so-called pod exit policies. Currently, there are two such policies (see docs):

• "continue" - the pod will just continue running
• "stop" - the pod will be stopped when the last container exits. This is the default in kube play but since one container is still running, the systemd service will not be restarted.

To meet your use case, we could add a new policy (e.g., "stop-any" ?) that would stop the pod when any container exits.

Plumbing that into kube-play would probably require a podman-specific annotation (io.podman.pod.exit-policy ?) such that it can be shipped directly in the YAML.

@fpoirotte, does that sound right to you?

@rhatdan @umohnani8 @baude PTAL

@alexlarsson @ygalblum could that be of interest to your use case?

[fpoirotte]

Thanks Valentin for your answer (and your other suggestions on the mailing lists as well).
Yes, I think your solution using annotations would fit the bill.

[vrothberg]

@rhatdan @ygalblum PTAL

That fits into the context we discussed earlier today.

[rhatdan]

Yes. although to be clear stop-any would be for any container other then init containers...

[vrothberg]

@rhatdan should it be another annotation? It's not a K8s feature AFAICS.

[rhatdan]

Well if we support jobs what is the behaviour in Kubernetes, @ygalblum

[ygalblum]

@rhatdan No, jobs also keep running the pod until all the containers complete even if one of them failed. Moreover, I don't think it makes sense do so for jobs. In jobs (unlike deployments), the containers are supposed to return. Since you do not assume that they all return at the same time, you want each of them to return at their own pace

[SergeyKanzhelev]

@rhatdan should it be another annotation? It's not a K8s feature AFAICS.

stumbled on this issue. There are sidecar container KEP that we are working on in k8s: kubernetes/enhancements#3759 and also another proposal: kubernetes/enhancements#3582

Reading thru this scenario I feel it is close to kubernetes/enhancements#3582. Will setting TerminatePod flag on each job be what you are looking for in your scenario?

[fpoirotte]

Thank you for @SergeyKanzhelev for the heads up.
Yes, kubernetes/enhancements#3582 is exactly what I was looking for!
I see it's been dropped in favor of kubernetes/enhancements#3759. I'll keep an eye on that as well.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.