podman play kube stops containers immediately with SIGKILL regardless of container stop timeout or kill signal

[andremarianiello]

Issue Description

If you run podman kube down or podman play kube --down the constituent containers are immediately terminated with a SIGKILL regardless of the container's StopTimeout and StopSignal

Steps to reproduce the issue

Steps to reproduce the issue

1. Create a pod with podman kube play. Use an image whose behavior differs based on how it terminates
2. Observe that the created container has the default StopTimeout=10 and default StopSignal=15 (sigterm)
3. Watch pod logs with podman logs -f
4. Tear down pod with podman kube down

Describe the results you received

The container is immediately stopped with a SIGKILL, not SIGTERM

Describe the results you expected

The container should receive StopSignal, followed by a SIGKILL after StopTimeout if container hasn't exited.

podman info output

host:                                                                                
  arch: amd64                                                                        
  buildahVersion: 1.31.3                                                             
  cgroupControllers:                                                                 
  - cpuset                                                                           
  - cpu                                                                              
  - io                                                                               
  - memory                                                                           
  - hugetlb                                                                          
  - pids                                                                             
  - rdma                                                                             
  - misc                                                                             
  cgroupManager: systemd                                                             
  cgroupVersion: v2                                                                  
  conmon:                                                                            
    package: conmon-2.1.8-1.el9.x86_64                                               
    path: /usr/bin/conmon                                                            
    version: 'conmon version 2.1.8, commit: aadb7c890ac6283eb4666d92690238e5fbdec5c7'
  cpuUtilization:                                                                    
    idlePercent: 89.15                                                               
    systemPercent: 3.65                                                              
    userPercent: 7.2                                                                 
  cpus: 16                                                                           
  databaseBackend: boltdb                                                            
  distribution:                                                                      
    distribution: '"rhel"'                                                           
    version: "9.3"                                                                   
  eventLogger: journald                                                              
  freeLocks: 2013                                                                    
  hostname: <redacted>                                               
  idMappings:                                                                        
    gidmap: null                                                                     
    uidmap: null                                                                     
  kernel: 5.14.0-362.24.1.el9_3.x86_64                                               
  linkmode: dynamic                                                                  
  logDriver: journald                                                                
  memFree: 5113741312                                                                
  memTotal: 33384919040                                                              
  networkBackend: netavark                                                           
  networkBackendInfo:                                                                
    backend: netavark                                                                
    dns:                                                                             
      package: aardvark-dns-1.7.0-1.el9.x86_64                                       
      path: /usr/libexec/podman/aardvark-dns                                         
      version: aardvark-dns 1.7.0                                                    
    package: netavark-1.7.0-2.el9_3.x86_64                                           
    path: /usr/libexec/podman/netavark                                               
    version: netavark 1.7.0                                                          
  ociRuntime:                                                                        
    name: crun                                                                       
    package: crun-1.8.7-1.el9.x86_64                                                 
    path: /usr/bin/crun                                                                                                                                      
    version: |-                                                                                                                                              
      crun version 1.8.7                                                                                                                                     
      commit: 53a9996ce82d1ee818349bdcc64797a1fa0433c4                                                                                                       
      rundir: /run/user/0/crun                                                                                                                               
      spec: 1.0.0                                                                                                                                            
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL                                                                                            
  os: linux                                                                                                                                                  
  pasta:                                                                                                                                                     
    executable: /usr/bin/pasta                                                                                                                               
    package: passt-0^20230818.g0af928e-4.el9.x86_64                                                                                                          
    version: |                                                                                                                                               
      pasta 0^20230818.g0af928e-4.el9.x86_64                                                                                                                 
      Copyright Red Hat                                                                                                                                      
      GNU Affero GPL version 3 or later <https://www.gnu.org/licenses/agpl-3.0.html>                                                                         
      This is free software: you are free to change and redistribute it.                                                                                     
      There is NO WARRANTY, to the extent permitted by law.                                                                                                  
  remoteSocket:                                                                                                                                              
    path: /run/podman/podman.sock                                                                                                                            
  security:                                                                                                                                                  
    apparmorEnabled: false                                                                                                                                   
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false                                                                                                                                          
    seccompEnabled: true                                                                                                                                     
    seccompProfilePath: /usr/share/containers/seccomp.json                                                                                                   
    selinuxEnabled: false                                                                                                                                    
  serviceIsRemote: false                                                                                                                                     
  slirp4netns:                                                                                                                                               
    executable: /usr/bin/slirp4netns                                                                                                                         
    package: slirp4netns-1.2.1-1.el9.x86_64                                                                                                                  
    version: |-                                                                                                                                              
      slirp4netns version 1.2.1                                                                                                                              
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194                                                                                                       
      libslirp: 4.4.0                                                                                                                                        
      SLIRP_CONFIG_VERSION_MAX: 3                                                                                                                            
      libseccomp: 2.5.2                                                                                                                                      
  swapFree: 14626136064                                                                                                                                      
  swapTotal: 17175670784                                                                                                                                     
  uptime: 748h 17m 12.00s (Approximately 31.17 days)                                                                                                         
plugins:                                                                                                                                                     
  authorization: null                                                                                                                                        
  log:                                                                                                                                                       
  - k8s-file                                                                                                                                                 
  - none                                                                                                                                                     
  - passthrough                                                                                                                                              
  - journald                                                                                                                                                 
  network:                                                                                                                                                   
  - bridge                                                                                                                                                   
  - macvlan                                                                                                                                                  
  - ipvlan                                                                                                                                                   
  volume:                                                                                                                                                    
  - local                                               
registries:                                             
  docker.io:                                            
    Blocked: false                                      
    Insecure: false                                     
    Location: docker.io                                 
    MirrorByDigestOnly: false                           
    Mirrors:                                            
    - Insecure: false                                   
      Location: <redacted>
      PullFromMirror: ""                                
    Prefix: docker.io                                   
    PullFromMirror: ""                                  
  search:                                               
  - registry.access.redhat.com                          
  - registry.redhat.io                                  
  - docker.io                                           
store:                                                  
  configFile: /usr/share/containers/storage.conf        
  containerStore:                                       
    number: 7                                           
    paused: 0                                           
    running: 4                                          
    stopped: 3                                          
  graphDriverName: overlay                              
  graphOptions: {}                                      
  graphRoot: /var/lib/containers/storage                
  graphRootAllocated: 481623015424                      
  graphRootUsed: 367911256064                           
  graphStatus:                                          
    Backing Filesystem: xfs                             
    Native Overlay Diff: "true"                         
    Supports d_type: "true"                             
    Using metacopy: "false"                             
  imageCopyTmpDir: /var/tmp                             
  imageStore:                                           
    number: 80                                          
  runRoot: /run/containers/storage                      
  transientStore: false                                 
  volumePath: /var/lib/containers/storage/volumes       
version:                                                
  APIVersion: 4.6.1                                     
  Built: 1705652564                                     
  BuiltTime: Fri Jan 19 03:22:44 2024                   
  GitCommit: ""                                         
  GoVersion: go1.20.12                                  
  Os: linux                                             
  OsArch: linux/amd64                                   
  Version: 4.6.1

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

No

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[andremarianiello]

As a workaround for this problem you can include

ExecStopPost=/usr/bin/podman pod stop <pod-name>

in your .kube file to get a graceful shutdown before the podman kube down is run to clean everything up