‹ddev poweroff› on more than one project sometimes leads to "Failed to stop project: Could not kill running container <traefik-router ID>"

[jonaseberle]

Is there an existing issue for this?

• I have searched the existing issues

Output of ddev debug test

Expand `ddev debug test` diagnostic information

======= Existing project config =========
These config files were loaded for project a: [/home/jonas/tmp/a/.ddev/config.yaml]
name: a
type: php
php_version: 8.1
webserver_type: nginx-fpm
webimage: ddev/ddev-webserver:v1.22.6
additional_hostnames: []
additional_fqdns: []
database: {mariadb 10.4}
project_tld: ddev.site
use_dns_when_possible: true
composer_version: 2
nodejs_version: 18
default_container_timeout: 120
======= Creating dummy project named  tryddevproject-1217 in ../tryddevproject-1217 =========
OS Information: Linux e480 6.5.0-14-generic #14-Ubuntu SMP PREEMPT_DYNAMIC Tue Nov 14 14:59:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
User information: uid=1000(jonas) gid=1000(jonas) groups=1000(jonas),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),122(lpadmin),132(lxd),133(sambashare),137(docker)
DDEV version:  ITEM             VALUE                                   
 DDEV version     v1.22.6-26-g94d77509a                   
 architecture     amd64                                   
 db               ddev/ddev-dbserver-mariadb-10.4:v1.22.6 
 ddev-ssh-agent   ddev/ddev-ssh-agent:v1.22.6             
 docker           24.0.5                                  
 docker-compose   v2.23.3                                 
 docker-platform  linux-docker                            
 mutagen          0.17.2                                  
 os               linux                                   
 router           ddev/ddev-traefik-router:v1.22.6        
 web              ddev/ddev-webserver:v1.22.6             
PROXY settings: HTTP_PROXY='' HTTPS_PROXY='' http_proxy='' NO_PROXY=''
======= DDEV global info =========
Global configuration:
instrumentation-opt-in=true
omit-containers=[]
performance-mode=none
router-bind-all-interfaces=false
internet-detection-timeout-ms=3000
disable-http2=false
use-letsencrypt=false
letsencrypt-email=
table-style=default
simple-formatting=false
use-hardened-images=false
fail-on-hook-fail=false
required-docker-compose-version=v2.23.3
use-docker-compose-from-path=false
project-tld=ddev.site
xdebug-ide-location=
no-bind-mounts=false
router=traefik
wsl2-no-windows-hosts-mgt=false
router-http-port=80
router-https-port=443
mailpit-http-port=8025
mailpit-https-port=8026
traefik-monitor-port=10999

======= DOCKER info =========
docker location: -rwxr-xr-x 1 root root 26016184 Aug 16 23:32 /usr/bin/docker
docker version: 
Client:
 Version:           24.0.5
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        24.0.5-0ubuntu1
 Built:             Wed Aug 16 21:32:36 2023
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          24.0.5
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.7
  Git commit:       24.0.5-0ubuntu1
  Built:            Wed Aug 16 21:32:36 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.2
  GitCommit:        
 runc:
  Version:          1.1.7-0ubuntu2
  GitCommit:        
 docker-init:
  Version:          0.19.0
  GitCommit:        
DOCKER_DEFAULT_PLATFORM=notset
======= Mutagen Info =========
======= Docker Info =========
Docker platform: linux-docker 
Using Docker context: default (unix:///var/run/docker.sock) 
docker-compose: v2.23.3 
Using DOCKER_HOST=unix:///var/run/docker.sock 
Docker version: 24.0.5 
Able to run simple container that mounts a volume. 
Able to use internet inside container. 
Docker disk space:
Filesystem                Size      Used Available Use% Mounted on
overlay                 384.2G    350.9G     15.3G  96% /

The ddev-ssh-agent container has been removed. When you start it again you will have to use 'ddev auth ssh' to provide key authentication again.
Existing docker containers: 
CONTAINER ID   IMAGE                               COMMAND     CREATED        STATUS    PORTS     NAMES
1330682a246d   phpstorm_helpers:PS-233.13135.108   "/bin/sh"   3 weeks ago    Created             phpstorm_helpers_PS-233.13135.108
7c4cfffb6587   phpstorm_helpers:PS-233.11799.297   "/bin/sh"   5 weeks ago    Created             phpstorm_helpers_PS-233.11799.297
89d04bad171d   phpstorm_helpers:PS-233.11799.232   "/bin/sh"   6 weeks ago    Created             phpstorm_helpers_PS-233.11799.232
7d60b72ccb68   8f1adb6c4311                        "/bin/sh"   8 weeks ago    Created             phpstorm_helpers_PS-232.10227.13
9c042cb0190c   7d83f3df7b3e                        "/bin/sh"   2 months ago   Created             phpstorm_helpers_PS-232.10072.32
Network ddev_default created 
Starting tryddevproject-1217... 
Network ddev-tryddevproject-1217_default created 
 Container ddev-ssh-agent  Created 
 Container ddev-ssh-agent  Started 
ssh-agent container is running: If you want to add authentication to the ssh-agent container, run 'ddev auth ssh' to enable your keys. 
The hostname tryddevproject-1217.ddev.site is not currently resolvable, trying to add it to the hosts file 
DDEV needs to run with administrative privileges.
You may be required to enter your password for sudo or allow escalation. DDEV is about to issue the command:
  sudo --preserve-env=HOME /home/jonas/src/github.com/jonaseberle/ddev/.gotmp/bin/linux_amd64/ddev hostname tryddevproject-1217.ddev.site 127.0.0.1
 
Not populating custom commands or hostadditions because running with root privileges
 
Building project images... 
Project images built in 0s. 
 Container ddev-tryddevproject-1217-web  Created 
 Container ddev-tryddevproject-1217-db  Created 
 Container ddev-tryddevproject-1217-web  Started 
 Container ddev-tryddevproject-1217-db  Started 
Waiting for web/db containers to become ready: [web db] 
Starting ddev-router if necessary... 
 Container ddev-router  Created 
 Container ddev-router  Started 
Waiting for additional project containers to become ready... 
All project containers are now ready. 
Successfully started tryddevproject-1217 
Project can be reached at https://tryddevproject-1217.ddev.site https://127.0.0.1:32893 
Instrumentation is opted in, but AmplitudeAPIKey is not available. This usually means you have a locally-built ddev binary or one from a PR build. It's not an error. Please report it if you're using an official release build. 
======== Curl of site from inside container:
HTTP/1.1 200 OK
Server: nginx
Date: Wed, 17 Jan 2024 17:27:54 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Vary: Accept-Encoding

======== curl -I of http://tryddevproject-1217.ddev.site from outside:
HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8
Date: Wed, 17 Jan 2024 17:27:54 GMT
Server: nginx
Vary: Accept-Encoding

======== full curl of http://tryddevproject-1217.ddev.site from outside:
Success accessing database... db via TCP/IP
ddev is working. You will want to delete this project with 'ddev delete -Oy tryddevproject-1217'
======== Project ownership on host:
drwxr-xr-x 4 jonas jonas 4096 Jan 17 18:27 ../tryddevproject-1217
======== Project ownership in container:
drwxr-xr-x 4 jonas jonas 4096 Jan 17 17:27 /var/www/html
======== In-container filesystem:
Filesystem                Type 1K-blocks      Used Available Use% Mounted on
/dev/mapper/neon--vg-home ext4 402852288 368172584  15815632  96% /var/www/html
======== curl again of tryddevproject-1217 from host:
Success accessing database... db via TCP/IP
ddev is working. You will want to delete this project with 'ddev delete -Oy tryddevproject-1217'
Thanks for running the diagnostic. It was successful.
Please provide the output of this script in a new gist at gist.github.com
Running ddev launch in 5 seconds

Please run cleanup after debugging with 'ddev debug testcleanup'

Expected Behavior

Executing ddev poweroff, all global and project DDEV containers are stopped and removed.

Actual Behavior

I am often getting this output:

Failed to stop project b: 
API error (500): Could not kill running container d8079abded3b448b54fc3d22bc0abfe10ffbeb69cd7066027655bbf5598992d5, cannot remove - container d8079abded3b PID 1968501 is zombie and can not be killed. Use the --init option when creating containers to run an init inside the container that forwards signals and reaps processes 

Using docker inspect <id> reveals that this ID is the container ddev-router

Steps To Reproduce

This produces the error often (not always) for me:

mkdir -p a b \
    && cd a && ddev config --auto && cd - \
    && cd b && ddev config --auto && cd - \
    && ddev start a b \
    && ddev poweroff

Anything else?

DDEV version: current master

[stasadev]

Hi @jonaseberle,

You can try #5707 (comment) to see if it makes a difference.

And I want to ask you how you kill this zombie container in the end, because I don't know what else we can try (we're already using the --force flag).

[jonaseberle]

Thank you!

Running ddev poweroff a second time gets rid of the container and its processes for me.

[rfay]

@jonaseberle you say you're running ddev poweroff in more than one project at the same time? (Is that what the title implies?) As you already know, ddev poweroff gets a list of projects and then stops them, and the last stop kills ddev-router. But there would certainly be conflicts if you were running two ddev poweroff commands at the same time, especially in different projects.

[stasadev]

I think this means:

If ddev list --active-only has 1 project, running ddev poweroff will succeed.

If ddev list --active-only has 2+ projects, running ddev poweroff may leave behind a zombie container for ddev-router, and you will need to run ddev poweroff a second time to remove it.

PR #5707 probably fixes this edge case.

[jonaseberle]

Sorry, I haven't seen the answers here in time.

The title is unclear. I should have written "‹ddev poweroff› with more than one project running sometimes leads to...".

[rfay]

This was fully fixed by

• fix: use init option to prevent zombie ddev-router, for #5705 #5707

true?

[rfay]

Closing, as I think this was fixed in #5707 - Please let me know or reopen if I'm wrong.