zombie issues

[therealnb]

Bug description

See the sequence of commands below. There seem to be accidental long lived process for thv.

Steps to reproduce

nigels-MacBook-Pro:code nigel$ ps aux | grep 'thv'
nigel            48437   0.3  0.1 411547648  16608   ??  Ss   11:17a.m.   0:03.11 /Users/nigel/go/bin/thv restart github --foreground
nigel            56254   0.0  0.3 411530208  50256   ??  Ss   12:29p.m.   0:00.65 /Users/nigel/go/bin/thv restart meta-mcp --foreground
nigel            53714   0.0  0.2 412040784  39824 s006  S+   12:12p.m.   0:00.71 /Users/nigel/code/toolhive-studio/bin/darwin-arm64/thv serve --openapi --experimental-mcp --experimental-mcp-host=127.0.0.1 --experimental-mcp-port=60887 --host=127.0.0.1 --port=50065
nigel            47643   0.0  0.1 411554992  18832   ??  Ss   10:35a.m.   0:05.46 /Users/nigel/go/bin/thv restart fetch --foreground
nigel            15761   0.0  0.0 411542112   4848   ??  S     5:14p.m.   0:06.39 thv proxy stdio mcpserver-proxy-stdio-target-15280-1760976842757344000-1760976427
nigel            57598   0.0  0.0 410733664   1568 s019  S+   12:45p.m.   0:00.01 grep thv
nigels-MacBook-Pro:code nigel$ thv restart fetch
A new version of ToolHive is available: v0.3.11
Currently running: v0.3.7-14-g595e5e76-dirty
12:46PM	INFO	Container fetch is already running
Workload fetch restarted successfully
nigels-MacBook-Pro:code nigel$ ps aux | grep 'thv'
nigel            56254   0.0  0.3 411530208  43488   ??  Ss   12:29p.m.   0:00.69 /Users/nigel/go/bin/thv restart meta-mcp --foreground
nigel            53714   0.0  0.2 412032592  39776 s006  S+   12:12p.m.   0:00.75 /Users/nigel/code/toolhive-studio/bin/darwin-arm64/thv serve --openapi --experimental-mcp --experimental-mcp-host=127.0.0.1 --experimental-mcp-port=60887 --host=127.0.0.1 --port=50065
nigel            48437   0.0  0.1 411547904  23424   ??  Ss   11:17a.m.   0:03.16 /Users/nigel/go/bin/thv restart github --foreground
nigel            47643   0.0  0.2 411554992  25168   ??  Ss   10:35a.m.   0:05.52 /Users/nigel/go/bin/thv restart fetch --foreground
nigel            15761   0.0  0.0 411542112   4832   ??  S     5:14p.m.   0:06.41 thv proxy stdio mcpserver-proxy-stdio-target-15280-1760976842757344000-1760976427
nigel            57852   0.0  0.0 410734688   1664 s019  S+   12:46p.m.   0:00.00 grep thv
nigels-MacBook-Pro:code nigel$ thv stop fetch
A new version of ToolHive is available: v0.3.11
Currently running: v0.3.7-14-g595e5e76-dirty
12:48PM	INFO	Stopping proxy process (PID: 47643)...
12:48PM	INFO	Proxy process stopped
12:48PM	INFO	Stopping containers for fetch...
12:48PM	INFO	Server fetch belongs to group default, updating 0 registered client(s)
12:48PM	INFO	No target clients found for server fetch
12:48PM	INFO	Client configurations for fetch removed
12:48PM	INFO	Successfully stopped fetch...
workload fetch stopped successfully
nigels-MacBook-Pro:code nigel$ ps aux | grep 'thv'
nigel            56254   0.0  0.3 411530208  43280   ??  Ss   12:29p.m.   0:00.78 /Users/nigel/go/bin/thv restart meta-mcp --foreground
nigel            53714   0.0  0.1 412032592  18592 s006  S+   12:12p.m.   0:00.80 /Users/nigel/code/toolhive-studio/bin/darwin-arm64/thv serve --openapi --experimental-mcp --experimental-mcp-host=127.0.0.1 --experimental-mcp-port=60887 --host=127.0.0.1 --port=50065
nigel            48437   0.0  0.1 411547904  22976   ??  Ss   11:17a.m.   0:03.23 /Users/nigel/go/bin/thv restart github --foreground
nigel            15761   0.0  0.1 411542112  15120   ??  S     5:14p.m.   0:06.42 thv proxy stdio mcpserver-proxy-stdio-target-15280-1760976842757344000-1760976427
nigel            57903   0.0  0.0 410584080    736 s019  R+   12:48p.m.   0:00.01 grep thv
nigels-MacBook-Pro:code nigel$ thv restart fetch
A new version of ToolHive is available: v0.3.11
Currently running: v0.3.7-14-g595e5e76-dirty
12:49PM	INFO	Loaded configuration from state for fetch
12:49PM	INFO	Starting tooling server fetch...
12:49PM	INFO	Logging to: /Users/nigel/Library/Application Support/toolhive/logs/fetch.log
12:49PM	INFO	MCP server is running in the background (PID: 57975)
12:49PM	INFO	Use 'thv stop fetch' to stop the server
Workload fetch restarted successfully
nigels-MacBook-Pro:code nigel$ ps aux | grep 'thv'
nigel            56254   0.0  0.3 411530208  50432   ??  Ss   12:29p.m.   0:00.82 /Users/nigel/go/bin/thv restart meta-mcp --foreground
nigel            53714   0.0  0.1 412032592  18768 s006  S+   12:12p.m.   0:00.80 /Users/nigel/code/toolhive-studio/bin/darwin-arm64/thv serve --openapi --experimental-mcp --experimental-mcp-host=127.0.0.1 --experimental-mcp-port=60887 --host=127.0.0.1 --port=50065
nigel            48437   0.0  0.1 411547904  22592   ??  Ss   11:17a.m.   0:03.25 /Users/nigel/go/bin/thv restart github --foreground
nigel            15761   0.0  0.0 411542112   4848   ??  S     5:14p.m.   0:06.42 thv proxy stdio mcpserver-proxy-stdio-target-15280-1760976842757344000-1760976427
nigel            57985   0.0  0.0 410734688   1632 s019  S+   12:49p.m.   0:00.01 grep thv
nigel            57975   0.0  0.3 412049360  51760   ??  Ss   12:49p.m.   0:00.04 /Users/nigel/go/bin/thv restart fetch --foreground

Expected behavior

No zombies

Actual behavior

Zombies

Environment (if relevant)

• OS/version: Mac 15.6.1
• ToolHive version: v0.3.7-14-g595e5e76-dirty

Additional context

These probably don't do that much harm. The don't look good though.

[therealnb]

a little more testing

ps aux | grep 'thv'
nigel            52211   0.2  0.1 411556768  16896   ??  Ss    1:54p.m.   0:05.35 /Users/nigel/go/bin/thv restart postgres-mcp-pro --foreground
nigel            71364   0.0  0.1 411953008  16480   ??  Ss    3:20p.m.   0:02.07 /Users/nigel/code/toolhive-studio/bin/darwin-arm64/thv restart internal---meta-mcp --foreground
nigel            55448   0.0  0.1 412036160  10432 s006  S+    2:09p.m.   0:04.74 /Users/nigel/code/toolhive-studio/bin/darwin-arm64/thv serve --openapi --experimental-mcp --experimental-mcp-host=127.0.0.1 --experimental-mcp-port=56189 --host=127.0.0.1 --port=50009
nigel            52398   0.0  0.1 411557024  16304   ??  Ss    1:54p.m.   0:05.10 /Users/nigel/go/bin/thv restart everything --foreground
nigel            52383   0.0  0.1 411529472  16704   ??  Ss    1:54p.m.   0:04.67 /Users/nigel/go/bin/thv restart sequentialthinking --foreground
nigel            52366   0.0  0.1 411554624  16320   ??  Ss    1:54p.m.   0:04.37 /Users/nigel/go/bin/thv restart terraform --foreground
nigel            52361   0.0  0.1 411557008  17104   ??  Ss    1:54p.m.   0:05.24 /Users/nigel/go/bin/thv restart semgrep --foreground
nigel            52348   0.0  0.1 411547504  16080   ??  Ss    1:54p.m.   0:04.84 /Users/nigel/go/bin/thv restart memory --foreground
nigel            52334   0.0  0.1 411556656  15440   ??  Ss    1:54p.m.   0:04.43 /Users/nigel/go/bin/thv restart sqlite --foreground
nigel            52327   0.0  0.1 411538688  15136   ??  Ss    1:54p.m.   0:03.98 /Users/nigel/go/bin/thv restart fetch --foreground
nigel            52310   0.0  0.1 411556000  16256   ??  Ss    1:54p.m.   0:04.63 /Users/nigel/go/bin/thv restart time --foreground
nigel            52299   0.0  0.1 411557584  15936   ??  Ss    1:54p.m.   0:04.64 /Users/nigel/go/bin/thv restart filesystem --foreground
nigel            52274   0.0  0.1 411549488  16160   ??  Ss    1:54p.m.   0:04.58 /Users/nigel/go/bin/thv restart git --foreground
nigel            52251   0.0  0.1 411563136  15280   ??  Ss    1:54p.m.   0:04.19 /Users/nigel/go/bin/thv restart playwright --foreground
nigel            52245   0.0  0.1 411572048  16736   ??  Ss    1:54p.m.   0:05.07 /Users/nigel/go/bin/thv restart github --foreground
nigel            52190   0.0  0.1 411572736  16528   ??  Ss    1:54p.m.   0:05.00 /Users/nigel/go/bin/thv restart redis --foreground
nigel            52174   0.0  0.1 411549632  16432   ??  Ss    1:54p.m.   0:04.70 /Users/nigel/go/bin/thv restart elasticsearch --foreground
nigel            50699   0.0  0.1 411547008  15744   ??  Ss    1:46p.m.   0:04.98 /Users/nigel/go/bin/thv restart sqlite --foreground
nigel            50280   0.0  0.1 411562944  15792   ??  Ss    1:45p.m.   0:04.66 /Users/nigel/go/bin/thv restart sqlite --foreground
nigel            36402   0.0  0.1 411565056  15664   ??  Ss   11:39a.m.   0:08.73 /Users/nigel/go/bin/thv restart sqlite --foreground
nigel            29752   0.0  0.1 411548816  21104   ??  Ss   10:54a.m.   0:09.50 /Users/nigel/go/bin/thv restart unhealthy-test-1761126576 --foreground
nigel            27824   0.0  0.0 411564224   7152   ??  S    10:49a.m.   0:02.12 thv proxy stdio telemetry-test-27714-1761126596177460000-1761126576
nigel            27795   0.0  0.0 411522912   6960   ??  S    10:49a.m.   0:02.00 thv proxy stdio telemetry-test-27714-1761126589007426000-1761126576
nigel            27763   0.0  0.0 411537568   7024   ??  S    10:49a.m.   0:02.02 thv proxy stdio telemetry-test-27714-1761126582805320000-1761126576
nigel            25575   0.0  0.1 411555248  15760   ??  Ss   10:46a.m.   0:10.68 /Users/nigel/go/bin/thv restart sqlite --foreground
nigel            22314   0.0  0.0 411538464   6640   ??  S    10:37a.m.   0:02.17 thv proxy stdio mcpserver-proxy-stdio-target-21390-1761125830164297000-1761125702
nigel            85429   0.0  0.1 411549056  12592   ??  Ss    4:52p.m.   0:38.68 /Users/nigel/go/bin/thv restart gitlab --foreground
nigel            83660   0.0  0.1 411550640  19632   ??  Ss    4:46p.m.   0:35.56 /Users/nigel/go/bin/thv restart firecrawl --foreground
nigel            83635   0.0  0.1 411548720  12000   ??  Ss    4:46p.m.   0:34.14 /Users/nigel/go/bin/thv restart perplexity-ask --foreground
nigel            80182   0.0  0.1 411574256  12592   ??  Ss    4:16p.m.   0:36.22 /Users/nigel/go/bin/thv restart firecrawl --foreground
nigel            80129   0.0  0.1 411564928  11872   ??  Ss    4:16p.m.   0:36.38 /Users/nigel/go/bin/thv restart perplexity-ask --foreground
nigel            76849   0.0  0.1 411556656  12528   ??  Ss   Tue03p.m.   0:40.84 /Users/nigel/go/bin/thv restart gitlab-20251021-155319 --foreground
nigel            76613   0.0  0.1 411557808  13024   ??  Ss   Tue03p.m.   0:39.14 /Users/nigel/go/bin/thv restart gitlab --foreground
nigel            15761   0.0  0.0 411542112   3888   ??  S    Mon05p.m.   0:15.99 thv proxy stdio mcpserver-proxy-stdio-target-15280-1760976842757344000-1760976427
nigel            14721   0.0  0.0 410724448   1424 s019  S+    4:01p.m.   0:00.00 grep thv

[therealnb]

pkill -f 'thv restart'
did remove the zombies, but left them in an unhealthy state

thv ls --all
A new version of ToolHive is available: v0.4.2
Currently running: v0.4.0-3-g9f5a7e75
NAME                  PACKAGE                                          STATUS      URL                                             PORT    TOOL TYPE   GROUP               CREATED AT
elasticsearch         docker.elastic.co/mcp/elasticsearch:0.4.5        unhealthy   http://127.0.0.1:46227/mcp                      46227   mcp         default             2025-10-23 12:38:43.461613 +0100 BST
filesystem            docker.io/mcp/filesystem:latest                  unhealthy   http://127.0.0.1:20415/sse#filesystem           20415   mcp         default             2025-10-23 12:40:13.545925 +0100 BST
git                   docker.io/mcp/git:latest                         unhealthy   http://127.0.0.1:32978/sse#git                  32978   mcp         default             2025-10-23 12:40:05.478601 +0100 BST
github                ghcr.io/github/github-mcp-server:v0.19.1         unhealthy   http://127.0.0.1:61947/sse#github               61947   mcp         default             2025-10-23 12:39:09.127878 +0100 BST
gitlab                iwakitakuma/gitlab-mcp:2.0.7                     unhealthy   http://127.0.0.1:21069/mcp                      21069   mcp         default             2025-10-23 12:41:35.249389 +0100 BST
internal---meta-mcp   mcp-optimizer:latest                             running     http://127.0.0.1:12092/mcp                      12092   mcp         __mcp-optimizer__   2025-10-23 12:59:28.640262 +0100 BST
k8s                   ghcr.io/stackloklabs/mkp/server:0.2.4            unhealthy   http://127.0.0.1:58630/sse#k8s                  58630   mcp         default             2025-10-23 12:40:07.047399 +0100 BST
memory                docker.io/mcp/memory:latest                      unhealthy   http://127.0.0.1:19577/sse#memory               19577   mcp         default             2025-10-23 12:40:30.309196 +0100 BST
playwright            mcr.microsoft.com/playwright/mcp:v0.0.43         unhealthy   http://127.0.0.1:36714/mcp                      36714   mcp         default             2025-10-23 12:39:54.44352 +0100 BST
postgres-mcp-pro      crystaldba/postgres-mcp:0.3.0                    unhealthy   http://127.0.0.1:18885/sse#postgres-mcp-pro     18885   mcp         default             2025-10-23 12:39:02.793398 +0100 BST
redis                 docker.io/mcp/redis:latest                       unhealthy   http://127.0.0.1:46105/sse#redis                46105   mcp         default             2025-10-23 12:38:53.492787 +0100 BST
sequentialthinking    docker.io/mcp/sequentialthinking:latest          unhealthy   http://127.0.0.1:51214/sse#sequentialthinking   51214   mcp         default             2025-10-23 12:40:59.99469 +0100 BST
sqlite                ghcr.io/stackloklabs/sqlite-mcp/server:0.0.1     unhealthy   http://127.0.0.1:28062/sse#sqlite               28062   mcp         default             2025-10-23 12:40:23.427473 +0100 BST
terraform             docker.io/hashicorp/terraform-mcp-server:0.3.1   unhealthy   http://127.0.0.1:39341/sse#terraform            39341   mcp         default             2025-10-23 12:40:55.850734 +0100 BST
time                  docker.io/mcp/time:latest                        unhealthy   http://127.0.0.1:20498/sse#time                 20498   mcp         default             2025-10-23 12:40:19.761351 +0100 BST

restart and run didn't work. I had to rm and run them.

[therealnb]

These are NOT zombies - they are intentional long-running supervisor processes. Here's how the system works:

1. When you run thv run (without --foreground), it calls RunWorkloadDetached which spawns a detached supervisor process:

   detachedArgs := []string{"restart", runConfig.BaseName, "--foreground"}
   detachedCmd := exec.Command(execPath, detachedArgs...)

(See manager.go:393)

2. This creates a thv restart --foreground process that acts as a supervisor for the MCP server. This supervisor:

• Starts and monitors the container/transport
• Manages the HTTP/SSE proxy that clients connect to
• Handles lifecycle events (signals, container exits)
• Blocks indefinitely waiting for shutdown signals or container exit

(See runner.go:321-326)

[therealnb]

My first impression of these were that they were accidentally long lived commands, rather than supervisors. It would be better if these had a name something like thv supervisor. I assume other users will get the same impression.

It is also a little strange that if they are supervisors, they don't come back when you kill them. It might be an idea to have a watchdog for this, or combine them into one supervisor.

I have fixed the restart behaviour in #2306 which will mean a further thv restart command will work if the supervisor died or if the container died.