piped never notices when a plugin process exits, /healthz stays ok

[vipulpandey21]

What happened:

Piped runs each plugin (Kubernetes, Terraform, etc.) as its own separate program, not inside piped itself. Think of piped as a manager and each plugin as a worker running in its own room.

If one of those workers (a plugin) suddenly dies, for example it crashes or gets killed for using too much memory, the manager (piped) never checks on it again. Piped just keeps running like nothing happened. It does not print an error, it does not tell anyone, and its health-check page (/healthz) keeps saying everything is fine.

Meanwhile, every deployment that needed that plugin will keep failing quietly in the background, and nobody gets told why, until someone manually notices and restarts piped.

Why this matters:

Piped is usually run inside Kubernetes, and Kubernetes is already set up to restart piped automatically whenever its health-check page says something is wrong. So the fix does not even need new infrastructure, piped just needs to actually say "something is wrong" when a plugin dies, instead of always saying "ok".

How I found this:

I looked at the code that starts a plugin (pkg/app/pipedv1/cmd/piped/piped.go) and traced what piped does with it afterwards.

Piped does have a way to know if a plugin process died, it is right here in pkg/lifecycle/binary.go:

go func() {
	err := cmd.Wait()   // this returns whenever the plugin process exits, for any reason
	c.result.Store(&err)
	close(c.stoppedCh)  // this is piped's only signal that the plugin is dead
}()

But if you look at piped.go, this signal is only checked in one place, when piped itself is shutting down:

group.Go(func() error {
	<-ctx.Done()   // only runs when piped is shutting down on purpose
	...
	// stops all plugins here
})

So if a plugin dies at 3am while piped is otherwise fine, nothing in the code ever notices, because the only place that checks is the shutdown path.

I even checked if this "is it dead" signal is used anywhere else in the whole project, and it turns out it is, just not for plugins:

$ grep -rn "\.IsRunning()" --include=*.go .
./pkg/app/launcher/cmd/launcher/launcher.go:291

That one use is for supervising piped itself (for auto-upgrades). So the idea of "check if the process died" already exists and is trusted in this codebase, it is just never applied to plugins.

And the health-check endpoint really does just say "ok" no matter what:

admin.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
	w.Write([]byte("ok"))
})

I also checked how piped is actually deployed, and confirmed Kubernetes is already told to watch this exact endpoint (manifests/piped/templates/deployment.yaml, quickstart/manifests/pipedv1-exp.yaml):

livenessProbe:
  httpGet:
    path: /healthz
readinessProbe:
  httpGet:
    path: /healthz

So the fix here is not about adding a whole new alerting system, it is about making an endpoint that already exists, and is already being watched, actually tell the truth.

How to reproduce it:

1. Run pipedv1 with at least one plugin configured.
2. Kill that plugin's process directly, for example kill -9 <plugin-pid>, while piped keeps running.
3. Check /healthz, it still returns 200 ok.
4. Try a deployment that needs that plugin, it fails, but nothing in the logs says "the plugin is dead", it just looks like a random connection error.

Why I think it is worth fixing, not just a one-time crash:

Piped used to run all this deployment logic inside itself. If something broke, piped itself would crash loudly, and Kubernetes would restart it (through the same health-check). The newer plugin design moved that logic out into separate programs so that one broken plugin cannot crash the rest of piped, which is good, but it also means piped needs to actually watch those programs now, and right now it does not. As more plugins get written by people outside the core team, a bug in just one of them could quietly break only that one feature for a long time before anyone notices.

Environment:

• piped version: pipedv1, master at commit 56fb80e7
• control-plane version: not related
• Others: applies to any plugin started the normal local way (runPlugins / RunBinary)

I already have a fix and tests ready, opening the PR for it now.