Deleted projects leave orphaned Instances/WorkloadDeployments and never-ending projector errors

[scotwells]

Problem

When a project is deleted, the compute resources that were federated into the control plane on its behalf are left behind. We found orphaned Instances and WorkloadDeployments in staging belonging to projects that no longer exist, and they never get cleaned up.

The most visible symptom is noise that never stops: the instance projector keeps trying to reconcile an orphaned Instance, can't resolve the deleted project's cluster, and logs an error on every attempt — forever, at a backoff-widening cadence. Because these reconciles fail, they inflate the controller error metrics and add steady error-log volume that can bury or mask genuine projection failures. As projects are created and torn down over time (e.g. every e2e run), the orphaned objects accumulate with no bound.

Impact

• Never-ending error churn from a permanently unresolvable reconcile — inflates error metrics and log volume, and can mask real failures. This also risks tripping controller health/alerting on what is actually dead data.
• Leaked resources grow unbounded as projects churn; nothing reclaims them.
• Confusing operator signal — an error that looks like a live problem but corresponds to a project that's already gone.

What we found in staging

Audited every federated compute resource against the live project list:

• 1 orphaned Instance — belonged to a deleted project; it had no owner and no finalizer, drove no running workload, and was the source of the perpetual projector error. Removed manually to stop the noise.
• 2 orphaned WorkloadDeployments — federated copies for two other deleted projects, still present.

(There is also a much larger set of leftover namespaces on the federation plane for deleted projects, but those are managed by the networking/federation layer, not compute, and are out of scope here.)

Root cause

Federated compute resources don't get reclaimed when their project goes away. The write-back and federated copies carry no owner reference to anything that is itself deleted when the project is removed, so nothing cascades — and the project's cluster deregisters, so the projector can never resolve it again. On top of that, the projector treats an unresolvable project cluster as a hard, retryable error rather than recognizing it as terminal, which turns a dead object into an infinite error loop.

Proposed fixes (two layers)

1. Stop the noise (small, low-risk). When the projector can't resolve a project's cluster, handle it gracefully instead of erroring on every attempt — the federator already does exactly this for the equivalent case. This also removes the harmless but noisy errors during the normal startup window before projects are engaged.
2. Reclaim the orphans (root cause). Garbage-collect federated Instances and WorkloadDeployments when their project (and its cluster) is deleted, so leftover objects don't accumulate.

Acceptance criteria

• Deleting a project leaves no orphaned Instances or WorkloadDeployments behind on the federation plane.
• A project whose cluster can't be resolved no longer produces perpetual reconcile errors or error-metric inflation.

[aflor024]

This is in prod now, not just staging — but having read the projector code, it turns out to be a different bug. Filed separately as #207.

instance-projector on prod-infrastructure-control-plane: 213 errors out of 223 reconciles over the 14h to Aug 4 14:50 UTC, zero successes in the final hour. Errors start Jul 21 07:30 UTC and have been continuous since Aug 2. All of it traces to three Instances in a single namespace, and none of them is a deleted-project orphan — they share one namespace whose project is alive, and datum-cloud resolves fine for the one that gets far enough to try. They're federation-plane write-back copies that stranded because the only thing that reclaims them is a finalizer on the cell-side Instance, and it never ran. That's #207.

Which makes this issue narrower than its title reads: the genuinely-deleted-project case, where the cluster deregisters and can never resolve again. Layer-2 GC keying on project deletion is right for that and wouldn't have touched the prod objects.

On layer 1 I'd hold off. It looks like the safe half, but the projector errors deliberately — "surface it as an error for backoff and visibility rather than silently dropping the projection" — and its comment names the assumption being violated: "a deleted WD ends the retries once its write-back copies are gone". Applied to a stranded copy, graceful handling swaps a loud permanent error for a silent permanent leak. It probably wants the reclaim path from #207 landing first or alongside rather than going in on its own. Same note is on #207 for @ecv.

One correction to what I wrote on network-services-operator#326: I'd guessed the alert stayed quiet because success bursts pulled the ratio back under 0.5 and reset the for: 10m. Measured across 85 ten-minute windows that never happens — 0 of 57 data-bearing windows were at or below 0.5. The series just disappears in the 28 windows with no reconciles at all, and Alertmanager reads absence as resolved.