Fix project creation delays caused by slow deletions

[scotwells]

Note

CI shows 6 failing e2e tests — these are pre-existing failures unrelated to this PR. See #549 for the fix.

Summary

• Project creation could stall for up to 10 minutes while another project was being deleted. The controller now handles deletions without blocking other work.
• Deletion progress is visible via a new ResourceCleanup status condition on the project, so operators can see exactly what phase the cleanup is in.
• The controller now processes up to 4 projects concurrently instead of 1.

What changed

The project controller previously ran a synchronous cleanup operation during deletion that could block for up to 10 minutes. During that time, no other project — including newly created ones — could be reconciled.

The cleanup is now split into two non-blocking steps: issuing delete commands (fast) and checking whether resources have drained (cheap poll). The controller tracks progress via a ResourceCleanup condition with three reasons:

• CleanupStarted → delete commands are being issued
• CleanupAwaitingCompletion → waiting for resources to be removed
• CleanupComplete → all resources removed, finalizer can be released

If resources haven't drained after a check, the controller transitions back to CleanupStarted to automatically re-issue delete commands on the next reconcile.

Test plan

• Existing unit tests pass (task test:unit)
• Delete a project and verify the ResourceCleanup condition progresses through CleanupStarted → CleanupAwaitingCompletion → CleanupComplete
• Create a new project while another project is being deleted — creation should not be delayed
• Kill the controller pod mid-deletion and verify cleanup resumes after restart
• End-to-end tests pass (task test:end-to-end) — new project-deletion test passes in 20.7s

🤖 Generated with Claude Code

[joggrbot]

📝 Documentation Analysis

Joggr found 2 outdated docs in the pull request.

Autofix

Joggr opened 1 pull request(s) to fix the outdated docs.

• docs: for pull #553, fixed 2 outdated doc(s) #556

Outdated

file	reason	confidence
docs/api/resourcemanager.md	The API doc for the Project resource only lists a single 'Ready' status condition and reason, but the code adds a 'ResourceCleanup' condition type with new reasons. Users referencing only this document will not know about or be able to interpret the new condition types and reasons related to project deletion and cleanup.	74.3%
docs/architecture/controllers/project-controller/README.md	The code changes introduce more granular project resource cleanup status conditions (ResourceCleanup, CleanupStarted, CleanupAwaitingCompletion, CleanupComplete) and a multi-phase project deletion workflow not described in the document, which still presents only a basic finalizer/removal pattern. The documentation's deletion and cleanup flow omits the nuanced status and multi-reconciliation process now present in the implementation.	51.2%

---

✅ Latest commit analyzed: 07a0193 | Powered by Joggr

[ecv]

that joggr thing is kinda cool.