MIR-970: Unblock addon deprovisioning when provisioning saga is in-flight

[evanphx]

Summary

• Bump addon reconciler worker count from 1 → 4 so a long-running provisioning saga (e.g. valkey waiting on sandbox pool readiness) no longer blocks every other addon event. Same-entity serialization is still guaranteed by ReconcileController.inFlight.
• Add a pre-flight re-read at the top of provision(): if the association is no longer pending (e.g. user ran addon destroy during the crash/resync window) skip instead of re-starting a saga that would conflict with leftover resources.

Why

Observed sequence:

1. Reconciler starts provisioning rabbitmq and valkey.
2. Valkey saga blocks waiting on sandbox pool readiness — occupies the sole worker.
3. User runs addon destroy; status flips to deprovisioning.
4. Deprovisioning event sits in the work queue forever.

On restart, the resync re-enqueues stale pending events and re-triggers provisioning on associations the user had already destroyed, creating conflicting resources.

Test plan

• go test ./controllers/addon/ — new TestProvisionSkipsWhenAssociationNoLongerPending covers the stale-event path.
• go vet ./controllers/addon/... ./components/coordinate/...
• Smoke in make dev: install valkey + rabbitmq, confirm rabbitmq finishes while valkey is still waiting on its sandbox pool.

Out of scope

Coordinating saga Recover() with a concurrent reconciler-driven re-provision is tracked as a follow-up — the fixes here narrow the window significantly but do not fully close it.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 4eaa390b-a3c4-48f2-a90d-22afca0c7dd7

📥 Commits

Reviewing files that changed from the base of the PR and between d00e1a2 and 86c960a.

📒 Files selected for processing (3)

• components/coordinate/coordinate.go
• controllers/addon/controller.go
• controllers/addon/controller_test.go

---

📝 Walkthrough

Walkthrough

This pull request increases the concurrency of the addon association reconciliation controller from one to four workers. Additionally, the provisioning logic now includes a safeguard that re-fetches the current AddonAssociation by ID before transitioning its state. If the re-fetched association's status is no longer "pending", provisioning is skipped. If the re-fetch fails, an error is returned. A new test case validates that provisioning correctly skips when an association's status changes between initial retrieval and the provisioning step.

Warning

Review ran into problems

🔥 Problems

Git: Failed to clone repository. Please run the @coderabbitai full review command to re-trigger a full review. If the issue persists, set path_filters to include or exclude specific files.

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[phinze]

Any reason to go to 4 rather than 2 here? Two workers already solve the stated problem (one slow saga can't starve the queue), and each provisioning saga does a fair bit of entity store work (config versions, app versions, active version patches). Starting conservative and bumping later if someone actually hits the limit seems like the lower-risk move. Don't feel strongly tho, totally shippable as is!

[evanphx]

We already done 3 elsewhere and if these are slower, due to startup, felt like it was a good fit to reach for 4 for now.

[phinze]

SGTM!