storage: project control plane storage layer does not scale with project count

[scotwells]

Problem

Milo's virtualized project control plane allocates dedicated storage infrastructure per project — watchcaches, etcd connections, informer factories — and never reclaims it. Every project that makes an API call permanently increases the server's memory and connection footprint until restart. This model works fine at small scale but degrades sharply as project count grows, and fails well before the scale a SaaS product needs.

etcd compounds the problem. A single etcd cluster has practical limits on concurrent watch streams, total key count, and write throughput. As project count grows, the number of per-project watch streams drives etcd toward these limits independently of the API server — and etcd's operational model (compaction, defragmentation, backup/restore) becomes increasingly difficult to manage as the dataset size grows with tenant count. At sufficient scale, a single etcd instance is not a viable storage backend regardless of how efficiently the API server uses it.

Success looks like

• Baseline performance benchmarks established so current scalability limits are understood and future improvements are measurable — covering both the API server and etcd
• A single Milo instance can serve tens of thousands of projects without degradation in memory, connection count, or API latency as project count grows
• Adding a new project does not allocate new storage infrastructure (connections, watchcaches, goroutine pools)
• The storage backend can scale independently of the API server, with a path to replace etcd if it becomes the bottleneck
• Existing projects continue to be fully isolated from one another

Context

kplane.dev is an open-source project that solves the same multi-tenant control plane problem at roughly 170× higher density. Their approach embeds tenant identity within the storage key rather than using per-tenant storage instances, which allows watchcaches and etcd connections to be shared across all tenants. Their kplane-dev/storage and kplane-dev/informer libraries directly address the bottlenecks Milo has today, and kplane-dev/kubernetes (a Kubernetes fork) unlocks shared watchcaches across tenants. kplane also provides kplane-dev/spanner, an alternative storage.Interface implementation backed by Google Cloud Spanner, which demonstrates a path beyond etcd for deployments that need to scale storage independently.

Delivering on the success criteria above will require evaluating and likely adopting parts of kplane's stack, designing a key layout migration for existing projects, understanding etcd's practical ceiling in Milo's deployment model, and determining how Milo's Organization → Project hierarchy maps onto kplane's tenant model.

References

• internal/apiserver/storage/project/ — storage mux and child lifecycle
• internal/controllers/garbagecollector/wiring.go — per-project informer factory pattern
• internal/controllers/resourcemanager/project_controller.go — ProjectControlPlane lifecycle
• kplane-dev/storage
• kplane-dev/informer
• kplane-dev/kubernetes
• kplane-dev/spanner

[savme]

Phase 1 update — resource utilization baseline

🟢 Baseline benchmarks for the API server and etcd are established.

Running TestProjectControlPlaneOverhead (#615) against a dev cluster with 20 projects:

metric	baseline	post-bootstrap	per-project
apiserver (go_memstats_heap_inuse_bytes)	111 MiB	371 MiB	+13.0 MiB
apiserver (go_memstats_sys_bytes)	184 MiB	485 MiB	+15.1 MiB
apiserver (go_goroutines)	1,654	20,698	+952
etcd (etcd_debugging_mvcc_watcher_total)	54	1,054	+50

Latency stayed flat at both baseline and post-bootstrap (p50=1ms, p99=3ms).

The test was capped at 20 projects because the dev deployment cannot sustain higher load without becoming unstable. Extrapolating linearly, 1K projects would require ~12 GiB of heap and ~950K goroutines from the API server alone, well before etcd watcher limits become the binding constraint.

Cross-reference: staging (244 ProjectControlPlanes, live as of 2026-05-12):

metric	per-pod (avg)	rough per-project
apiserver (go_memstats_heap_inuse_bytes)	4,778 MiB	~19 MiB
apiserver (go_goroutines)	274,510	1,118
etcd (etcd_debugging_mvcc_watcher_total)	13,493	~55

Per-project estimates broadly corroborate the benchmark: goroutines (~1,118 vs ~952) and etcd watchers (~55 vs ~50) are in the same range; heap is somewhat higher (~19 MiB vs ~13 MiB), likely reflecting longer-running accumulated state or a different resource mix per project in staging.

For reference, kplane/bench reports ~2 MiB heap and ~10 goroutines per VCP at steady state across a 200-VCP run. kplane's architecture is different enough (shared watchcaches, tenant identity embedded in storage keys) that this isn't an apples to apples comparison, but it sets a reference point for what the current per-project overhead costs relative to an alternative approach. No architecture decisions have been made.

Next: Extend benchmarks to better understand inactive project eviction. I'll use that data to map out optimization opportunities vs. their complexity. No timeline on architectural decisions until we have that data in hand.