Integrate compute service with Milo quota system

[scotwells]

Parent Issue

Tracked by datum-cloud/enhancements#682 (Launch Workload Compute Service — "UFOs")

Summary

Integrate the compute service with the Milo quota service (quota.miloapis.com) so that resource consumption is tracked and enforced per project. This covers Instance, WorkloadDeployment, and the underlying compute dimensions (instance count, vCPUs, memory, and eventually GPUs).

Motivation

Without quota integration, there is no mechanism to:

• Enforce per-project resource limits set by the pricing tier
• Give users visibility into how much of their quota they are consuming
• Prevent runaway workloads from monopolizing platform capacity
• Gate new projects to safe default limits during onboarding

Resources to Quota

The following resource dimensions need ResourceRegistration entries:

Resource Type	Unit	Description
compute.datumapis.com/instances	count	Total running instances per project
compute.datumapis.com/vcpus	millicores	Total vCPU allocation across all instances
compute.datumapis.com/memory	MiB	Total memory allocation across all instances
compute.datumapis.com/gpus	count	GPU units (future — phase 2)

Integration Pattern

Compute should use Pattern 2: Service-Managed Claims (not admission-blocking ClaimCreationPolicy). Reasons:

• WorkloadDeployment creates Instance objects ahead of scheduling — creation should not be blocked, but provisioning should be gated
• Quota denial state needs to be visible to users as a resource condition, not a silent API error
• Auto-scaling scenarios need instances to exist in a pending state while quota is being negotiated

The instance controller creates a ResourceClaim when an Instance is created, watches for Granted/Denied on the claim, and gates actual VM provisioning on claim grant.

The ConsumerRef on each ResourceClaim will reference the Project the instance belongs to.

User Experience: Quota Exceeded States

This is the primary UX concern to resolve before implementation.

Scenario 1: User creates an Instance that exceeds their quota

$ kubectl apply -f my-instance.yaml
instance.compute.datumapis.com/my-instance created   # Creation succeeds

The instance is created, but it immediately surfaces a non-ready condition:

$ kubectl get instance my-instance
NAME          READY   REASON          NETWORK IP   EXTERNAL IP
my-instance   False   QuotaExceeded                

$ kubectl describe instance my-instance
...
Conditions:
  Type           Status  Reason          Message
  ----           ------  ------          -------
  QuotaGranted   False   QuotaExceeded   Quota for compute.datumapis.com/vcpus exceeded: 
                                         requested 4 vCPUs, 0 available (16/16 vCPUs used)
  Programmed     False   PendingQuota    Waiting for quota claim to be granted
  Running        False   PendingQuota    Waiting for quota claim to be granted
  Ready          False   PendingQuota    Waiting for quota claim to be granted

Open questions:

• Should the instance be auto-deleted after some TTL when quota is denied, or left in place for the user to clean up?
• Should we surface a link/annotation pointing the user to quota management UI or docs?

Scenario 2: WorkloadDeployment scales up and partially exceeds quota

If a WorkloadDeployment has replicas: 10 but only 3 additional instances fit within quota:

$ kubectl get workloaddeployment my-wld
NAME      REPLICAS  READY  DESIRED  UP-TO-DATE
my-wld    7         7      10       7

The 3 new instances exist but are stuck in QuotaExceeded state. The WorkloadDeployment status should reflect the shortfall and surface a condition:

Conditions:
  Type             Status  Reason          Message
  ----             ------  ------          -------
  ReplicasReady    False   QuotaExceeded   3 of 10 desired replicas are pending quota 
                                           (compute.datumapis.com/instances: 7/7 used)

Open questions:

• Should quota-blocked instances count against currentReplicas or be in a separate status field?
• Should the deployment controller stop creating new quota-blocked instances once the first denial is observed, or keep trying (important for when quota is later increased)?

Scenario 3: User has quota restored (upgrade or admin grant)

When a project's quota is increased (tier upgrade, manual grant), the pending ResourceClaim should be re-evaluated automatically by the quota system. The instance controller watches the claim and resumes provisioning when Granted=True flips — no user action needed.

This should be tested explicitly: quota increase → instances automatically begin provisioning without user intervention.

Scenario 4: Instance deleted before claim is granted

When a QuotaExceeded instance is deleted, the ResourceClaim (owned by the instance) is garbage collected. No quota was ever consumed, so no release is needed. The controller should handle this gracefully and not attempt to provision after deletion begins.

Status Condition Design

Add QuotaGranted as a new condition type on Instance:

Condition	Status	Reason	Meaning
QuotaGranted	Unknown	PendingEvaluation	Claim created, awaiting evaluation
QuotaGranted	True	QuotaAvailable	Quota allocated, provisioning may proceed
QuotaGranted	False	QuotaExceeded	Quota denied — instance will not be provisioned
QuotaGranted	False	ValidationFailed	Misconfiguration in quota policy (platform error)

The QuotaGranted=False state should cascade to block Programmed, Running, and Ready conditions from ever becoming True.

Error Message Quality

Messages on denied claims (sourced from ResourceClaim.status.allocations[].message) should be user-actionable. The goal is for a user to understand:

1. What was exceeded (vCPUs, instance count, memory)
2. How much they're currently using vs. their limit
3. What to do — upgrade tier, delete unused resources, or contact support

Example good message:

"Quota for compute.datumapis.com/vcpus exceeded: requested 4, available 0 (16/16 vCPUs allocated). To increase your limit, upgrade your plan or contact support."

Example bad message (avoid):

"QuotaExceeded"

Operator Alerting

As operators, we need internal visibility when consumers are approaching or exceeding their quota limits — both to proactively reach out to users who may be hitting growth ceilings, and to catch misconfigured grants or runaway usage before it becomes a support escalation.

This requires surfacing AllowanceBucket consumption data into our observability stack. Concretely:

• Metrics: Export per-project, per-resource-type utilization from AllowanceBucket status as Prometheus metrics (e.g., quota_used, quota_limit, quota_utilization_ratio). This enables alerting rules like "any project above 80% vCPU utilization for more than 10 minutes."
• Alerts: Define alerting thresholds at ~80% and 100% per quota dimension. The 80% threshold gives the ops team and GTM a lead time to reach out to users approaching upgrade triggers before they hit a hard wall.
• Dashboards: An operator-facing view showing quota utilization across all projects, sortable by utilization ratio, to identify which projects are closest to their limits at a glance.

Open questions:

• Where do these metrics live — in the compute service's observability stack, or should Milo expose AllowanceBucket metrics directly?
• What is the right escalation path when the 100% threshold fires — automated notification to the user, a GTM alert, or both?
• Should operators be able to configure per-tenant alert thresholds (e.g., VIP customers get alerted at 70%)?

Implementation Tasks

• Register ResourceRegistration resources for each quota dimension above (consumerType: Project)
• Create GrantCreationPolicy to provision default quota grants when a Project is created (tier-based defaults TBD with GTM)
• Add QuotaGranted condition type to Instance API
• Update instance_controller.go to create/watch ResourceClaim with consumerRef pointing to the instance's Project, and gate provisioning on grant
• Update workloaddeployment_controller.go to surface quota-shortfall condition when instances are quota-blocked
• Export AllowanceBucket utilization as Prometheus metrics for operator alerting
• Define alerting rules at ~80% and 100% utilization per quota dimension
• Write integration tests covering: quota granted flow, quota exceeded flow, quota increase unblocks pending instances
• Determine default quota values per tier (requires GTM input)

Open Questions

1. Claim namespace: Where do ResourceClaim objects live — in the same namespace as the Instance, or in a dedicated quota-system namespace?
2. Partial WorkloadDeployment scaling: Should the controller back off from creating more instances after the first quota denial, or create all desired instances and let them all show quota status?
3. Auto-cleanup TTL: Should quota-denied instances be auto-deleted after a period, or left indefinitely for user cleanup?
4. Quota visibility API: Should there be a read API (or AllowanceBucket surfacing) so users can check remaining quota before attempting to create resources?
5. Tier defaults: What are the default vCPU/instance/memory limits per tier? (Needs GTM/commercial input)
6. Operator alerting ownership: Do AllowanceBucket metrics come from Milo directly, or does the compute service own this instrumentation?
7. Operator alert escalation: When a project hits 100% quota, what is the automated response — user notification, GTM alert, both?
8. Configurable operator thresholds: Should alert thresholds be configurable per project (e.g., VIP customers alerted earlier)?

Related

• Milo quota API
• capability-quota skill — implementation patterns
• Billing and Metering for Datum Cloud using Integration of 3rd Party Metering Service enhancements#681 — Metering integration (quota and metering will share resource dimensions)

[scotwells]

Competitive Research: Free Tier Quota Limits

Research across 10 providers to inform our quota UX design. Findings independently cross-checked by a second review pass; corrections noted inline.

---

Summary Table

Provider	Quota Scope	Hard Block or Soft?	UX When Exceeded
GCP	Billing account (Always Free) + per-project per-region	Hard block (403)	403 QUOTA_EXCEEDED; no auto-upgrade prompt
AWS	Per-region per-account	Hard block (vCPU quota) or silent billing (free hours)	VcpuLimitExceeded error; free-hour overages just start billing
Azure	Per-subscription per-region	Hard block	OperationNotAllowed / ResourceQuotaExceeded; VM size shown as "not available" in UI
Railway	Account-wide (spend limit) + per-service (CPU/RAM)	Hard stop when spend limit hit	Services go offline; email at 75%/90%/100% of spend limit
Koyeb	Organization-level (instance count)	Hard block on second instance; sleep on inactivity	Inline dashboard error; auto sleep-to-zero after 1hr no traffic
Cloudflare Workers	Account-level (req/day) + per-Worker (CPU ms)	Hard block (Error 1027 / 1102)	End users see error page; no proactive alerts
Vercel Hobby	Account-level (monthly rolling)	Feature suspension for rest of month	App goes offline until window resets; proactive warning emails sent before limit
Fly.io	Per-organization	Hard stop when trial hours exhausted	Apps stop immediately; no billing alerts
Render	Workspace-level (hours/month) + per-service (CPU/RAM)	Hard suspension when hours exhausted	Email warnings approaching limit; services suspended for rest of month
Oracle Cloud	Tenancy-level	Hard block (LimitExceeded) or capacity shortage	Console error; free accounts cannot request increases without upgrading

---

Key Findings by Provider

GCP

• 1 × e2-micro (2 shared vCPUs, 1 GB RAM) per billing account in us-west1/us-central1/us-east1 only
• GPU quota set to 0; increase requests blocked during the Free Trial period; Always Free has no GPU offer at all
• Quotas are billing-account-level for Always Free, but per-project per-region for Service Quotas
• Enforcement: HTTP 403 QUOTA_EXCEEDED (not 429) with message like "Quota 'CPUS' exceeded. Limit: X.0 in region us-central1"
• No auto-upgrade prompt at moment of failure; manual quota increase request required (24–48h approval)

AWS

• 750 hrs/month t2.micro or t3.micro for accounts created before July 15, 2025; accounts after July 2025 get a 6-month window with a broader set of instance types (t3.micro, t3.small, t4g.micro, t4g.small, c7i-flex.large, m7i-flex.large) — still hour-based, not a separate credit model
• Default vCPU quota: 5 vCPUs per region for standard families; new accounts may start as low as 1 vCPU as an anti-abuse throttle
• GPUs: 0 vCPUs by default; must explicitly request an increase
• Free-hour overages on older accounts silently start billing — no hard block, no warning at the moment of overage
• vCPU quota violations return VcpuLimitExceeded with a message stating current vs. allowed vs. requested
• AWS automatically emails at 85% of free-tier usage (enabled by default)

Azure

• 750 hrs/month B1s VM (1 vCPU, 1 GB RAM) for 12 months; also eligible: B2pts v2 and B2ats v2
• Two-tier vCPU quota: 4 total regional vCPUs + per-VM-family caps (many families set to 0 by default)
• GPUs: quota 0 on all free/student/trial subscriptions; not raiseable without upgrading to Pay-As-You-Go
• Quota is per-subscription per-region (not resource-group-level)
• Enforcement: hard block with OperationNotAllowed (vCPU exceeded) or ResourceQuotaExceeded (resource count), both including current/max/requested values
• Free/trial accounts cannot request quota increases at all — must upgrade to Pay-As-You-Go first

Railway

• No permanent free tier; 30-day trial with $5 in credits — no credit card required to start
• Trial resource cap: 1 GB RAM account-wide (shared across services) with shared vCPU; not a per-service hard limit
• Account-wide spend hard limit (minimum $10); all services go offline when hit
• Proactive email warnings at 75%, 90%, 100% of spend limit
• Services pause (not deleted) on exhaustion; data preserved — but trial account volumes are deleted 30 days after credit expiry if not upgraded

Koyeb

• Permanent free tier: 1 web service per org (0.1 vCPU, 512 MB RAM); no worker services, no volumes
• Automatic scale-to-zero after 1 hr of no traffic; 1–5s cold start from deep sleep
• Attempting a second free instance returns an inline blocking error
• Quota is organization-scoped (not per-project)
• No proactive warnings; errors surface reactively at the point of action
• No billing surprises — users without a card cannot be charged

Cloudflare Workers

• 100,000 requests/day (account-level); 10 ms CPU per request; 128 MB memory per isolate; 100 Workers max
• End users hit Error 1027 when daily request quota exhausted
• Per-request CPU exceeded returns Error 1102; execution is terminated — though infrequent violations may be tolerated before termination kicks in
• No throttling — hard stop only; no overage billing
• Configurable fail-open mode: route bypasses Worker and hits origin instead of serving the error page
• No proactive email alerts for approaching free limits

Vercel Hobby

• 1M function invocations/month; 4 CPU-hours/month; 100 GB bandwidth/month (account-level, 30-day rolling)
• 100 deployments/day; 1 concurrent build
• Enforcement: full feature suspension for the remainder of the billing window when a limit is hit
• Deployment rate limit returns HTTP 429 (retry-after header presence is unverified)
• Does send proactive "approaching your limits" warning emails before limits are hit
• Non-commercial personal use only; commercial use can result in account suspension

Fly.io

• No permanent free tier — new accounts get a free trial: 2 VM-hours total runtime, 7 calendar days, max 10 machines; legacy users on deprecated plans retain old allowances
• Trial machines auto-stop after 5 minutes of runtime to conserve the 2-hour pool
• Auto-stop/auto-start for paid machines is opt-in via fly.toml; suspend mode preserves memory state (hundreds of ms resume); stop mode requires full boot (several seconds)
• Apps stop immediately when trial hours or 7 days expire; payment method required to continue
• No billing alerts — Fly.io docs acknowledge this gap; users must check the dashboard manually
• Quota is per-organization; limit on how many orgs one account can create
• GPUs deprecated entirely; unavailable after August 1, 2026

Render

• Per-service: 0.1 vCPU (shared), 512 MB RAM
• Workspace-level: 750 instance hours/month, 100 GB bandwidth/month, 500 build minutes/month
• Spin down after 15 minutes of no inbound traffic; cold start ~30–60 seconds (Render shows a loading page during restart)
• On hour exhaustion: all free services in workspace are suspended for the rest of the month
• Bandwidth overage: billed if payment method on file; suspended if not
• Email warnings sent when approaching a limit; dashboard messaging reportedly lacks specificity about which threshold triggered suspension
• GPUs not available on free tier

Oracle Cloud (Always Free)

• Ampere A1 Flex (ARM): 4 OCPUs + 24 GB RAM total per tenancy, configurable across up to ~4 instances; expressed as 3,000 OCPU-hours + 18,000 GB-hours/month
• AMD Micro: 2 instances per tenancy, each 1/8 OCPU (burstable) + 1 GB RAM
• GPUs: not included on Always Free; require paid account
• Two distinct failure modes:
  • Quota exceeded (LimitExceeded): hard API block; free accounts cannot request increases — must upgrade
  • Out of Host Capacity: separate from quota; regional ARM capacity shortage (high demand for generous free tier); no waiting list — users must retry, often via automation
• Limits are tenancy-scoped; all Always Free compute must be in the home region
• Idle reclamation: instances with CPU, network, and memory all below 20% at the 95th percentile for 7 days can be reclaimed with warning emails
• Abuse prevention: card required at signup (no prepaid/virtual cards); one tenancy per email address; no support escalation path on free tier

---

Patterns Relevant to Datum Compute

Enforcement style: All hyperscalers (GCP, AWS, Azure, Oracle) use hard blocks at the API level with machine-readable error codes and human-readable messages showing current/limit/requested. This is table stakes — users expect it.

Quota scope: Hyperscalers scope quotas per-project/subscription per-region. Smaller platforms (Koyeb, Railway, Render) scope at the org/workspace/account level. Our per-project design aligns with the hyperscaler model.

Error message quality: GCP, AWS, and Azure all surface the specific resource type, current usage, and the limit in their error messages. This validates our requirement for actionable messages.

Proactive warnings: AWS (85%), Railway (75%/90%/100%), Vercel (approaching limit), and Render (approaching limit) all send proactive warnings before quota is exhausted. We should emit a warning condition or notification when a project reaches ~80% of a quota dimension — this is the market norm, not an exception.

Auto-cleanup vs. leave-in-place: No provider auto-deletes resources on quota exhaustion. Railway, Render, and Vercel all leave services in a suspended/paused state. This supports leaving quota-denied instances in a QuotaExceeded condition for user cleanup rather than a TTL-based auto-delete.

Sleep/suspend patterns: Koyeb, Render, and Fly.io all implement auto-sleep on idle as their primary free-tier resource management mechanism. Koyeb's 1hr and Render's 15min thresholds are notable reference points if we ever implement scale-to-zero.

GPU quota = 0 by default: Every provider either excludes GPUs entirely from free tier or sets quota to 0 with no increase path. We should do the same in our GrantCreationPolicy defaults.

Billing surprise risk: AWS is the standout outlier — free-hour overages silently convert to billing. Every other provider either hard-blocks (GCP, Azure, Oracle, Cloudflare, Fly) or suspends (Vercel, Render, Railway). Our model should hard-block or suspend, never silently bill.

[scotwells]

PR up at #104 (draft).

Implemented Pattern 2 (service-managed claims) as designed — instances create immediately, provisioning holds on a ResourceClaim until quota is granted. Three dimensions tracked per project: instance count, vCPUs, and memory. Self-recovery is wired up, so pending instances resume automatically when quota is increased with no user action needed.

A few decisions made along the way:

• No auto-delete TTL for quota-denied instances — they stay put until the user removes them
• Flat default grant (10 instances / 40 vCPUs / 80 GiB) per project for now — tier-based grants are out of scope until GTM finalizes limits
• Operator alerting deferred — better handled holistically across services

Still needs: quota component wired into the target environment overlay, and final quota defaults confirmed with the commercial team before going live.