Horizontal autoscaling, opt-in per deployment

CHANGELOG.md

@@ -14,6 +14,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   - **deploying into a namespace that does not exist now requires `namespaces:write`.** Ring still auto-creates the namespace, but only for a caller allowed to create one; a token holding `deployments:write` alone returns `403` instead of provisioning one implicitly. Roles are unaffected (`operator` and `admin` both hold the scope); add `namespaces:write` to any narrow PAT that relied on implicit creation, or create the namespace up front
 
 ### Added
+- Horizontal autoscaling, opt-in per deployment via an `autoscale` block (`min`, `max`, `target_cpu`): Ring adjusts the instance count from the average CPU per instance instead of holding `replicas` fixed. A deployment without the block never has its count changed by Ring, so an external controller can keep owning the count for its own deployments. `replicas` stays as the manifest declared it (the decision lives beside it), so re-running `ring apply` does not fight the autoscaler, and a rolling update carries the current capacity over to the new deployment. Decisions move one instance at a time, ignore CPU within 10 points of the target, wait 60s before adding and 300s before removing, and hold entirely when metrics are missing or older than 120s. Rejected for `kind: job`, for host networking with `max` above 1, and on the containerd runtime, which does not report CPU usage yet
 - Role-based access control: `admin` (full access), `operator` (read everything, write deployments, configs, secrets, volumes, webhooks and namespaces) and `viewer` (read-only). Changing an account's role revokes its sessions and tokens so the change takes effect immediately, and the last remaining admin can be neither demoted nor deleted (`409 Conflict`)
 - `volumes:read` and `volumes:write` scopes: the `/volumes` routes were not mapped to any scope, so deny-by-default made them admin-only and an `operator` could not manage the volumes of the workloads it administers
 

documentation/reference/api.md

@@ -219,6 +219,11 @@ Create a new deployment, or trigger a rolling update if one with the same `name`
   "namespace": "production",
   "kind": "worker",
   "replicas": 3,
+  "autoscale": {
+    "min": 2,
+    "max": 10,
+    "target_cpu": 70
+  },
   "image": "nginx:1.25",
   "labels": {
     "app": "nginx",
@@ -247,6 +252,8 @@ Create a new deployment, or trigger a rolling update if one with the same `name`
 
 Each port entry maps a host port (`published`) to a container port (`target`). Omit the field or pass `[]` to keep the container unpublished. Bindings are forwarded to Docker's `HostConfig.PortBindings`; a publish conflict is reported by Docker at start time.
 
+`autoscale` is optional and opt-in: omit it and the deployment holds exactly `replicas`, which Ring never changes on its own. When present, the response and `GET /deployments/{id}` also carry `desired_replicas`, the count the autoscaler is currently targeting — `replicas` keeps reporting what was declared. See [the manifest reference](/documentation/reference/manifest#autoscale) for the decision rules and limits.
+
 Environment values support two forms:
 
 - **Plain value**: `"KEY": "value"`, passed as-is to the container.
@@ -283,6 +290,10 @@ Environment values support two forms:
 | `ports` set with `replicas > 1` would race; surfaces on both fields          | `deployment.ports.replicas_conflict` + `deployment.replicas.ports_conflict` |
 | `kind: job` requires `replicas: 1`                                           | `deployment.replicas.job_must_be_one`                      |
 | `kind: job` doesn't take readiness checks                                    | `deployment.health_checks.job_readiness_unsupported`       |
+| `kind: job` cannot be autoscaled                                             | `deployment.autoscale.job_unsupported`                     |
+| `autoscale` needs `min >= 1`, `max >= min`, `0 < target_cpu < 100`           | `deployment.autoscale.invalid`                             |
+| `network.mode=host` forbids `autoscale.max > 1`                              | `deployment.autoscale.host_network_conflict`               |
+| `autoscale` is unsupported on containerd (no CPU metric yet)                 | `deployment.autoscale.runtime_unsupported`                 |
 | Environment keys must match `[A-Za-z_][A-Za-z0-9_]*`                         | `deployment.environment.key.invalid`                       |
 | `resources.{limits,requests}.{cpu,memory}` must parse                        | `deployment.resources.{limits,requests}.{cpu,memory}.invalid` |
 | `config.image_pull_policy` must be `Always`, `IfNotPresent`, or `Never`      | `deployment.config.image_pull_policy.unsupported`          |
@@ -832,7 +843,7 @@ Management routes require the `webhooks:write` scope (`webhooks:read` for `GET`)
 | `deployment.status_changed`     | A deployment transitions to a new status                             |
 | `deployment.health_check_failed`| A health check fails enough to trigger its `on_failure` action       |
 | `deployment.rolling_update`     | A rolling update progresses (instance drained / complete / failed)   |
-| `deployment.scaled`             | The reconciler added or removed an instance to reach `replicas`      |
+| `deployment.scaled`             | The reconciler added or removed an instance to reach the target count |
 | `deployment.error`              | The runtime failed to bring a deployment up (image, network, …)      |
 
 Every payload shares a common envelope (`schema_version`, `deployment_id`, `namespace`, `name`, `kind`) plus the per-kind fields below.

documentation/reference/cli.md

@@ -205,6 +205,8 @@ ring deployment list [OPTIONS]
 
 The table has ten columns: `Id`, `Created at (UTC)`, `Updated at (UTC)`, `Namespace`, `Name`, `Image`, `Runtime`, `Kind`, `Replicas` (formatted `instances/desired`), `Status`.
 
+On an [autoscaled](/documentation/reference/manifest#autoscale) deployment the desired count is the one the autoscaler is currently targeting, not the number declared in the manifest, and it is suffixed with `*` (`3/8*`). `ring deployment inspect` spells both out.
+
 Timestamps are rendered to the second (`2026-05-03 22:22:21`); sub-second
 digits and the `UTC` suffix are dropped from the cells since every Ring
 timestamp is UTC, as the column header says. The `json` output keeps the

documentation/reference/manifest.md

@@ -137,7 +137,8 @@ A map of deployment declarations. The map key is internal; Ring keys the deploym
 | Field | Type | Default | Description |
 |---|---|---|---|
 | `kind` | enum | `worker` | `worker` (long-running) or `job` (one-shot). On CH, a job moves to `completed` when the guest powers off cleanly; the workload's exit code is not surfaced. See [how-to: run a job](/documentation/how-to/run-a-job). |
-| `replicas` | integer | `1` | Number of instances. Jobs always run a single instance regardless. |
+| `replicas` | integer | `1` | Number of instances. Jobs always run a single instance regardless. When `autoscale` is set, this is the starting count, not a fixed one. |
+| `autoscale` | object | unset | Adjust the instance count from observed CPU. Opt-in: without it the count never changes on its own. See [autoscale](#autoscale). |
 | `command` | string list | `[]` | Override the image's entrypoint/CMD. **Docker only**, rejected at the API on the CH runtime. |
 | `environment` | map | `{}` | Environment variables, either plain values or `secretRef` references. See [environment](#environment). |
 | `volumes` | object list | `[]` | Volume mounts. See [volumes](#volumes). |
@@ -354,6 +355,47 @@ Both `limits` and `requests` are optional. Within each, `cpu` and `memory` are a
 
 > **Cloud Hypervisor:** `resources.limits.cpu` becomes the VM's vCPU count (minimum 1) and `resources.limits.memory` becomes the VM's RAM (minimum 128 MiB). `requests` is ignored on the CH runtime.
 
+## `autoscale`
+
+Let Ring pick the instance count from observed CPU instead of holding `replicas` fixed:
+
+```yaml
+replicas: 2          # starting count
+autoscale:
+  min: 2
+  max: 10
+  target_cpu: 70     # aim for 70% CPU per instance
+```
+
+| Field | Type | Description |
+|---|---|---|
+| `min` | integer | Never scale below this. Must be at least 1 — scaling to zero is not supported. |
+| `max` | integer | Never scale above this. Must be greater than or equal to `min`. |
+| `target_cpu` | number | Average CPU percentage **per instance** to aim for, between 0 and 100 (exclusive). |
+
+**Opt-in.** Without an `autoscale` block a deployment holds exactly `replicas` and Ring never changes it on its own. That is what lets an external controller (a CI job, a platform on top of Ring) own the count for its own deployments while Ring autoscales only what was explicitly handed to it.
+
+`replicas` keeps meaning "what this manifest asked for" and is never rewritten by the scheduler, so re-running `ring apply` does not fight the autoscaler. On a rolling update the current capacity carries over to the new deployment, so a redeploy under load does not drop you back to the starting count.
+
+### How it decides
+
+- **One instance at a time.** A decision moves the count by ±1, never straight to a computed target.
+- **A dead band.** CPU within 10 points of `target_cpu` counts as on-target, so ordinary jitter decides nothing.
+- **Asymmetric cooldowns.** At least 60s between adding instances, 300s before removing one. Shedding capacity slowly is what stops a load that oscillates around the target from driving the count up and down with it.
+- **No measurement means no decision.** If the runtime is unreachable, nothing is running yet, or the stats are stale (older than 120s), Ring holds the current count rather than scaling blind.
+
+### Limits
+
+> **Ring is single-node.** Autoscaling multiplies instances on **one machine**: it divides that machine's CPU more finely, it does not add capacity. It is meant to absorb spikes on a host with headroom, and to release resources when load drops. It cannot rescue a saturated host — if the machine is already at its limit, adding instances only increases contention. Size `max` against what the host can actually carry.
+>
+> Memory is still admitted per instance (see [resources](#resources)), so an autoscaled deployment that outgrows the host is stopped with `insufficient_resources` rather than taking the machine down.
+
+Rejected combinations, reported at `ring apply` time:
+
+- **`kind: job`** — a job runs once and exits, it has no steady-state CPU to aim at.
+- **`network.mode: host` with `max` above 1** — every instance would compete for the same host ports.
+- **the `containerd` runtime** — it does not report CPU usage yet, so a CPU target would be measured against a constant zero and walk the deployment down to `min`.
+
 ## `health_checks`
 
 A list of probe definitions. Each probe runs independently with its own counter and its own failure action. Three types: `tcp`, `http`, `command`.

migrations/20220101000024_autoscale.sql

@@ -0,0 +1,21 @@
+-- Horizontal autoscaling: adjust a deployment's instance count from its
+-- observed CPU usage instead of holding the fixed number from the manifest.
+--
+-- Two columns, on purpose:
+--
+--   `autoscale`         the policy the user declared (min/max/target_cpu), JSON,
+--                       NULL when the deployment is not autoscaled.
+--   `desired_replicas`  the autoscaler's current decision.
+--
+-- `replicas` keeps meaning "what the manifest asked for" and is never written
+-- by the autoscaler. Overwriting it would make the manifest lie about the
+-- running state and put `ring apply` in a tug-of-war with the scheduler: apply
+-- would reset the count on every run, the autoscaler would climb back, and
+-- neither would be wrong. Keeping the declared value and the current decision
+-- apart also makes both observable in the API.
+--
+-- NULL `desired_replicas` means "no decision yet" -- the runtimes fall back to
+-- `replicas`, so an autoscaled deployment starts from its declared count and
+-- the first tick refines it.
+ALTER TABLE deployment ADD COLUMN autoscale JSON DEFAULT NULL;
+ALTER TABLE deployment ADD COLUMN desired_replicas INTEGER DEFAULT NULL;

src/api/action/deployment/create.rs

@@ -81,6 +81,21 @@ fn validate_network_constraints(input: &DeploymentInput, errors: &mut ViolationL
             "deployment.replicas.host_network_conflict",
         ));
     }
+
+    // Same conflict, one step removed: an autoscaler allowed to reach more than
+    // one instance would recreate the situation the check above rejects.
+    if let Some(policy) = &input.autoscale
+        && policy.max > 1
+    {
+        errors.push(Violation::new(
+            "autoscale.max",
+            format!(
+                "host networking is incompatible with autoscale.max > 1 (got {}): all instances would compete for the same host ports",
+                policy.max
+            ),
+            "deployment.autoscale.host_network_conflict",
+        ));
+    }
 }
 
 /// Validate environment variable names against POSIX/Docker rules:
@@ -109,6 +124,37 @@ fn validate_environment(input: &DeploymentInput, errors: &mut ViolationList) {
 /// strings parse correctly. `parse_cpu_string` accepts forms like `"500m"` or
 /// `"2"`; `parse_memory_string` handles binary (`Ki`, `Mi`, …) and decimal
 /// (`K`, `M`, …) suffixes. Anything else used to be a silent runtime crash.
+/// Reject an autoscaling policy that cannot be satisfied (min below 1, max
+/// below min, a target outside 0-100). Rejecting at the API boundary keeps the
+/// scheduler free of "what does this even mean" cases.
+fn validate_autoscale(input: &DeploymentInput, errors: &mut ViolationList) {
+    let Some(policy) = &input.autoscale else {
+        return;
+    };
+
+    if let Err(message) = policy.validate() {
+        errors.push(Violation::new(
+            "autoscale".to_string(),
+            message,
+            "deployment.autoscale.invalid".to_string(),
+        ));
+    }
+
+    // containerd reports `cpu_usage_percent` as a hard-coded 0 (see
+    // src/runtime/containerd/stats.rs — a percentage needs two samples, and the
+    // sampling loop does not exist yet). A CPU-driven controller fed a constant
+    // zero reads "idle" forever and walks the deployment down to `min` whatever
+    // the real load. Refusing is the honest answer; silently scaling on a fake
+    // measurement is not.
+    if input.runtime == "containerd" {
+        errors.push(Violation::new(
+            "autoscale",
+            "the containerd runtime does not report CPU usage yet, so it cannot be autoscaled on a CPU target",
+            "deployment.autoscale.runtime_unsupported",
+        ));
+    }
+}
+
 fn validate_resources(input: &DeploymentInput, errors: &mut ViolationList) {
     let Some(resources) = &input.resources else {
         return;
@@ -320,6 +366,17 @@ fn validate_cross_field_constraints(input: &DeploymentInput, errors: &mut Violat
         ));
     }
 
+    // `kind: job + autoscale`: same reasoning as the replicas guard above, and
+    // there is nothing to measure anyway — a job has no steady-state CPU, it
+    // runs and exits, so a controller aiming at a CPU setpoint is meaningless.
+    if matches!(input.kind, DeploymentKind::Job) && input.autoscale.is_some() {
+        errors.push(Violation::new(
+            "autoscale",
+            "kind=job runs once and exits; it cannot be autoscaled",
+            "deployment.autoscale.job_unsupported",
+        ));
+    }
+
     // `kind: job + readiness check`: readiness gates a rolling update.
     // Jobs don't roll — they run once. A readiness flag here is a config
     // gap that would never trigger anything useful.
@@ -612,6 +669,8 @@ pub(crate) struct DeploymentInput {
     #[serde(default)]
     resources: Option<Resource>,
     #[serde(default)]
+    autoscale: Option<crate::models::deployments::Autoscale>,
+    #[serde(default)]
     ports: Vec<DeploymentPort>,
     #[serde(default)]
     network: Option<NetworkConfig>,
@@ -650,6 +709,7 @@ pub(crate) async fn create(
     validate_ports(&input, &mut violations);
     validate_environment(&input, &mut violations);
     validate_resources(&input, &mut violations);
+    validate_autoscale(&input, &mut violations);
     validate_config(&input, &mut violations);
     validate_cross_field_constraints(&input, &mut violations);
     if !violations.is_empty() {
@@ -712,6 +772,7 @@ pub(crate) async fn create(
     // - it has health checks configured
     // - --force flag is not set
     let mut rolling_parent_id: Option<String> = None;
+    let mut inherited_desired_replicas: Option<u32> = None;
     // Captured to log a `ForceReplace` event on the new deployment once
     // it exists. We collect the reason here so the caller of the API
     // gets a clear explanation for why rolling didn't happen, instead
@@ -761,6 +822,13 @@ pub(crate) async fn create(
                         existing.id
                     );
                     rolling_parent_id = Some(existing.id.clone());
+                    // Carry the parent's scaled-up capacity into the child.
+                    // Without this a redeploy silently drops an autoscaled
+                    // deployment back to the manifest count — a service running
+                    // 8 instances under load would restart at 2 and have to
+                    // climb again, one cooldown at a time, exactly when it is
+                    // least able to afford it.
+                    inherited_desired_replicas = existing.desired_replicas;
                 } else {
                     // Immediate replace. Pick the most specific reason so
                     // operators can fix the root cause: `force=true` is a
@@ -835,6 +903,13 @@ pub(crate) async fn create(
         volumes,
         health_checks: input.health_checks.unwrap_or_default(),
         resources: input.resources,
+        autoscale: input.autoscale.clone(),
+        // Inherited from the parent on a rolling update, so a redeploy keeps
+        // the capacity the autoscaler had reached. `None` for a fresh
+        // deployment: the first tick with usable metrics makes the first
+        // decision, and until then `target_replicas()` falls back to
+        // `replicas`.
+        desired_replicas: inherited_desired_replicas,
         image_digest: None,
         ports: input.ports,
         pending_events: vec![],