adapter: make WAIT FOR roll back on timeout

doc/developer/design/20260522_cluster_autoscaling.md

@@ -156,7 +156,7 @@ An `ALTER CLUSTER SET (...)` that changes a replica's **config shape** writes th
 **Per-`ALTER` timeout:** The current `ALTER CLUSTER ... SET (...) WITH (WAIT ...)` syntax is retained unchanged and repurposed to drive the durable mechanism. Two forms:
 
 - `WAIT UNTIL READY (TIMEOUT '<dur>', ON TIMEOUT COMMIT|ROLLBACK)` — the hydration-aware form. `TIMEOUT` is added to the transaction timestamp and written as the `reconfiguration` record's `deadline`; `ON TIMEOUT` is written as its `on_timeout`. The controller cuts over once the target replicas hydrate; only if the deadline passes un-hydrated does it apply `on_timeout`.
-- `WAIT FOR '<dur>'` — retained as sugar for `WAIT UNTIL READY (TIMEOUT '<dur>', ON TIMEOUT COMMIT)`. Its legacy contract was "wait the fixed duration, then cut over regardless of hydration"; a change from the legacy behavior is that the controller cuts over the instant the target hydrates, so we drop the "wait the *full* duration even when ready early" property, which carries no value in a background model. The blind cut-over-at-the-deadline it implied is exactly `ON TIMEOUT COMMIT`. A `WAIT FOR '0s'` (equivalently `TIMEOUT '0s'` with `ON TIMEOUT COMMIT`) thus requests an immediate cut-over: the deadline is already past, so the controller commits on its first pass without ever provisioning overlap replicas — the non-graceful, drop-and-recreate resize, now an explicit opt-in rather than the no-`WAIT` default.
+- `WAIT FOR '<dur>'` is retained as sugar for `WAIT UNTIL READY (TIMEOUT '<dur>', ON TIMEOUT ROLLBACK)`. The controller cuts over once the target hydrates, including when it observes hydration in the same tick that notices an expired deadline. If the deadline is reached and the target remains unhydrated, it rolls back the reconfiguration and keeps the realized cluster shape. Users who want a forced cut-over can request it explicitly with `WAIT UNTIL READY (TIMEOUT '<dur>', ON TIMEOUT COMMIT)`.
 
 `ON TIMEOUT` defaults to **`ROLLBACK`** when omitted. This is the safe, conservative choice that never surprises a user with downtime by cutting over to a not-yet-hydrated target. This is a change from today's implicit `COMMIT` default, applied uniformly (the foreground wait shim picks it up too). Omitting `WITH (WAIT ...)` entirely on a reconfiguring `ALTER` falls back to the `default_cluster_reconfiguration_timeout` dyncfg for the deadline (so a transition is never unbounded) and the default `ROLLBACK` action. The session-side wait shim does not run a timer of its own; it watches the durable outcome and returns when the controller resolves. `Finalized` is success if the realized config reached the awaited target. `Cancelled` is success for the ALTER that cancelled back to the realized config. `TimedOut` and `ResourceExhausted` return the existing ALTER timeout error. There is no shim-side timeout at all: the controller resolves the reconfiguration by its deadline, and the shim simply waits for that.
 
@@ -228,7 +228,7 @@ The following behaviors fall out of the design rather than being its headline ou
 
 - **Burst and reconfiguration-transient replicas appear in billing and metering identically to ordinary replicas.** A user with an `AUTO SCALING STRATEGY` set sees additional billing during hydration windows (at the configured `HYDRATION SIZE`); a user issuing a background `ALTER CLUSTER` sees additional billing during the overlap between the old and new replica sets. This is not new, replicas are always billed except if we create them as unbilled replicas. Just calling out for completeness, to make it clear that burst or reconfiguration replicas are not free.
 - **Background `ALTER CLUSTER` returns immediately** after writing the `reconfiguration` record with `status = InProgress` to the catalog. The actual replica transition happens asynchronously and is observable via the new introspection view. Note a deliberate difference from how some other async DDL reads back: the cluster's realized config (`cluster.size`, `SHOW CREATE CLUSTER`, `mz_clusters.size`) keeps showing the **pre-reconfiguration** config until the controller cuts over. The realized config is the committed steady-state config, not a target, and the pending target is surfaced separately (the introspection view, `SHOW CLUSTERS`).
-- **A plain `ALTER CLUSTER SET (SIZE = ...)` (no `WITH (WAIT ...)`) is now graceful and non-disruptive by default.** Today a bare config-shape `ALTER` without `WITH (WAIT ...)` tears the cluster's replicas down and recreates them at the new shape, so the cluster is unavailable while the new replicas rehydrate. The new model runs the graceful overlap path instead — old replicas keep serving until the new ones hydrate and cut over — at the cost of the transient double billing noted above. This flip should be called out in release notes. A user who wants the old immediate behavior — cut over at once and accept the rehydration gap, avoiding the overlap cost — can request it explicitly with `WITH (WAIT FOR '0s')` (equivalently `WAIT UNTIL READY (TIMEOUT '0s', ON TIMEOUT COMMIT)`): the deadline is already in the past, so the controller commits the cut-over on its first pass without provisioning overlap replicas.
+- **A plain `ALTER CLUSTER SET (SIZE = ...)` (no `WITH (WAIT ...)`) is now graceful and non-disruptive by default.** Today a bare config-shape `ALTER` without `WITH (WAIT ...)` tears the cluster's replicas down and recreates them at the new shape, so the cluster is unavailable while the new replicas rehydrate. The new model runs the graceful overlap path instead. Old replicas keep serving until the new ones hydrate and cut over, at the cost of the transient double billing noted above. This flip should be called out in release notes. A user who wants the old immediate behavior can cut over at once and accept the rehydration gap, avoiding the overlap cost, by explicitly requesting `WAIT UNTIL READY (TIMEOUT '0s', ON TIMEOUT COMMIT)`. The deadline is already in the past, so the controller commits the cut-over on its first pass without provisioning overlap replicas.
 - **`SHOW CLUSTERS` gains reconfiguration visibility.** The command is extended to surface both the current size (the realized `cluster.size`) and the target size (the latest `reconfiguration` record, when one exists), plus an indication of whether a reconfiguration is in progress, done, cancelled, timed out, or resource exhausted. This is a change from today's behavior, where `SHOW CLUSTERS` shows only the old size until the graceful reconfiguration finalizes and there is no in-band way to ask "is a reconfig in flight". See [Observability](#observability).
 - **`WITH (WAIT ...)` is now durable, and the implicit `ON TIMEOUT` default flips from `COMMIT` to `ROLLBACK`.** Today's `WAIT UNTIL READY (TIMEOUT ...)` was session-bound: closing the session aborted the reconfiguration. In the new model the deadline and the `ON TIMEOUT` action are written to the catalog as the `reconfiguration` record's `deadline` and `on_timeout`; closing the session no longer aborts anything, and the controller enforces both regardless of session lifetime. The session-side wait shim reflects the controller's durable status rather than running its own timer, so the in-session UX of "ALTER returns or errors within the timeout" is unchanged. Two changes worth release-noting: (1) the implicit `ON TIMEOUT` default is now **`ROLLBACK`** rather than `COMMIT`. A timed-out reconfiguration with no explicit action reverts to the pre-reconfiguration shape instead of cutting over to a possibly-unhydrated target. This is the safe default that never silently induces downtime, and it applies to the foreground path too. (2) On a `ROLLBACK` expiry the cluster keeps running at its pre-reconfiguration size (the realized `cluster.size`, which never moved). The timeout is recorded in the audit log and in the retained latest record, and the user re-issues `ALTER` to retry or change course.
 

doc/user/content/sql/alter-cluster.md

@@ -191,9 +191,11 @@ Customizing the resize timeout with `WAIT UNTIL READY` or `WAIT FOR`
   SET (SIZE = '100cc') WITH (WAIT UNTIL READY (TIMEOUT = '10m'));
   ```
 
-- `WAIT FOR '<duration>'` sets the timeout and commits when it expires,
-  regardless of hydration status, which can cause downtime. Prefer
-  `WAIT UNTIL READY`.
+- `WAIT FOR '<duration>'` is equivalent to `WAIT UNTIL READY (TIMEOUT =
+  '<duration>', ON TIMEOUT = 'ROLLBACK')`. Materialize cuts over once the target
+  replicas hydrate. When Materialize processes an expired timeout, it
+  rolls back the resize and keeps the current size if the target replicas are
+  still unhydrated.
 
 See [Monitoring a resize](#monitoring-a-resize) to track progress and
 [cancel](#monitoring-a-resize) an in-flight resize.

doc/user/data/examples/alter_cluster.yml

@@ -97,7 +97,7 @@
         | Option  | Description |
         |--------|-------------|
         | `WAIT UNTIL READY(...)`    | ***Private preview.** This option has known performance or stability issues and is under active development.* {{< include-from-yaml data="examples/alter_cluster" name="wait-until-ready-cmd-option" >}} |
-        | `WAIT FOR` |  ***Private preview.** This option has known performance or stability issues and is under active development.* A fixed duration to wait for the new replicas to be ready. This option can lead to downtime. As such, we recommend using the `WAIT UNTIL READY` option instead.|
+        | `WAIT FOR` |  ***Private preview.** This option has known performance or stability issues and is under active development.* Equivalent to `WAIT UNTIL READY` with `ON TIMEOUT = 'ROLLBACK'`. Materialize cuts over once the new replicas hydrate. When Materialize processes an expired timeout, it rolls back the resize and keeps the current size if the target replicas are still unhydrated.|
 
 - name: "syntax-reset-to-default"
   code: |

src/adapter/src/coord/sequencer/inner/cluster.rs

@@ -756,8 +756,7 @@ impl Coordinator {
         //                          user set on the reconfiguration in progress.
         //   - no `WAIT`, nothing in flight -> the system-default timeout and the
         //                          implicit `on_timeout` default (`ROLLBACK`).
-        //   - `WAIT FOR`        -> sugar for `ON TIMEOUT COMMIT` (cut over at the
-        //                          deadline regardless of hydration).
+        //   - `WAIT FOR`        -> sugar for `ON TIMEOUT ROLLBACK`.
         //   - `WAIT UNTIL READY -> the explicit `TIMEOUT` / `ON TIMEOUT`, with
         //                          `ON TIMEOUT` defaulting to `ROLLBACK` when
         //                          omitted.
@@ -785,7 +784,7 @@ impl Coordinator {
                 ),
             },
             AlterClusterPlanStrategy::For(timeout) => {
-                (deadline_from(*timeout), OnTimeoutAction::Commit)
+                (deadline_from(*timeout), OnTimeoutAction::Rollback)
             }
             AlterClusterPlanStrategy::UntilReady {
                 timeout,

src/cluster-controller/src/strategy.rs

@@ -282,12 +282,12 @@ impl Strategy for GracefulReconfigurationStrategy {
         //      `Commit` (cut over to the not-yet-hydrated target anyway).
         //
         // NOTE: the deadline is reached at `now >= deadline`, not `now > deadline`.
-        // A `WAIT FOR '0s'` writes `deadline = now` to request an immediate
-        // cut-over. With a strict `>`, a first tick landing at exactly that
-        // timestamp would miss the deadline, so phase 2 would provision the overlap
-        // target replicas and only a later tick would cut over. `>=` fires the
-        // deadline the instant it is reached, so the zero-timeout cut-over happens
-        // on the first tick, before any overlap replica is desired.
+        // An `ON TIMEOUT COMMIT` with a zero timeout writes `deadline = now` to
+        // request an immediate cut-over. With a strict `>`, a first tick landing at
+        // exactly that timestamp would miss the deadline, so phase 2 would provision
+        // the overlap target replicas and only a later tick would cut over. `>=`
+        // fires the deadline the instant it is reached, so the zero-timeout cut-over
+        // happens on the first tick, before any overlap replica is desired.
         let hydrated = self.target_hydrated(state, signals, record);
         let deadline_reached = now >= record.deadline;
         let commit_on_timeout = deadline_reached && matches!(record.on_timeout, OnTimeout::Commit);

test/cloudtest/test_managed_cluster.py

@@ -220,7 +220,7 @@ def wait_for_replica_names(names: list[str], cluster: str = "zdtaltertest"):
     # (r1 -> r2).
     mz.environmentd.sql(
         """
-        ALTER CLUSTER zdtaltertest SET ( SIZE = 'scale=1,workers=2' ) WITH ( WAIT FOR '1ms' )
+        ALTER CLUSTER zdtaltertest SET ( SIZE = 'scale=1,workers=2' ) WITH ( WAIT UNTIL READY (TIMEOUT '1ms', ON TIMEOUT 'COMMIT') )
         """,
         port="internal",
         user="mz_system",
@@ -236,7 +236,7 @@ def wait_for_replica_names(names: list[str], cluster: str = "zdtaltertest"):
     # target replicas.
     mz.environmentd.sql(
         """
-        ALTER CLUSTER zdtaltertest SET ( SIZE = 'scale=1,workers=1', REPLICATION FACTOR 2 ) WITH ( WAIT FOR '1ms' )
+        ALTER CLUSTER zdtaltertest SET ( SIZE = 'scale=1,workers=1', REPLICATION FACTOR 2 ) WITH ( WAIT UNTIL READY (TIMEOUT '1ms', ON TIMEOUT 'COMMIT') )
         """,
         port="internal",
         user="mz_system",
@@ -247,7 +247,7 @@ def wait_for_replica_names(names: list[str], cluster: str = "zdtaltertest"):
     # oldest replica is kept.
     mz.environmentd.sql(
         """
-        ALTER CLUSTER zdtaltertest SET ( SIZE = 'scale=1,workers=1', REPLICATION FACTOR 1 ) WITH ( WAIT FOR '1ms' )
+        ALTER CLUSTER zdtaltertest SET ( SIZE = 'scale=1,workers=1', REPLICATION FACTOR 1 ) WITH ( WAIT UNTIL READY (TIMEOUT '1ms', ON TIMEOUT 'COMMIT') )
         """,
         port="internal",
         user="mz_system",
@@ -257,7 +257,7 @@ def wait_for_replica_names(names: list[str], cluster: str = "zdtaltertest"):
     # Fresh names continue past the highest index ever observed.
     mz.environmentd.sql(
         """
-        ALTER CLUSTER zdtaltertest SET ( SIZE = 'scale=1,workers=2', REPLICATION FACTOR 2 ) WITH ( WAIT FOR '1ms' )
+        ALTER CLUSTER zdtaltertest SET ( SIZE = 'scale=1,workers=2', REPLICATION FACTOR 2 ) WITH ( WAIT UNTIL READY (TIMEOUT '1ms', ON TIMEOUT 'COMMIT') )
         """,
         port="internal",
         user="mz_system",
@@ -266,7 +266,7 @@ def wait_for_replica_names(names: list[str], cluster: str = "zdtaltertest"):
 
     mz.environmentd.sql(
         """
-        ALTER CLUSTER zdtaltertest SET ( SIZE = 'scale=1,workers=1', REPLICATION FACTOR 1 ) WITH ( WAIT FOR '1ms' )
+        ALTER CLUSTER zdtaltertest SET ( SIZE = 'scale=1,workers=1', REPLICATION FACTOR 1 ) WITH ( WAIT UNTIL READY (TIMEOUT '1ms', ON TIMEOUT 'COMMIT') )
         """,
         port="internal",
         user="mz_system",
@@ -322,7 +322,7 @@ def wait_for_replica_names(names: list[str], cluster: str = "zdtaltertest"):
     # "-pending" replica.
     mz.environmentd.sql(
         """
-        ALTER CLUSTER zdtaltertest SET (SIZE = 'scale=1,workers=2') WITH ( WAIT FOR '5s')
+        ALTER CLUSTER zdtaltertest SET (SIZE = 'scale=1,workers=2') WITH (WAIT UNTIL READY (TIMEOUT '5s', ON TIMEOUT 'COMMIT'))
         """,
         port="internal",
         user="mz_system",
@@ -382,7 +382,7 @@ def query_with_conn(
     pid = query_with_conn("select pg_backend_pid();", conn)[0][0]
     query_with_conn(
         """
-        ALTER CLUSTER cluster1 SET (SIZE = 'scale=1,workers=2') WITH ( WAIT FOR '5s')
+        ALTER CLUSTER cluster1 SET (SIZE = 'scale=1,workers=2') WITH (WAIT UNTIL READY (TIMEOUT '5s', ON TIMEOUT 'COMMIT'))
         """,
         conn,
         True,

test/cluster/mzcompose.py

@@ -6367,12 +6367,12 @@ def workflow_test_zero_downtime_reconfigure(
         # replica set and background ALTER on, this writes a durable
         # reconfiguration record and returns immediately; the controller brings up
         # a fresh target replica alongside r1, re-hydrates it, then cuts the
-        # realized size over and drops r1. `WAIT FOR` resolves to ON TIMEOUT
-        # COMMIT, so the reconfiguration commits even if its deadline passes during
-        # the restart below.
+        # realized size over and drops r1. Explicit ON TIMEOUT COMMIT ensures the
+        # reconfiguration commits even if its deadline passes during the restart
+        # below.
         c.sql(
             """
-            ALTER CLUSTER cluster1 SET (SIZE = 'scale=1,workers=2') WITH (WAIT FOR '10s')
+            ALTER CLUSTER cluster1 SET (SIZE = 'scale=1,workers=2') WITH (WAIT UNTIL READY (TIMEOUT '10s', ON TIMEOUT 'COMMIT'))
             """,
             port=6877,
             user="mz_system",
@@ -6474,8 +6474,8 @@ def workflow_test_pending_replica_audit_events(
     # the replica set and background ALTER on, this writes a durable
     # reconfiguration record and returns immediately; the controller brings up a
     # fresh target replica, cuts the size over, and drops the old one. `WAIT FOR`
-    # resolves to ON TIMEOUT COMMIT with a long deadline, so the in-flight
-    # reconfiguration commits once the (empty) target hydrates.
+    # uses the safe rollback action with a long deadline, and the in-flight
+    # reconfiguration finalizes once the (empty) target hydrates.
     c.sql(
         """
         ALTER CLUSTER test_audit SET (SIZE = 'scale=1,workers=2') WITH (WAIT FOR '300s')

test/testdrive/cluster-controller.td

@@ -256,8 +256,8 @@ scale=1,workers=2
 > SELECT size FROM mz_clusters WHERE name = 'cc_timeout'
 scale=1,workers=4
 
-# Explicit ON TIMEOUT COMMIT, and WAIT FOR (which desugars to ON TIMEOUT COMMIT):
-# both accepted under the gate and drive a cut-over.
+# Explicit ON TIMEOUT COMMIT and WAIT FOR (which uses ROLLBACK on timeout) both
+# drive a cut-over when the target hydrates before the deadline.
 > ALTER CLUSTER cc_timeout SET (SIZE 'scale=1,workers=2') WITH (WAIT UNTIL READY (TIMEOUT '60s', ON TIMEOUT 'COMMIT'))
 > SELECT size FROM mz_clusters WHERE name = 'cc_timeout'
 scale=1,workers=2
@@ -284,8 +284,9 @@ scale=1,workers=1
 $ postgres-execute connection=postgres://mz_system@${testdrive.materialize-internal-sql-addr}
 ALTER SYSTEM SET unsafe_enable_unstable_dependencies = true
 
-# ROLLBACK: the controller drops the target set at the deadline, leaves the
-# realized config untouched, and audits the record as timed-out.
+# WAIT FOR uses ROLLBACK on timeout: the controller drops the target set at the
+# deadline, leaves the realized config untouched, and audits the record as
+# timed-out.
 > CREATE CLUSTER cc_deadline (SIZE 'scale=1,workers=1', REPLICATION FACTOR 1)
 
 > CREATE TABLE cc_deadline_t (id int)
@@ -295,7 +296,7 @@ ALTER SYSTEM SET unsafe_enable_unstable_dependencies = true
 > CREATE MATERIALIZED VIEW cc_deadline_slow IN CLUSTER cc_deadline AS
   SELECT mz_unsafe.mz_sleep(id * 3600) AS s FROM cc_deadline_t
 
-> ALTER CLUSTER cc_deadline SET (SIZE 'scale=1,workers=4') WITH (WAIT UNTIL READY (TIMEOUT '1s', ON TIMEOUT 'ROLLBACK'))
+> ALTER CLUSTER cc_deadline SET (SIZE 'scale=1,workers=4') WITH (WAIT FOR '1s')
 
 > SELECT details->>'transition' FROM mz_catalog.mz_audit_events WHERE event_type = 'alter' AND object_type = 'cluster' AND details->>'cluster_name' = 'cc_deadline' AND details->>'transition' IS NOT NULL ORDER BY id
 started