Skip to content

fix(database): create scale members through the staged patch; stage deletions; never delete the acting primary - #1160

Merged
paulocsanz merged 1 commit into
masterfrom
pcs/ha-scale-contract-fixes
Aug 31, 2026
Merged

fix(database): create scale members through the staged patch; stage deletions; never delete the acting primary#1160
paulocsanz merged 1 commit into
masterfrom
pcs/ha-scale-contract-fixes

Conversation

@paulocsanz

@paulocsanz paulocsanz commented Aug 31, 2026

Copy link
Copy Markdown
Collaborator

Rebased onto v5.47.0. #1159 already landed the joining-nodes-only restamp scope, the staged volume instance, the scoped revert sweep and the tag-parse parity this branch originally carried — what remains here is the rest of the scale contract:

1. New member service records are created through the staged patch

serviceCreate bound to the environment pre-creates the ServiceInstance, so at commit the patch-apply workflow takes its update path — which applies clusterRole but silently drops parentServiceId (only the instance-create path persists the parent link). Every CLI-scaled member landed role-stamped and orphaned from parent-chain membership walks: ha status, scale rosters, the platform's cluster-primacy guard, the admin monitors.

New member records are now created bare (environmentId: null, no source) with their instances created by the staged patch (isCreated) — the split the dashboard's hook already uses. The volume record goes fully detached too (no serviceId; the mount is declared by the patch). With the cause gone, the post-commit "platform dropped the parent link" warning goes with it. The image, the cluster's canvas group, and the sibling's whole deploy config (healthcheck, region placement — a node cloned into a different region than its cluster replicates cross-region forever) ride the patch.

2. Scale-down deletions are staged

Direct ServiceDelete/VolumeDelete calls bypassed the platform's cluster-primacy commit guard, which inspects the patch for replica deletions. Deletions are now staged tombstones (volume first, then the service), so the whole scale commits atomically and the guard can see it.

3. Scale-down never deletes the acting primary

Deletion order is by node number, and after a failover the primary can be any replica. ha scale now probes the live role first, removes a lower-numbered replica instead where the count allows it, and refuses with the switchover remedy where it does not; when no member answers it degrades to a warning (same posture as revert's primacy precheck).

Also: the internal (coordinator) identity variable moves to joining-nodes-only, matching the replica identity scope #1159 set — coordinators restarting together is quorum loss; and the routing-edge lookup excludes the pooler through the engine's declared pooling spec instead of an image name compiled into the filter.

Verification

  • cargo test — 1312 pass. The stub-backboard scale-up test now also pins the bare service record (environmentId/source null) and the staged isCreated/image; new stub tests pin the staged scale-down tombstones, the primary being skipped in favor of a lower-numbered replica, and the refusal with the switchover remedy when the count cannot be honored.
  • cargo clippy / cargo fmt clean (no new warnings in any touched file).
  • ServiceCreateInput.environmentId and VolumeCreateInput.serviceId are nullable server-side (the dashboard already calls both that way); the mutation documents now declare them nullable and the existing callers pass Some(...) unchanged in behavior.
  • mono#36918 fixes the update-path drop server-side and re-parents already-orphaned members; this PR makes the CLI correct against today's backboard either way.

…eletions; never delete the acting primary

Rebased onto v5.47.0: #1159 already landed the joining-nodes-only restamp
scope, the staged volume instance, the scoped revert sweep and the tag-parse
parity this branch originally carried. What remains here is the rest of the
scale contract:

1. New member SERVICE records are now created bare (environmentId: null,
   no source) and their instances created by the staged patch (isCreated),
   the way the dashboard's hook does -- because the patch-apply workflow's
   instance-create path is the ONLY one that persists parentServiceId. An
   instance pre-created by serviceCreate takes the update path at commit,
   which applies clusterRole but silently drops the parent link, so every
   CLI-scaled member landed role-stamped and orphaned from parent-chain
   membership walks (ha status, scale rosters, the platform's cluster-primacy
   guard, the admin monitors). The volume record goes fully detached too (no
   serviceId; the mount is declared by the patch). With the cause gone, the
   post-commit "platform dropped the parent link" warning goes with it.
   The image, the cluster's canvas group, and the sibling's whole deploy
   config (healthcheck, region placement -- a node cloned into a different
   region than its cluster replicates cross-region forever) ride the patch.

2. Scale-down deletions are STAGED (volume first, then the service) instead
   of issued as direct ServiceDelete/VolumeDelete calls, so the whole scale
   commits atomically and the platform's cluster-primacy commit guard can
   inspect the patch it exists to inspect.

3. Replica scale-down never deletes the node currently ACTING as primary.
   Deletion order is by node number, and after a failover the primary can be
   any replica: ha scale now probes the live role first, removes a
   lower-numbered replica instead where the count allows it, and refuses
   with the switchover remedy where it does not; when no member answers it
   degrades to a warning (same posture as revert's primacy precheck).

Also: the internal (coordinator) identity variable moves to joining-nodes-
only, matching the replica identity scope #1159 set (coordinators restarting
together is quorum loss); the routing-edge lookup excludes the pooler
through the engine's declared pooling spec instead of an image name compiled
into the filter.

cargo test: 1312 pass. The stub-backboard scale-up test now also pins the
bare service record (environmentId/source null) and the staged
isCreated/image; new stub tests pin the staged scale-down tombstones, the
primary being skipped in favor of a lower-numbered replica, and the refusal
with the switchover remedy when the count cannot be honored.
@paulocsanz
paulocsanz force-pushed the pcs/ha-scale-contract-fixes branch from 196183c to 2a6ad63 Compare August 31, 2026 22:22
@paulocsanz paulocsanz changed the title fix(database): scale through the staged patch, restamp joining nodes only, never delete the acting primary fix(database): create scale members through the staged patch; stage deletions; never delete the acting primary Aug 31, 2026
@paulocsanz
paulocsanz merged commit 9ef6969 into master Aug 31, 2026
9 checks passed
@paulocsanz
paulocsanz deleted the pcs/ha-scale-contract-fixes branch August 31, 2026 22:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release/patch Author patch release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant