fix(database): create scale members through the staged patch; stage deletions; never delete the acting primary - #1160
Merged
Conversation
This was referenced Aug 31, 2026
…eletions; never delete the acting primary Rebased onto v5.47.0: #1159 already landed the joining-nodes-only restamp scope, the staged volume instance, the scoped revert sweep and the tag-parse parity this branch originally carried. What remains here is the rest of the scale contract: 1. New member SERVICE records are now created bare (environmentId: null, no source) and their instances created by the staged patch (isCreated), the way the dashboard's hook does -- because the patch-apply workflow's instance-create path is the ONLY one that persists parentServiceId. An instance pre-created by serviceCreate takes the update path at commit, which applies clusterRole but silently drops the parent link, so every CLI-scaled member landed role-stamped and orphaned from parent-chain membership walks (ha status, scale rosters, the platform's cluster-primacy guard, the admin monitors). The volume record goes fully detached too (no serviceId; the mount is declared by the patch). With the cause gone, the post-commit "platform dropped the parent link" warning goes with it. The image, the cluster's canvas group, and the sibling's whole deploy config (healthcheck, region placement -- a node cloned into a different region than its cluster replicates cross-region forever) ride the patch. 2. Scale-down deletions are STAGED (volume first, then the service) instead of issued as direct ServiceDelete/VolumeDelete calls, so the whole scale commits atomically and the platform's cluster-primacy commit guard can inspect the patch it exists to inspect. 3. Replica scale-down never deletes the node currently ACTING as primary. Deletion order is by node number, and after a failover the primary can be any replica: ha scale now probes the live role first, removes a lower-numbered replica instead where the count allows it, and refuses with the switchover remedy where it does not; when no member answers it degrades to a warning (same posture as revert's primacy precheck). Also: the internal (coordinator) identity variable moves to joining-nodes- only, matching the replica identity scope #1159 set (coordinators restarting together is quorum loss); the routing-edge lookup excludes the pooler through the engine's declared pooling spec instead of an image name compiled into the filter. cargo test: 1312 pass. The stub-backboard scale-up test now also pins the bare service record (environmentId/source null) and the staged isCreated/image; new stub tests pin the staged scale-down tombstones, the primary being skipped in favor of a lower-numbered replica, and the refusal with the switchover remedy when the count cannot be honored.
paulocsanz
force-pushed
the
pcs/ha-scale-contract-fixes
branch
from
August 31, 2026 22:22
196183c to
2a6ad63
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebased onto v5.47.0. #1159 already landed the joining-nodes-only restamp scope, the staged volume instance, the scoped revert sweep and the tag-parse parity this branch originally carried — what remains here is the rest of the scale contract:
1. New member service records are created through the staged patch
serviceCreatebound to the environment pre-creates the ServiceInstance, so at commit the patch-apply workflow takes its update path — which appliesclusterRolebut silently dropsparentServiceId(only the instance-create path persists the parent link). Every CLI-scaled member landed role-stamped and orphaned from parent-chain membership walks:ha status, scale rosters, the platform's cluster-primacy guard, the admin monitors.New member records are now created bare (
environmentId: null, no source) with their instances created by the staged patch (isCreated) — the split the dashboard's hook already uses. The volume record goes fully detached too (noserviceId; the mount is declared by the patch). With the cause gone, the post-commit "platform dropped the parent link" warning goes with it. The image, the cluster's canvas group, and the sibling's whole deploy config (healthcheck, region placement — a node cloned into a different region than its cluster replicates cross-region forever) ride the patch.2. Scale-down deletions are staged
Direct
ServiceDelete/VolumeDeletecalls bypassed the platform's cluster-primacy commit guard, which inspects the patch for replica deletions. Deletions are now staged tombstones (volume first, then the service), so the whole scale commits atomically and the guard can see it.3. Scale-down never deletes the acting primary
Deletion order is by node number, and after a failover the primary can be any replica.
ha scalenow probes the live role first, removes a lower-numbered replica instead where the count allows it, and refuses with the switchover remedy where it does not; when no member answers it degrades to a warning (same posture asrevert's primacy precheck).Also: the internal (coordinator) identity variable moves to joining-nodes-only, matching the replica identity scope #1159 set — coordinators restarting together is quorum loss; and the routing-edge lookup excludes the pooler through the engine's declared pooling spec instead of an image name compiled into the filter.
Verification
cargo test— 1312 pass. The stub-backboard scale-up test now also pins the bare service record (environmentId/sourcenull) and the stagedisCreated/image; new stub tests pin the staged scale-down tombstones, the primary being skipped in favor of a lower-numbered replica, and the refusal with the switchover remedy when the count cannot be honored.cargo clippy/cargo fmtclean (no new warnings in any touched file).ServiceCreateInput.environmentIdandVolumeCreateInput.serviceIdare nullable server-side (the dashboard already calls both that way); the mutation documents now declare them nullable and the existing callers passSome(...)unchanged in behavior.