Document journal-copy replacement and correct failure-domain sizing rules - #203
Merged
Merged
Conversation
noctarius
force-pushed
the
docs/failure-domain-journal-replacement
branch
from
August 31, 2026 20:16
1dbcfc7 to
7f2a6df
Compare
Collaborator
|
@wmousa I've added another commit with style alignment. Can you please verify it 7f2a6df1b337c62fe338dacb91ef7150ee880334? |
Contributor
|
🤖: Deployment available as https://docstest.simplyblock.io/docs-failure-domain-journal-replacement |
Collaborator
|
@wmousa ? |
Contributor
|
🤖: Deployment available as https://docstest.simplyblock.io/docs-failure-domain-journal-replacement |
Contributor
Author
|
Verified Checked it as a factual review rather than just prose, since it touches the sizing rules:
I also happened to confirm this on hardware today: a cluster whose previous status was Rebasing onto main now (the branch had fallen 16 commits behind); no content changes with it. |
… removal Failure-domain node removal already documented that failover-path relocation can require a cross-domain target and refuses the removal if none exists. It never documented the journal (JM) redundancy set's own replacement behavior, which runs at the same time but in the opposite direction: the control plane prefers a replacement from the departed node's own failure domain, to leave the set's domain distribution unchanged rather than reshuffled -- a best-effort preference, never a removal blocker. Adds a new "Journal Copy Replacement on Removal" section to the architecture concept page, and cross-references it from both the Kubernetes and non-Kubernetes "failure domain" operations pages, next to their existing failover-path relocation coverage. Verified with scripts/quality-gate.sh (spelling, terminology, american, prose, voice, punctuation, syntax) -- clean for all three files.
…zing table Two claims in the failure-domain docs no longer matched the code: - "At least parity chunks + 1 domains" for surviving a domain outage was the bare rotation-correctness floor, not the actual activation gate. Fresh activation now hard-requires parity chunks + 2 distinct domains (2 domains is never sufficient, at any parity-chunk count) so a later single add/remove always has a spare host to relocate onto, rather than stranding a failover path immediately. - "If no such node exists, the removal is refused" skipped a fallback: the control plane now splices a stranded failover path into an already-formed pairing (rebuilding that host's own path elsewhere) before actually refusing the removal. Replaces the old goal/recommendation table with a domain-count x erasure-coding-scheme sizing table showing exactly how much simultaneous node or domain loss each combination tolerates, derived from the same per-domain risk-budget formula the node-shutdown capacity check uses (chunks_per_domain = ceil((ndcs+npcs) / domains), capped contribution per domain, summed against a parity-chunks budget). Verified with scripts/quality-gate.sh (spelling, terminology, american, prose, voice, punctuation, syntax) -- clean for both files; the syntax gate's remaining failures are pre-existing broken CLI-reference links in unrelated, untouched files.
The failure-domain pages covered the balance rules a removal has to satisfy but never said what removal is for, and the omission hides a trap that only shows up on the second failure. Domain balance counts every host that has not been removed, whatever its state (failure_domain_host_map skips only STATUS_REMOVED and dedicated secondaries), so a dead host left in the cluster still occupies its domain's slot while serving nothing. The first such host can still be removed -- its domain drops one below the others, which the +/-1 rule permits. If a second host in the same domain fails first, only one of the two can go: removing the second would leave that domain two hosts below the others and, on a three-host domain, below the two-host floor, and admission refuses it. The degradation then cannot be cleared by removal at all until hosts are added. Adds a "When to Remove a Node" section to the architecture concept page stating the two intended uses (retiring a failed host, shrinking a cluster), that an offline or unreachable node can still be removed so removal is the intended path for a dead host rather than a last resort, the two-failure trap above, the two ways to restore balance afterwards, and the admission conditions an operator can act on -- no volumes or snapshots on the node, the +/-1 and two-host rules, a valid host-disjoint target for every failover path the node hosts, and node headroom above data+parity counting hosts already not online. Cross-references it from the Kubernetes and non-Kubernetes "failure domain" operations pages, next to their existing removal coverage. Verified with scripts/quality-gate.sh: spelling, terminology, american, prose, voice and punctuation clean. The syntax gate's failures are the same pre-existing broken CLI-reference links in unrelated, untouched files; none of the three files changed here appears among them.
wmousa
force-pushed
the
docs/failure-domain-journal-replacement
branch
from
September 4, 2026 09:33
476961a to
d69870f
Compare
Contributor
|
🤖: Deployment available as https://docstest.simplyblock.io/docs-failure-domain-journal-replacement |
noctarius
enabled auto-merge
September 4, 2026 11:10
noctarius
disabled auto-merge
September 4, 2026 11:10
Contributor
|
🤖: Deployment available as https://docstest.simplyblock.io/docs-failure-domain-journal-replacement |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Documents the journal-copy replacement behavior on failure-domain node removal,
and corrects two claims that no longer matched the code:
bare rotation-correctness floor, not the actual activation gate. Fresh
activation now hard-requires parity chunks + 2 distinct domains (2 domains
is never sufficient, at any parity-chunk count) so a later single add/remove
always has a spare host to relocate onto, rather than stranding a failover
path immediately.
control plane now splices a stranded failover path into an already-formed
pairing (rebuilding that host's own path elsewhere) before actually
refusing the removal.
Adds a "Journal Copy Replacement on Removal" section to the architecture
concept page (cross-referenced from both the Kubernetes and non-Kubernetes
operations pages), and replaces the old goal/recommendation table with a
domain-count x erasure-coding-scheme sizing table showing exactly how much
simultaneous node or domain loss each combination tolerates -- derived
directly from the same per-domain risk-budget formula the node-shutdown
capacity check uses (chunks_per_domain = ceil((ndcs+npcs) / domains), capped
contribution per domain, summed against a parity-chunks budget), not guessed.
Verified with scripts/quality-gate.sh (spelling, terminology, american,
prose, voice, punctuation, syntax) -- clean for all three touched files; the
syntax gate's remaining failures are pre-existing broken CLI-reference links
in unrelated, untouched files.