fix(rest): node evict must not demote healthy diskful to tiebreaker (Bug 385)

[Andrei Kvapil (kvaps)]

What

linstor n e <node> (node evict) misbehaved when the cluster had a free TieBreaker replica: the evict did not take effect. The TieBreaker stayed pinned to the just-evicted node, and a healthy diskful replica drifted into the tiebreaker role.

Operator-reported repro (3-node dev stand), before evict: worker-1 UpToDate, worker-2 UpToDate, worker-3 TieBreaker. After linstor n e worker-3: worker-2 was wrongly demoted UpToDate → TieBreaker and the evicted worker-3 still held a diskful resource — the eviction "did not happen".

Why

The RD reconciler's tiebreaker pass (ensureTiebreaker) counted every replica of an RD, including a TIE_BREAKER witness sitting on a just-EVICTED node. The witness invariant therefore read as already-satisfied (2 diskful + 1 witness), so the reconciler never relocated the witness off the drained node. The stranded witness then blocked a fresh witness from landing on a healthy spare, which is how the operator observed a healthy diskful drift into the tiebreaker role.

The placer already documents and enforces the correct semantic — replicas on EVICTED/LOST nodes are not counted — but the RD-level witness/quorum decision did not mirror it.

Fix

Treat replicas on EVICTED/LOST nodes as draining placements, not live ones:

• Exclude evicted/lost-node replicas from the witness and quorum decision (mirroring the placer's disabledNodes semantic).
• Reap a TIE_BREAKER stranded on a disabled node so a fresh witness can relocate to a healthy spare, or quorum falls back to off when no spare remains.
• A healthy diskful is never demoted to TieBreaker. Diskful replicas on disabled nodes are left to the NodeReconciler's existing migration path (placer gap-fill for EVICTED, source-delete for LOST).

The change is confined to the RD reconciler; the shared witness-counting helpers (splitByDiskless, shouldTieBreakerExist) are untouched.

Tests (CLI-bug-fix protocol)

• L1 regression — internal/controller/ensure_tiebreaker_evict_bug_385_test.go: evicting the tiebreaker node reaps the stranded witness without demoting a healthy diskful; with a spare node the witness relocates to it; evicting a diskful node does not let its stranded replica keep the witness alive.
• L6 cli-matrix — tests/e2e/cli-matrix/n-evict-tiebreaker-no-shuffle.sh: real linstor n e on the 2r+tb shape, asserts the witness leaves the evicted node and neither healthy diskful is demoted.
• L7 replay — tests/operator-harness/replay/n-evict-tiebreaker-no-shuffle.yaml: codifies the operator sequence and convergence assertions (witness reaped off the evicted node, both diskful stay UpToDate).

go build ./..., go test ./..., and golangci-lint run are green locally. Live-stand replay validation is left to the launcher.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 14a0d5e0-cbc9-4edf-9c24-d9e9e94e3394

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/bug-385-evict-tb-shuffle

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[gemini-code-assist]

Code Review

This pull request addresses Bug 385, where evicting a node hosting a TieBreaker replica misbehaves because the reconciler counts replicas on evicted or lost nodes as live. The changes filter out replicas on disabled nodes from the live count and explicitly remove stranded witnesses. Additionally, unit and E2E tests are introduced to verify this behavior. A critical review comment points out a safety issue where evicting a stranded witness when no healthy spare nodes are available can result in a 2-replica resource definition running under majority quorum without a witness, presenting a data-availability hazard. A code suggestion is provided to fallback quorum to off if no candidate tiebreaker node is available.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

There is a critical correctness and safety issue when a stranded witness is evicted and there are no healthy spare nodes available to host a new witness.

The Issue

1. removeStrandedWitnesses deletes the stranded witness from the store.
2. live replicas will have len(witness) == 0.
3. wantWitness is computed as true (since we have 2 live diskful replicas and no live witness).
4. applyWitnessDecision is called and enters the wantWitness && len(witness) == 0 case, calling createWitness.
5. Since there are no healthy spare nodes, createWitness finds no candidate node and returns nil (no error).
6. However, applyWitnessDecision blindly appends a dummy witness to diskless and returns it anyway.
7. quorumPolicy receives disklessAfter with 1 element and returns QuorumPolicyMajority.
8. setQuorum sets quorum: majority on the RD.

This results in a 2-replica RD with no witness running under quorum: majority, which is a severe data-availability hazard (the volume will freeze if either of the two remaining diskful nodes fails).

The Fix

We should check if a candidate tiebreaker node is actually available before deciding we want a witness. If no node is available, we set wantWitness to false so that quorum correctly falls back to off.

Suggested change

	wantWitness := shouldTieBreakerExist(rd, diskful, diskless, witness)
	wantWitness := shouldTieBreakerExist(rd, diskful, diskless, witness)
	if wantWitness && len(witness) == 0 {
	hostingReplica := make(map[string]bool)
	for i := range replicas {
	hostingReplica[replicas[i].NodeName] = true
	}
	node, err := r.pickTiebreakerNodeForRD(ctx, rd.Name, hostingReplica)
	if err != nil {
	return err
	}
	if node == "" {
	wantWitness = false
	}
	}

[Andrei Kvapil (kvaps)]

Superseded by #83 (merged): the four operational-lifecycle fixes (Bugs 384–387) plus the L7 harness hardening (Bug 388) were validated as a unit on the live stand and merged together in #83. The commits from this branch are included there.