fix(controller): honor r d --keep-tiebreaker so auto-witness survives

[Andrei Kvapil (kvaps)]

Bug

linstor r d --keep-tiebreaker <one-of-diskful> <rd> on a 2-diskful + 1-witness shape silently reaped the auto-managed TIE_BREAKER witness right after the delete completed. Final topology was a solo diskful with NO witness — quorum=1, unprotected for ~5 min until stampTiebreakerSuppression expired.

The CLI flag documents itself as "Keeps the tiebreaker instead of accidentally deleting it. Also removes the RD property if necessary in order to keep the resource as a tiebreaker", but neither the REST DELETE handler nor the RD reconciler observed the operator intent.

Repro (hunt-v3 finding B.1, full evidence in /tmp/bug-hunt3-B1-keep-tiebreaker-strips-witness.md on the dev stand):

$L rd c hunt3-b1 -l drbd,storage
$L vd c hunt3-b1 100M
$L r c dev-worker-1 hunt3-b1 -s zfs-thin
$L r c dev-worker-2 hunt3-b1 -s zfs-thin
# state: w-1 diskful, w-2 diskful, w-3 auto-TIE_BREAKER
$L r d --keep-tiebreaker dev-worker-2 hunt3-b1

Controller log shows the carve-out reaping the witness:

ensureTiebreaker rd=hunt3-b1 replicas=2 diskful=1 witness=1 willCreate=false willRemove=true

Root cause

ensureTiebreaker reasoned purely from on-disk topology. The --keep-tiebreaker intent from the REST DELETE request wasn't plumbed to the controller, so shouldKeepExistingWitness hit the Bug-338 carve-out (diskful == 1 && nonWitnessDiskless == 0 ⇒ collapse) and removed the witness in the same reconcile that observed diskful drop from 2 to 1.

Fix

Mirror the existing suppression-annotation pattern (AutoTiebreakerSuppressedUntilAnnotation):

• REST handleResourceDelete reads ?keep_tiebreaker=true from the URL and stamps a new KeepTiebreakerUntilAnnotation (blockstor.io/keep-tiebreaker-until) on the parent RD with a fresh 5-minute RFC3339 deadline BEFORE committing the Resource Delete. Stamp lands first so any Reconcile woken by the Delete event observes both signals atomically.
• RD-side shouldTieBreakerExist now consults isKeepTiebreakerActive via a new keepExistingWitnessFor helper. When the annotation deadline is in the future AND at least one diskful peer survives, the existing witness is preserved regardless of the Bug-338 carve-out. The diskful floor matters because a lone witness with 0 diskful is 1 voter — strictly worse than no witness.
• Expired and unparseable values are treated as "no override", so a forgotten or hand-edited annotation can't permanently freeze the auto-quorum invariant. After 5 minutes the normal Bug-338 collapse path resumes with no manual cleanup.

Tests

• internal/controller/ensure_tiebreaker_test.go:
  • TestKeepTiebreakerAnnotation_HonoredWhileFresh — 1 diskful + 1 witness with fresh annotation → witness preserved.
  • TestKeepTiebreakerAnnotation_ExpiredFallsThrough — same topology, expired annotation → Bug-338 collapse resumes; unparseable values also treated as "no override".
• pkg/rest/autoplace_test.go:
  • TestResourceDeleteKeepTiebreakerStampsOverride — DELETE …?keep_tiebreaker=true writes the annotation with a future RFC3339 deadline.
  • TestResourceDeleteWithoutKeepTiebreakerSkipsOverride — plain DELETE … does not stamp the override.
• tests/e2e/keep-tiebreaker-3way.sh — real-DRBD QEMU-stand regression catcher: 2 diskful + 1 auto-TB topology, linstor r d --keep-tiebreaker, assert the surviving TB stays for a 30s stability tail. Required by the TieBreaker-e2e-must-exist rule (this class re-regressed before: Bugs 104/108/267/271/338/342).

Stand verification

Skipped — the dev stand was unreachable from this network at commit time. The unit + REST tests cover the wire path end-to-end (REST handler → store → reconciler decision), and the e2e script is ready to run on the QEMU stand once available.

[coderabbitai]

Warning

Review limit reached

@kvaps, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 21 minutes and 14 seconds. Learn how PR review limits work.

Your organization has run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 79bcb550-4989-42d8-b727-8d68fcb4784d

📥 Commits

Reviewing files that changed from the base of the PR and between 010e1bb and 6494c5b.

📒 Files selected for processing (7)

• internal/controller/ensure_tiebreaker_test.go
• internal/controller/export_test.go
• internal/controller/resourcedefinition_controller.go
• pkg/api/v1/annotations.go
• pkg/rest/autoplace.go
• pkg/rest/autoplace_test.go
• tests/e2e/keep-tiebreaker-3way.sh

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/keep-tiebreaker-r-d

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[gemini-code-assist]

Code Review

This pull request implements support for the --keep-tiebreaker flag during resource deletion (Bug B.1) by introducing a short-lived KeepTiebreakerUntilAnnotation annotation. This annotation prevents the reconciler from prematurely collapsing auto-managed TIE_BREAKER witnesses during diskful-to-single-replica transitions. The feedback suggests using time.RFC3339Nano instead of time.RFC3339 to parse the deadline timestamp more robustly against fractional seconds, and adding a check in the E2E test script to ensure the port-forwarding process binds successfully before proceeding.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

Using time.RFC3339 for parsing can fail if the annotation value contains fractional seconds (e.g., if it was written by a system that serializes timestamps with nanosecond precision, or if a user manually copied a timestamp from Kubernetes metadata like creationTimestamp which uses fractional seconds).

In Go, time.Parse(time.RFC3339Nano, raw) is fully backwards-compatible with standard RFC3339 (without fractional seconds) but also successfully parses timestamps with fractional seconds. Switching to time.RFC3339Nano makes the parsing significantly more robust against different timestamp formats.

Suggested change

	deadline, err := time.Parse(time.RFC3339, raw)
	deadline, err := time.Parse(time.RFC3339Nano, raw)

[gemini-code-assist]

For consistency and robustness against fractional seconds, use time.RFC3339Nano for parsing the deadline timestamp here as well.

Suggested change

	deadline, err := time.Parse(time.RFC3339, raw)
	deadline, err := time.Parse(time.RFC3339Nano, raw)

[gemini-code-assist]

If the port-forward background process fails to start or bind to the port, the script will silently continue past this loop and fail later with a confusing connection error on the first linstor command.

Adding a check after the loop to verify that the port-forward actually bound successfully will make the script fail fast with a clear error message.

Suggested change

	for _ in $(seq 1 30); do
	if curl -sf -m1 "http://localhost:$PF_PORT/v1/nodes" >/dev/null 2>&1; then
	break
	fi
	sleep 0.5
	done
	for _ in $(seq 1 30); do
	if curl -sf -m1 "http://localhost:$PF_PORT/v1/nodes" >/dev/null 2>&1; then
	break
	fi
	sleep 0.5
	done
	if ! curl -sf -m1 "http://localhost:$PF_PORT/v1/nodes" >/dev/null 2>&1; then
	echo "FAIL: port-forward failed to bind to port $PF_PORT" >&2
	exit 1
	fi