Skip to content

Add Stretch Cluster diagrams, 2/2/2 layout, and failure guidance#1820

Open
JakeSCahill wants to merge 9 commits into
betafrom
doc-1775-stretch-follow-ups
Open

Add Stretch Cluster diagrams, 2/2/2 layout, and failure guidance#1820
JakeSCahill wants to merge 9 commits into
betafrom
doc-1775-stretch-follow-ups

Conversation

@JakeSCahill

Copy link
Copy Markdown
Contributor

Follow-up to #1681, which merged while this work was in flight.

Chooser page (k-choose-deployment.adoc)

  • Topology comparison diagram (multi-AZ, Stretch Cluster, Shadowing) above the comparison table
  • Stretch Cluster vs Shadowing head-to-head table: replication, RPO, RTO, client failover, topology, produce latency, best-for
  • Tiered Storage and relative-cost rows in the topology table, concrete produce-latency figures, and a Whole Cluster Restore pointer

Stretch page (k-stretch-clusters.adoc)

  • Architecture diagram separating the operator control plane (Raft over TLS, port 9443) from the broker data plane (Raft, port 33145), with split-brain protection and the two-level leader election explained (per eng discussion: fixed peer list, local Lease election per cluster, only the local leader joins the cross-cluster Raft group)
  • 2/2/2 six-broker recommendation replacing 2/2/1: survives any one location loss and restores full RF=5 replication after a single broker failure. Validated end-to-end on 3× kind v1.34.0 (StretchCluster Ready, all six brokers in rpk cluster info, RF=5 topic)
  • New Limitations section: no Tiered Storage (which also rules out Iceberg topics and Cloud Topics); same-continent placement with example region trios
  • New Understand failure scenarios section: one location down (automatic re-election, ghost ejection, fresh rejoin), two or more down (availability outage, not data loss; contact Support), beyond quorum (pair with Shadowing as defense in depth)
  • Spec-drift behavior documented from operator source (SpecSynced condition, reason DriftDetected, reconciliation blocks but pod remediation continues) with a troubleshooting entry
  • Fixed peer list and the procedure to add/remove a participating cluster; managed EKS/GKE/AKS support; plugin auto-download on first rpk k8s use; kubeconfig caching for instant leader takeover; ordered_racks leader preference paired with follower fetching; automatic stale-disk wipe recovery (--wipe-stale-disk-after)

Notes for reviewers

🤖 Generated with Claude Code

Chooser page:
- Add a topology comparison diagram (multi-AZ, Stretch Cluster,
  Shadowing) above the comparison table
- Add a Stretch Cluster vs Shadowing comparison table (replication,
  RPO, RTO, client failover, topology, produce latency, best for)
- Add Tiered Storage and relative cost rows to the topology table,
  concrete produce-latency figures, and a Whole Cluster Restore pointer

Stretch page:
- Add an architecture diagram separating the operator control plane
  from the broker data plane, with split-brain protection and
  two-level leader election explained
- Recommend a 2/2/2 six-broker layout (survives location loss and
  restores full RF=5 replication after a single broker failure);
  validated end-to-end on kind
- Add a Limitations section: no Tiered Storage (and no Iceberg topics
  or Cloud Topics), same-continent placement with example region trios
- Add an Understand failure scenarios section (one location down, two
  or more down, beyond quorum with Shadowing as defense in depth)
- Document spec-drift behavior (SpecSynced condition, DriftDetected)
  with a troubleshooting entry, the fixed peer list and how to change
  it, managed EKS/GKE/AKS support, plugin auto-download, kubeconfig
  caching, ordered_racks leader preference with follower fetching, and
  the operator's automatic stale-disk wipe recovery
@JakeSCahill
JakeSCahill requested a review from a team as a code owner July 22, 2026 15:55
@netlify

netlify Bot commented Jul 22, 2026

Copy link
Copy Markdown

Deploy Preview for redpanda-docs-preview ready!

Name Link
🔨 Latest commit 510e702
🔍 Latest deploy log https://app.netlify.com/projects/redpanda-docs-preview/deploys/6a639e7aed558a000873d8ff
😎 Deploy Preview https://deploy-preview-1820--redpanda-docs-preview.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@coderabbitai

coderabbitai Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 530373fb-5a53-4a36-b900-c0246316bc4a

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch doc-1775-stretch-follow-ups

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Per engineering (K8S-907 discussion): in this release the
RedpandaBrokerPool CRD is installed only in multicluster mode and its
controller reconciles pools only for StretchCluster resources; NodePool
(beta) remains the pool mechanism for standalone clusters.

- Replace the release-notes section that claimed NodePool documentation
  was removed (the NodePool docs are still published) with an accurate
  'Broker pool resources by deployment type' section
- Add a Limitations bullet on the stretch page pointing standalone
  deployments at NodePool

Verified against operator source at operator/v26.2.1-beta.3:
multiclusterCRDs gating in cmd/crd/crd.go and the StretchCluster-only
clusterRef resolution in brokerpool_controller.go
- Fix the Cloud Topics xref, which moved under cloud-topics/ on beta
- Spell out RPO on first use on the stretch page
- Rework the Limitations bullets to the page's bold-term-colon pattern
- Tie the 50 ms recommendation to the 100 ms liveness ceiling in one
  sentence and drop a doubled example marker
- Replace 'as well as' with 'and'; use active voice for the peer-list
  sentence
- Chooser: parallel phrasing in the Relative cost row, clearer Whole
  Cluster Restore trade-off sentence, table spacing consistency
Set flowchart.subGraphTitleMargin (bottom: 16) in the Mermaid
initialize options so subgraph labels no longer sit directly on the
nested boxes below them. The top margin must stay 0 because Mermaid 10
shifts the label up by the top margin without growing the subgraph
box, which clips the title.
Mermaid 10 and 11 never reserve vertical space for a cluster label
above nested subgraphs: the label overlaps the inner box borders, and
neither flowchart.subGraphTitleMargin nor flowchart.padding changes
the label-to-content distance (measured at -3px for any value). The
only working lever is a trailing blank line in the title itself.
@JakeSCahill
JakeSCahill requested a review from david-yu July 23, 2026 14:53
@david-yu

david-yu commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

We should be suggestion 2/2/1 not 2/2/2 so that if one region goes down we always have quorum due to an odd number of replicas also we need to recommend RF=5 so all replicas have copies of data in case customers want to preserve data even if the quorum is lost. Also the digram looks a bit hard to read is possible to arrange the clusters in a triangle pattern to show cross cluster communication instead of having them stacked on each other?

…side

Per review: 2/2/1 with default_topic_replications: 5 keeps an odd
broker count, so the cluster always holds a clean three-of-five
majority through the loss of any one location, and with five replicas
on five brokers every broker holds a copy of every partition, so any
surviving location retains a complete copy of the data even if quorum
is lost.

Also lay the architecture diagram out left-to-right so the three
Kubernetes clusters appear side by side with the full cross-cluster
mesh visible, instead of stacked vertically.
@JakeSCahill

Copy link
Copy Markdown
Contributor Author

@david-yu Done in 510e702:

  • 2/2/1 restored everywhere (recommendation text, YAML comments, both diagrams, failure-scenarios section). The recommendation paragraph now carries both of your rationales: the odd broker count keeps a clean three-of-five majority through any single location loss, and with default_topic_replications: 5 on five brokers, every broker holds a replica of every partition, so any surviving location retains a complete copy of the data even if quorum is lost.
  • Diagram rearranged: the three Kubernetes clusters now sit side by side with the full cross-cluster mesh drawn between them (dotted operator Raft ring, solid broker replication), instead of stacked vertically. A strict triangle isn't possible with Mermaid's layout engine (the third side of the triangle always forces the layout into a chain), but the side-by-side mesh reads the same way.

Preview rebuilds from the latest push.

@david-yu

Copy link
Copy Markdown
Contributor

Looks good but I'm not sure if its just me there are extraneous plus signs in the doc.

  • Apply the manifest to every cluster:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants