design-proposal: cross-cluster mesh for tenant access to host services

[kvaps]

Summary

Adds a design proposal for cross-cluster connectivity between Cozystack-managed tenant clusters and the host cluster.

The motivating use case: a host cluster running Ceph (managed by Rook) that should be reachable from inside tenant clusters as if it were local storage. Standard single-gateway approaches (Submariner, Kilo's default mesh-granularity=location) bottleneck Ceph traffic; this proposal uses Kilo's mesh-granularity=cross (squat/kilo#328) to build a node-to-node mesh that scales linearly with cluster size and handles Rook-driven failover without controller intervention on the data path.

The proposal covers:

• Topology and why cross-mesh fits Ceph's traffic patterns
• A new operator (cozystack-meshlink-operator) and TenantMeshLink CRD for managing Peer objects on both sides
• Trust model (one-way: Cozystack → tenant; tenants have no host-cluster API access)
• IP allocation: no new address space; existing pod-CIDRs are sufficient
• Failure semantics, edge cases, and alternatives considered

Looking for feedback on the open questions, especially the upstream Kilo PR #328 strategy and whether tenant-side Kilo should be a hard requirement.

Test plan

This is a design proposal; no code yet. Implementation testing is scoped in the proposal and will follow in implementation PRs:

• Unit tests for reconciliation logic
• Admission webhook tests for pod-CIDR overlap detection
• Integration tests with kind (two clusters)
• E2E with real Rook + tenant cluster + Ceph CSI

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 2e32b34d-3be4-4442-9731-a42feb48c5cd

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

Tip

💬 Introducing Slack Agent: The best way for teams to turn conversations into code.

Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.

• Generate code and open pull requests
• Plan features and break down work
• Investigate incidents and troubleshoot customer tickets together
• Automate recurring tasks and respond to alerts with triggers
• Summarize progress and report instantly

Built for teams:

• Shared memory across your entire org—no repeating context
• Per-thread sandboxes to safely plan and execute work
• Governance built-in—scoped access, auditability, and budget controls

One agent for your entire SDLC. Right inside Slack.

👉 Get started

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[gemini-code-assist]

Code Review

This pull request introduces a design proposal for a cross-cluster mesh using Kilo to allow tenant clusters to access host-cluster services like Ceph. The design utilizes a bipartite node-to-node topology managed by a new operator. The review feedback provides several technical improvements, including addressing MTU overhead for WireGuard, analyzing scalability limits of the N x M mesh, implementing fallback logic for node endpoints, using finalizers for robust resource cleanup, and expanding IP disjointness checks to include Service CIDRs.

[gemini-code-assist]

The proposal should address MTU configuration for the cross-cluster mesh. Since WireGuard adds encapsulation overhead (typically 60-80 bytes), packets from pods using the default 1500 MTU will exceed the tunnel MTU, leading to fragmentation or packet loss. The design should specify how this will be handled, for example, by configuring the Kilo interface MTU and ensuring MSS clamping is active or by adjusting the CNI MTU in the tenant clusters.

[gemini-code-assist]

The N x M bipartite mesh topology may face scalability challenges as the number of nodes increases. For instance, a 100-node host cluster and a 100-node tenant cluster would result in 10,000 WireGuard peers per node. The proposal should include an analysis of the practical limits for the number of peers the kg-agent and the Linux kernel can manage before performance or control-plane stability is impacted.

[gemini-code-assist]

The operator should have a fallback strategy if the kilo.squat.ai/force-endpoint annotation is missing on a host node. Without a defined endpoint, tenant nodes will not be able to initiate the WireGuard handshake. Consider falling back to the Node's ExternalIP or InternalIP, or surfacing a specific error in the TenantMeshLink status.

Suggested change

	2. Lists host cluster Nodes; for each node, ensures a `Peer` exists in the tenant cluster with: `publicKey` from the `kilo.squat.ai/wireguard-public-key` annotation, `endpoint` from `kilo.squat.ai/force-endpoint`, and `allowedIPs` containing the node's per-node pod-CIDR.
	2. Lists host cluster Nodes; for each node, ensures a Peer exists in the tenant cluster with: publicKey from the kilo.squat.ai/wireguard-public-key annotation, endpoint from kilo.squat.ai/force-endpoint (falling back to Node IP if missing), and allowedIPs containing the node's per-node pod-CIDR.

[gemini-code-assist]

To ensure that Peer objects created in the tenant cluster are reliably cleaned up when a TenantMeshLink is deleted, the operator should use Kubernetes finalizers. Without finalizers, if the operator is unavailable or the tenant API is unreachable during deletion, orphaned Peer resources will remain in the tenant cluster.

Suggested change

	4. Removes orphaned Peer objects on either side using a label selector tied to the `TenantMeshLink` name.
	4. Uses finalizers to ensure all remote Peer objects are removed from the tenant cluster before the TenantMeshLink is deleted.\n5. Removes orphaned Peer objects on either side using a label selector tied to the TenantMeshLink name.

[gemini-code-assist]

The disjointness requirement should be extended to include the Service CIDRs of both clusters. Overlaps between a tenant's pod-CIDR and the host's Service CIDR (or vice versa) can cause routing conflicts, making it impossible for pods to reach internal services or the advertised host services.

Suggested change

	- The host pod-CIDR and every tenant pod-CIDR must be pairwise disjoint.
	- The host pod-CIDR, host service-CIDR, and every tenant pod-CIDR must be pairwise disjoint.

[squat]

unless it is the cluster running the controller, in which case I guess they do get perms on peer clusters, but that's not new

[kvaps]

Yes, intentional — that's the design. The controller cluster does have perms on remote Peers via the kubeconfigs it holds, and that's exactly the trust direction we want.

[squat]

Should any of these be named fields in the struct rather than open fields in allowedNetworks? If the WireGuard mesh CIDR and the Pod CIDR are mandatory then maybe they get special treatment? Alternatively, using the open list can later be easily migrated into the stricter design

[squat]

I guess these are all technically optional and it just determines which networks from the Peer resources we want to honor / validate

[kvaps]

Kept the flat list for v1alpha1 — it's intentionally narrow ('nothing else than what's listed can end up in a Peer') and easy to migrate to typed fields later if needed. Recorded as Open Question #6 in 8830a7c.

[kvaps]

Right, exactly. The list is purely about what's permissible / honoured — not categorising CIDRs into kinds. That's why we kept it flat.

[squat]

This is kind of annoying, it means that there is functionally no difference between the cluster admin and the mesh admin. If you want to create a new mesh, then you have to edit the mesh controller deployment to add the allow list. I need to think about this a bit. What are we hoping to defend against here?

[kvaps]

Agreed, and dropped. The --allowed-cidr allowlist is gone in 8830a7c. The current design relies on RBAC on ClusterMesh as the address-surface chokepoint, and cluster-side defense-in-depth is now tracked in Alternatives as future Kilo work (your PeerClass suggestion below).

[squat]

Maybe in keeping with Kubernetes convention this should become a list of named structs, like how a Pod contains a list of named containers.

[kvaps]

Done in 8830a7c — spec.clusters is now a list of named entries (- name: cluster-a, ...).

[squat]

We should probably also add labels for the source cluster name so if two controllers running on different hosts are managing meshes on the same tenant (some triangle where the two hosts don't know about each other) then the controllers are less likely to compete for ownership of Peers if the mesh object has the same name.

[kvaps]

Added in 8830a7c — every generated Peer now carries kilo-clustermesh.io/mesh: <mesh-name> and kilo-clustermesh.io/source-cluster: <cluster-name>, with the controller's ownership scoped to its own (mesh, source-cluster) pairs. Two controllers can coexist on the same remote cluster as long as they're not declaring the same source cluster.

[squat]

The more I read about the controller allowlist, the more o actually started leaning in this direction. Maybe this needs to be a flag on Kilo, actually (or an entirely new PeerClass resource that declares what allowed IPs are permissible for every Peer in a cluster). This would allow individual clusters to guard against peers being created by a rogue cluster mesh controller. It's not blocking: this is orthogonal Kilo work that would be great to upstream to improve the administration of Kilo meshes.

[kvaps]

Agreed — tracked in Alternatives as future Kilo work in 8830a7c, with credit to you. Independent of this ClusterMesh proposal but a clear next direction for hardening Kilo administration.

[kvaps]

(withdrawn)

[Arsolitt]

Two related scalability concerns that I think are missing from the proposal.

Aggregate fan-out across tenants

The scaling analysis covers a single ClusterMesh, but on the host side peers from all tenants accumulate on every node:

total_wg_peers_per_host_node = Σ(nodes_in_tenant_i)

• Small (10 tenants × 10 nodes) → 100 peers/host node, ~3K Peer CRs in etcd
• Medium (50 × 15) → 750 peers/host node, ~22K Peer CRs
• Large (200 × 20) → 4,000 peers/host node, ~120K Peer CRs

WireGuard kernel datapath is fine (O(1) AllowedIPs lookup), but the control plane is a different story:

• etcd: tens of thousands of Peer objects, each generating watch events
• kg-agent: receives ALL Peers, rebuilds wg setconf on every change - at 4K peers that's ~200KB config reparsed per event
• controller: reconcile loop across all ClusterMesh CRs × all nodes × all peers

The proposal needs a scalability section with realistic limits and guidance on when to shard tenants across host clusters.

Full bipartite without topology awareness

Every node in cluster A gets a tunnel to every node in cluster B. In heterogeneous clusters only a subset of nodes run the target workload - tunnels to irrelevant nodes carry zero useful traffic.

A nodeSelector on the cluster spec would scope the mesh to relevant nodes:

spec:
  clusters:
    - name: host
      local: true
      allowedNetworks: [...]
      nodeSelector:
        storage-role: ceph

If 10 of 30 host nodes are selected, per-tenant peer count drops by 3x. Across 50 tenants: 750 → 250 peers per host node.

Combined effect

These multiply: topology awareness reduces per-mesh fan-out, aggregate analysis tells you how many meshes you can sustain. Without both, there's no way to give operators capacity planning guidance.