test(e2e): add cross mesh granularity

[kvaps]

Summary

This rebases #328 (--mesh-granularity=cross by @skirsten) onto current
main and adds the e2e test suite that was the only blocker for merge,
per the discussion in #489.

The three original commits from #328 are preserved as-is to keep
authorship attribution. A small conflict in docs/kg.md was resolved
in favour of the current --mtu description (which moved to the
auto default in #406) while keeping #328's addition of cross to
the granularity list.

What cross does

Direct WireGuard tunnels between every pair of nodes that live in
different locations; intra-location traffic stays on the CNI overlay.
This sits between location (one tunnel per location pair, leader as
relay → SPOF) and full (one tunnel per node pair, including
intra-location overhead).

New e2e suite (e2e/cross-mesh.sh)

Mirrors e2e/full-mesh.sh and e2e/location-mesh.sh:

• setup_suite annotates kind nodes into two locations (control-plane
  • first worker as loc-a, second worker as loc-b) so the test
    exercises the cross-location case
• test_cross_mesh_connectivity — pings + adjacency matrix
• test_cross_mesh_peer — kgctl peer create / showconf with
  granularity cross
• test_mesh_granularity_auto_detect — kgctl graph auto-detection
• test_cross_peer_topology — sanity that loc-a nodes only see the
  loc-b node as a peer (and vice versa), distinguishing cross from
  full (every node is a peer) and location (non-leaders have no
  peers at all)

Wired into the e2e: make target between location-mesh.sh and
multi-cluster.sh.

Refs

• Supersedes Add cross mesh granularity #328
• Addresses mesh-granularity=location: silently degraded leader blackholes relayed traffic with no failover #489 (location SPOF)

[skirsten]

Thanks for picking this back up.

FYI: While I was running this in my clusters for years and it worked great in general, there was one issue which I observed from time to time but could not track down. It might not even be related to this change:

When adding nodes to the cluster sometimes the networking would be weird and I was seeing a 10Hz (IIRC) reconcile metric on Grafana.
By rolling out a restart for the daemonset it resolved it every time. So I am pretty sure there is a race condition or some kind of distributed feedback loop happening.

I am not using it anymore at the moment so I don't have any concrete metrics or traces I can share unfortunately.

[kvaps]

@skirsten thanks for chiming in!

For the rebase + attribution: to keep this clean and merge under
your name, I pushed a ready-to-use branch with your three commits
cherry-picked onto current main, plus one fixture-only commit
that aligns the cross test cases with the new cniCompatibilityIPs
field on segment introduced by #409 (Cilium support). The branch
is at:

https://github.com/cozystack/kilo/tree/add-cross-mesh-granularity-rebased

You can fast-forward your fork's PR branch onto it with:

git remote add cozystack https://github.com/cozystack/kilo.git
git fetch cozystack add-cross-mesh-granularity-rebased
git checkout add-cross-mesh-granularity        # the branch behind #328
git reset --hard cozystack/add-cross-mesh-granularity-rebased
git push --force-with-lease origin add-cross-mesh-granularity

That refreshes #328 with three of your commits unchanged plus the
single follow-up. If you'd rather drop the fourth commit, that's
fine too — I can carry it in a separate PR after #328 merges. Once
your PR is in I'll close #490 and open a small follow-up with just
the e2e suite.

About the 10 Hz reconcile loop you saw: I had a quick look at the
current applyTopology path and couldn't pin down a clear cause,
but the reconcile path has moved on quite a bit since 2022 —
nodesAreEqual compares more fields now (CNICompatibilityIP,
Granularity, AllowedLocationIPs), and PR #409 adds annotation
write-back from each node. If the symptom resurfaces in current
master we can chase it as a separate issue with a kilo_reconciles_total
panel attached.