Best practices for multi-datacenter Geo-replication of GHES under strict air-gapped isolation?

[Tekhabeeb]

🏷️ Discussion Type

Question

Body

Hi all,

We are planning an upgrade and disaster-recovery architecture for an on-premise GitHub Enterprise Server (GHES 3.12+) deployment distributed across two geographically separated facilities.

We are planning an upgrade and disaster-recovery architecture for an on-premise GitHub Enterprise Server (GHES 3.12+) deployment distributed across two geographically separated facilities.

Because our primary workloads handle proprietary enterprise code and regulated build artifacts, both sites operate under strict physical and perimeter data center security frameworks (SOC 2 Type II / ISO 27001), including isolated rack cages, restricted outbound egress, and dedicated hardware security modules (HSM) for credential storage.

Our Proposed Topology:

• Primary Node: Bare-metal Dell PowerEdge R750 cluster in Tier-3 DC (Node A)
• Replica Node: Warm-standby cluster in secondary DC (Node B)
• Interconnect: Dedicated 10 Gbps private point-to-point dark fiber with IPsec encapsulation (MTU 1500)
• Storage: Hardware RAID-10 NVMe array behind independent fiber channel SANs

The Challenge:

While configuring asynchronous MySQL and Redis replication via ghe-cluster-config, we need to ensure that automated backup daemons (backup-utils) and replication heartbeats do not violate our microsegmentation firewalls or create unintended bridge vectors into internal server networks.

Questions for Enterprise Sysadmins & SREs:

1. For those maintaining geo-replicated GHES across segregated enterprise facilities, do you terminate IPsec tunnels directly on the host interface or handle it strictly at the DC perimeter edge switch?
2. How do you automate emergency failover (ghe-cluster-failover) when cross-facility communication drops without risking split-brain writes?

Appreciate any architecture tips or lessons learned from similar enterprise deployments!

[MrFabry]

A couple of things in the plan are worth checking before the IPsec part, because they change the answer.

GHES doesn't install on bare metal. It's a virtual appliance (VMware, Hyper-V, KVM or one of the clouds), so the R750s would be hypervisor hosts with GHES running as a VM on them. That also settles your first question in practice: there's no supported way to run your own IPsec stack inside the appliance, so it ends up on the network side, your edge firewalls or routers.

Also, the ghe-cluster-* commands are for clustering mode, which is a separate setup most instances don't use. A primary plus a warm standby is the normal high availability setup, done with ghe-repl-setup / ghe-repl-start, and the failover command is ghe-repl-promote instead of ghe-cluster-failover. Worth confirming which one you're actually on.

On the firewall side the surface is small. Between primary and replica it's 122 (admin SSH) and 1194/UDP, which the docs describe as the secure replication network tunnel for HA (network ports). backup-utils runs on its own host and only needs SSH on 122 to the appliance. So you can allow exactly those between the specific IPs and nothing else. One thing to watch: replication is already tunneled on 1194, so with IPsec on top and MTU 1500 on the link you're encapsulating twice. I'd test with big pushes and set MSS clamping on the IPsec side, because fragmentation there tends to look like replication stalling on large repos rather than a clear error.

On automatic failover, I wouldn't do it. Promotion is manual by design (the HA docs say you must manually promote the replica), and neither ghe-repl-promote nor ghe-cluster-failover has any quorum or witness logic. A script that promotes when the link between sites drops is exactly how you get two primaries taking writes, because from either side a dead site and a cut fiber look the same. Automate the detection and alerting, keep the promote as a human decision in a runbook, and make fencing the old primary the first step: maintenance mode if you can still reach it, otherwise shut it down or pull it from the load balancer before promoting.

And keep backup-utils running no matter what. The same page is clear that a replica doesn't replace backups, since corruption replicates too.