[question] high availability support?

[jsiebens]

Seeing the number of replicas set to two in the k8s examples, I was just wondering if headscale supports such a HA setup?
What if nodes are contacting different instances of the headscale server?

[SilverBut]

Not sure about if the database transactions are well managed. If so, HA might be simple within same region: just use a shared MySQL database, if the service itself is not stateful.

But I think this is not what you mean (and also not mine), since deploying service in different regions is still not possible (for example, two servers running in Russia and Japan). Maybe we should consider either support some distributed database (like TiDB) so state can be synced via database, or we should use something like raft or paxos to build a cluster.

[juanfont]

We need to think a bit about it. It is not trivial with the current architecture, as a TCP connection is opened and kept from the clients to the server.

This connection is used for keepalives and sending network map updates to the client. Should we have more than one server instance, we would need a mechanism to have cross-headscale communication to notify the peers polling in different instances - which requires some changes in our side.

On the other hand, having the control server down is not great, but not immediately terrible. Everything keeps working, but slowly decaying (Tailscale.com has a KB article on this https://tailscale.com/kb/1091/what-happens-if-the-coordination-server-is-down/)

New users and devices cannot be added to the network.
Keys cannot be refreshed and exchanged, meaning that existing devices will gradually lose access to each other.
Firewall rules cannot be updated.
Existing users cannot have their keys revoked.

Hope this helps...

[jsiebens]

hi @juanfont, that clarifies a lot, thanks for the feedback!

[SuperPauly]

Did anyone eventually work on this? Or is there another HA solution for Headscale?

[TKinslayer]

Maybe I'm coming a bit late to the party but basically, the same : any plan to add High Availability like Tailscale does ? (only about the failover part with route advertising). That would make headscale production ready in my book and I could start using it at work.

[nickdickinson]

Could each headscale peer post their IP/address in the database and then they have each an API endpoint to notify if there is a network change to be found in the database (or however this should actually work)? Anyway I guess it is a mute point if it is not in the roadmap