How to do HA catalog-sync

[jacksontj]

While looking into using consul-k8s for my cluster I am unable to find any docs/code/comments/issues regarding an HA setup for consul-k8s. In addition to the concerns in #57 RE liveliness/readiness checks -- how do I have more than one of these running?

I have basically 2 HA concerns: (1) failure duration and (2) failure impact.

1. Failure duration
   Assuming Liveliness/Readiness checks? #57 is resolved, we could in theory have more than one pod running behind a lock (presumably in consul) such that when the first pod fails the second can kick in pretty quickly. This would help mitigate the time that there isn't a consul-k8s working in the cluster

2. failure impact
   As it is today consul-k8s is a single worker which syncs the entire state of consul/k8s -- I'm interested in ways of reducing the amount of work it needs to do specifically to reduce impact in a failure. As an extreme example (which I realize doesn't exactly work given the current design, but it illustrates the point) if we had consul-k8s running on each node in a k8s cluster and was responsible for syncing the state of things local to the node (as I said before, doesn't quite work for some things like LBs etc.) then the failure of a single process impacts the sync state of all pods on that node.

[tsmgeek]

K8s should handle this if you set it up as a "Deployment", but only in the instance that the executable within the pod crashes or stops. If for any reason it just stops syncing but stays running it will cause an issue.
I guess what they need to have an open port that responds OK if the last sync was within X, then you can monitor this using a liveliness check and it will kill the pod.

[jacksontj]

After a lot of investigation it seems that fundamentally our needs won't be met by consul-k8s (a combination of this and #57) -- which lead me to create katalog-sync, the highlights are:

• syncing direct to a consul-agent: this means syncing is scoped per-node (when deployed as a deemonset)
• agent-services in consul: meaning the health of services is tied to node health
• sync readiness state from k8s as check to consul: meaning you only have to define the check in k8s and consul will reflect that state
• (optional) sidecar to put in with your services to go "ready" when the pod is registered in consul.

IMO these questions should still be answered by consul-k8s, but some of them (specifically the failure impact and readiness checks during pod startup) seem to not be tractable with the current design (although it does allow for syncing cluster-wide resources such as services). So it seems like maybe a combination is the best way to go?

[tsmgeek]

Agreed there seems to be key functionally missing from consul-k8s.
Im still working on another problem related to services doing hide and seek.

[hmlkao]

Any best practice or progress how to run consul-k8s in HA mode?

We solved an issue that reschedule to another node takes long time (5:40 min with default K8s component values) when node on which was running consul-k8s pod falls down. Even after tuning K8s component values it takes much longer (~30s) than it would be acceptable.

[lkysow]

Any best practice or progress how to run consul-k8s in HA mode?

We solved an issue that reschedule to another node takes long time (5:40 min with default K8s component values) when node on which was running consul-k8s pod falls down. Even after tuning K8s component values it takes much longer (~30s) than it would be acceptable.

Hi, which consul-k8s pod got rescheduled? Are you using service mesh or catalog sync?

[hmlkao]

Hi @lkysow , we are using catalog sync without service mesh, deployed over consul-helm (v0.8.1), in one-way sync K8s > Consul mode.

[lkysow]

Do you know why the rescheduling of the catalog sync pod took 5:40?

[hmlkao]

It is standard behaviour of K8s with default values of controller-manager node-monitor-grace-period (40 s) and pod-eviction-timeout (5 min), details are in article at Medium.

When node on which is catalog-sync pod running fails (dies or is reset or whatever). It takes 5:40 min until the pods are evicted. during this time are not any changes synced to Consul and it can cause problems with access to apps.

Is there any dangerous to run consul-k8s catalog-sync in more replicas?

[lkysow]

Ahh I see, sorry I missed it was because the node died.

Is there any dangerous to run consul-k8s catalog-sync in more replicas?

Currently that is not supported unfortunately but it looks like we need to fix this so you can run >1 replicas and maybe use some sort of leader election to swap between them.

[hmlkao]

It would be great if it was supported.

Meanwhile I did some tests with more running replicas on our dev cluster and it looks like it works as expected.
It can (maybe) cause some race condition but it is more acceptable then outage caused by failed node.

[lkysow]

Note: #479 will make the service mesh injector HA but it won't address the catalog sync HA.

[david-yu]

Hi there, Consul K8s PM here. I'm going to close this issue since at this time we likely won't be making changes to support running catalog sync on multiple replicas. I have to acknowledge that it's been a long time since this issue was filed. Since that time, our priorities have shifted towards building a robust service mesh which enables service discovery on K8s. If you do have an PRs that you would like us to review that enables multiple replicas, please go ahead and file a PR and we can review. Thank you.

[Dentrax]

Hey @david-yu, any reason why consul-k8s does not support leader election as of now? It'd be great to learn some background. I can file a proposal for this if reasonable. Then a follow-up PR to add leader-election since I have previous experience of adding this to some open-source projects. Our use-case is to run consul-k8s in HA mode. (podAntiAffinity + min 3 replicas). Waiting your thoughts.

[david-yu]

Hi @Dentrax if you're interested in working on a PR could you file a new issue with a proposal of how you'd like to see this problem solved? We probably would like to review before you go down this path. As said previously this is something that probably involves even deeper changes to Catalog sync than just deploying multiple replicas.