Flagger support for Consul

[oleksiyp]

Feature Description

I would like to install one helm chart in several DCs and get Consul Connect service mesh out of the box.

Use Case(s)

• multi-DC failover
• canary deployments

My experience

I spent a few weekends to make this work on PoC level. And here is my experience.

Problems I faced:

• standard helm chart doesn't support multi-DC scenario
• there are no tools for canary deployments
• not clear procedure to create common trust domain
• multi-DC failover doesn't work without health-check
• exposing health-check in Kubernetes doesn't work
• standard statistics exposed through statsd is limited

Didn't find in docs:

• how to generate SPIFFE SVID (thus used Vault, although it might be overkill for some cases)
• mention that ClusterID and thus trust domain should be same between all DCs

Modified:

• consul-k8s project to expose HTTP health checks
• consul-helm project to make join WAN possible
  • distribution of servers and clients by corresponding node labels
  • expose hostPorts
• flagger to support Consul Connect Canary deployment and multi-DC failover

I added to service chart additional envoy instance to expose statistics to Prometheus directly through envoys /metrics (although now I realize that I potentially can just expose this path)

Created test-bed:

• simple ingress forwarding traffic from outside: https://github.com/oleksiyp/consul-connect-multi-dc/tree/master/ingress-chart
• Kubernetes controller called prefixrouter to forward traffic to services based on CRD definitions: https://github.com/oleksiyp/consul-connect-multi-dc/tree/master/prefixrouter-chart
• Service instance which randomly picks up next service-to-service call and does a fixed amount of hops: https://github.com/oleksiyp/consul-connect-multi-dc/tree/master/service-ktor
• UI client that shows how traffic is distributed: https://github.com/oleksiyp/consul-connect-multi-dc/tree/master/ingress-js

As a result:

• I joined Consul Connect DCs in two Kubernetes clusters(kind)
• I exposed health-check and have an ability to create liveness/readiness probe
• I am forwarding traffic through mesh-gateways by sharing ClusterID
• I am able to perform Flagger deployments and shift traffic from old version to new version
• based on exposed health-checks traffic is redirected to other DC in case of failure

All setup available here: https://github.com/oleksiyp/consul-connect-multi-dc

[oleksiyp]

@nicholasjackson here is full context of my changes related to fluxcd/flagger#482

[jsosulska]

This relates to #7412 . Cross linking for feature request.

[david-yu]

Hi @oleksiyp

Thank you for your feedback. We just released WAN Federation over Mesh Gateways for Kubernetes to enable the multi-dc workflow: https://www.consul.io/docs/k8s/installation/multi-cluster/overview

In addition we are working towards enabling our newly built Ingress Gateway for Consul 1.8 beta (https://www.consul.io/docs/connect/ingress_gateway) into the Helm chart for deployment as well.

As far as Flagger, we have seen requests for flagger in the past. We'll take a review your PR as well so thank you for the PR.

[ssimard24]

Hi there, was there any movement on the flagger support?
We are currently using Consul for pod to pod encryption. But we are also moving towards a gitops solutions and therefore want to use flagger, prometheus, fluxcd, etc. to orchestrate deployments.
Without such a support, we will have to start evaluation other ServiceMesh solutions.

[nicholasjackson]

Hey, @ssimard24, @oleksiyp I have been chatting with Stephan from Weave and have created a new SMI controller that allows you to use Flagger without modifications. This is still pretty early stages but it works great for Flaggers requirement round Service Splitting. Personally, I love flagger and would not deploy without it, we are excited to bring this to Consul.

https://github.com/nicholasjackson/consul-smi-controller

I have also started to write up this information, and plan to produce a video within the next week to show how this all works.

https://github.com/nicholasjackson/consul-canary-deployment

Kind regards,

Nic

[ssimard24]

Great news Nicholas. Thanks for your help.

[ssimard24]

Hey Nicholas was there any more movement on the topic of Flagger? We are ready to start our proof of concept.

[shubhamsre]

@nicholasjackson could you please let know if there has been a progress on the feature, and is it still in POC phase or in GA. Please point to the official docs, if any. Thanks.