Support k8s-friendly gRPC deployments

[jalaziz]

Problem

I'll start by saying that I was hoping to contribute this myself, but haven't found the time yet. I may get to it this coming weekend, but figured I should file an issue first.

Deploying gRPC services on Kubernetes is notoriously troublesome without a service mesh. The problem is that gRPC uses HTTP/2 and the clients establish long lived connections which kube-proxy doesn't know how to load-balance.

Expecting users to use service meshes or linkerd is not very realistic. It turns out that there is another way, but it requires various things to be configured. The general steps are:

1. Set keepalive timeout parameters on the server to force periodic re-resolution of DNS on the client.
2. Deploy the server as a k8s headless service.
3. Configure the client to use DNS round-robin load balancing (e.g. dns:///flipt -- the third slash is significant)

This blog post has plenty of extra info.

For Flipt, I believe the changes required would be:

1. Add the keepalive parameters to the server.
2. Introduce a new headless k8s service to the helm chart (this allows the HTTP service to continue to be load-balanced by kube-proxy)

Ideal Solution

Both the flipt gRPC server and the helm chart should support client-side load balancing out of the box.

Search

• I searched for other open and closed issues before opening this

Additional Context

No response

[jalaziz]

Happy to discuss this in more details on Discord btw 🙏🏼

[GeorgeMac]

This all seems completely reasonable. We walked into many of these grpc rakes with k8s in my previous role, even with (sometimes because of) Isto our service mesh. Some much smarter than me team mates discovered all sorts of horrible pathological behaviours due to our usage of it (we had one hell of a DNS storm going on every time we rolled some of our deployments). Though we were doing something particularly messy, involving the need for clients to have full connectivity to every node in a large statefulset over gRPC and at one point we had the clients in the mesh and the stateful set out of the mesh (hell ensued).

Long story short, we ended up forgoing the mesh (just for that usecase) and using a headless service, so this seems very reasonable.

[markphelps]

@jalaziz this should be doable now with the latest release (v1.35)

@erka also updated the helm chart too! flipt-io/helm-charts#122

I have a docs pr up for the grpc config settings: flipt-io/docs#169

They will look like this:

Please let us know if this works for you!!