Enabling mTLS breaks Readiness/Liveness probes

[fl-max]

Enabling mTLS on the Alpha nodes breaks the livenessProbe and readinessProbe. A workaround is to run a command probe instead but the best fix IMO is to allow the /health path to be served on HTTP instead.

[fl-max]

My workaround to use a command probe via curl has issues too:

# curl https://dgraph-alpha-0:8080/health --cacert /dgraph/tls/ca.crt --cert /dgraph/tls/client.dgraphadmin.crt --key /dgraph/tls/client.dgraphadmin.key
curl: (16) Error in the HTTP2 framing layer
{"version":"v1.2.1","instance":"alpha","uptime":248}

This is a server-side error but there may be a fix (see here) in curl v7.65.0 (see changelog) to help mitigate it on the client-side. However, v7.58.0 is the latest for Ubuntu 18.04 with v7.68.0 only being available in experimental releases.

EDIT:
A workaround to the above is to force HTTP/1.1:
curl https://dgraph-alpha-0:8080/health --cacert /dgraph/tls/ca.crt --cert /dgraph/tls/client.dgraphadmin.crt --key /dgraph/tls/client.dgraphadmin.key --http1.1

[darkn3rd]

I am closing this issue as it is a server side issue and nothing can be done as far as helm chart code.

For general TLS, I recommend using cert-manager with ingress or api gateway for north-south traffic across an endpoint into the k8s cluster, and for east-west traffic within the k8s cluster, use a service mesh solution. Managing TLS outside of automation requires a lot of human resources.