The health endpoint should have its own listener TLS configuration

[ironcladlou]

Problem

It seems there is no way to simultaneously support mTLS authenticated /metrics and HTTP/S (not mTLS) /health endpoints.

The /health endpoint shares a listener with the /metrics endpoint. Using the -–client-cert-auth option thus subjects both endpoints to mTLS authentication. However, the use cases for metrics and health probing could be quite different. Metrics might tend to be secured, while health endpoints may be subject to less strict authentication requirements.

The health check docs seem to acknowledge this problem area, but in a way that's slightly confusing to me:

Since v3.3.0, in addition to responding to the /metrics endpoint, any locations specified by --listen-metrics-urls will also respond to the /health endpoint. This can be useful if the standard endpoint is configured with mutual (client) TLS authentication, but a load balancer or monitoring service still needs access to the health check.

The health endpoint seems like the practical choice for a Kubernetes HTTP health probe, but the probe APIs don't support mTLS.

This leaves users left implementing health checks through something like exec probes, which can add unnecessary complexity and inefficiency.

Solutions

One solution might be to decouple the health and metrics endpoint listeners. Then health authentication must be addressed in some way to provide the needed flexibility. For example:

• For the health listener, honor global the global TLS configuration by default (for compatibility with today's behavior), but provide a new flag to allow control over health endpoint client certificate auth (e.g. --health-client-cert-auth=false).
• For the health listener, introduce a complete parallel set of TLS configurations to allow granular control of the health endpoint independent of metrics.

Notes

Of course, if I've missed something and my core claim is invalid:

It seems there is no way to simultaneously support mTLS authenticated /metrics and HTTP/S (not mTLS) /health endpoints.

I'd love to understand how to implement this without any changes.

[wenjiaswe]

It seems there is no way to simultaneously support mTLS authenticated /metrics and HTTP/S (not mTLS) /health endpoints.

I think you are right, like you said "The /health endpoint shares a listener with the /metrics endpoint."

One solution might be to decouple the health and metrics endpoint listeners.

I think that makes sense. But I would like to defer to @gyuho @xiang90 @jingyih for opinion.

[wenjiaswe]

cc @hexfusion

[ptabor]

I wonder if usage of gRPC proxy is an option for your use-case:

It seems to register /health and /metrics handlers [1].
The /metrics request is being forwarded to underlying etcd server [2].

So e.g. you might expose mTLS protected etcd server and not-mTLS protected proxy.
Is your goal to guarantee that /metrics can be only accessed over mTLS or availability of not protected endpoint is still OK ?

Finally the proxy seems to define a pattern (flag --grpcProxyMetricsListenAddr) [3] to expose additional listeners.

[1]

etcd/etcdmain/grpc_proxy.go

Line 21 in 2b79442

"fmt"

[2]

etcd/proxy/grpcproxy/metrics.go

Line 79 in 2b79442

mux.HandleFunc(pathMetrics, func(w http.ResponseWriter, r *http.Request) {

[3]

etcd/etcdmain/grpc_proxy.go

Line 125 in 2b79442

cmd.Flags().StringVar(&grpcProxyMetricsListenAddr, "metrics-addr", "", "listen for endpoint /metrics requests on an additional interface")

[ironcladlou]

@ptabor

I wonder if usage of gRPC proxy is an option for your use-case:

It seems to register /health and /metrics handlers [1].
The /metrics request is being forwarded to underlying etcd server [2].

So e.g. you might expose mTLS protected etcd server and not-mTLS protected proxy.
Is your goal to guarantee that /metrics can be only accessed over mTLS or availability of not protected endpoint is still OK ?

Finally the proxy seems to define a pattern (flag --grpcProxyMetricsListenAddr) [3] to expose additional listeners.

[1]

etcd/etcdmain/grpc_proxy.go

Line 21 in 2b79442

"fmt"

[2]

etcd/proxy/grpcproxy/metrics.go

Line 79 in 2b79442

mux.HandleFunc(pathMetrics, func(w http.ResponseWriter, r *http.Request) {

[3]

etcd/etcdmain/grpc_proxy.go

Line 125 in 2b79442

cmd.Flags().StringVar(&grpcProxyMetricsListenAddr, "metrics-addr", "", "listen for endpoint /metrics requests on an additional interface")

Thanks for the idea. We are using grpc-proxy, but I'm not clear the approach you've describes allows us to continue securing the /metrics endpoint with mTLS, which is a requirement. Ultimately the client (the proxy being an example) is limited to communicating with the single TLS-configured listener on the etcd side, so I'm not sure yet how the proxy can ovecome the coupling issue.

[ironcladlou]

#10504 adds independent TLS configuration for metrics, and just based on the commentary there it seems that PR is still desirable. If so I wonder if the metrics TLS work should come first. I could do a PR that essentially copies the approach for health, but it really feels like overkill. I still wonder if something like an opinionated --insecure-health-endpoint=true would be easier and just as useful.

[ironcladlou]

#12370 is an attempt to address the problem in a much narrower and less intrusive way.