[GLBC] Support Host HTTP Header for GCE health checks

[itamaro]

According to the GCE controller docs, one of the 2 options for passing health checks sent by the GCE load balancers is to expose an arbitrary URL as a readinessProbe - but with a caveat that if the readinessProbe requires "special headers or HTTPS" it defaults back to /.

My request is to make the HTTP Host Header a "non-special header", because it is quite common, and it is explicitly supported by GCE health-checks. Also, this is explicitly mentioned in k8s docs as the way to work with pods that rely on vhosts.

I've hit this issue quite hard when using Ingress to expose a Django app - once I've set DEBUG=False, it went dark because / didn't respond with 200 and GCE load balancer didn't set the Host header which is whitelisted in Django production mode.

I was able to solve it by manually modifying the health-check to use /healthz and add the required Host header. Luckily, GLBC's control loop doesn't detect this as a change and undoes it... (but I'm hoping for a less manual solution)

Here's a sample YAML that can be used to reproduce this behavior:

apiVersion: v1
kind: Service
metadata:
  name: web-app
spec:
  type: NodePort
  ports:
  - port: 5000
    targetPort: web
    name: www
  selector:
    app: app
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: app
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: app
        version: "0.1"
    spec:
      containers:
      - name: app
        image: gcr.io/itamaro/app:0.1
        command: ["python", "-m", "app.app"]
        ports:
        - name: web
          containerPort: 5000
        readinessProbe:
          httpGet:
            path: /healthz
            port: web
            httpHeaders:
              - name: Host
                value: "app.example.com"
          periodSeconds: 30
          timeoutSeconds: 20
          successThreshold: 1
          failureThreshold: 3
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: app-ing
spec:
  backend:
    # Re-use fe as the default backend so we stay under the default
    # quota for gce BackendServices.
    serviceName: web-app
    servicePort: www
  rules:
  # app
  - host: app.example.com
    http:
      paths:
      - backend:
          serviceName: web-app
          servicePort: www

[nicksardo]

Thanks for creating this issue and preparing a PR.

There's some discussion about changing how GLBC creates health checks. The current methodology isn't a great indicator of anything other than a node's ability to talk to a pod. In the worst case (a single pod in a large cluster), N health checks hit the pod at every interval. We could address your issue, but it could be irrelevant before the next GLBC release if we move to health check kube-proxy instead of the service. Let's wait for decisions to be made on that front before merging/closing your PR.

[nicksardo]

/close