helm:Support for external Ingress controllers

[ghost]

I'm not sure if this is a pattern that's already supported, which if it is then the documentation isn't very clear.

I'm running multiple GKE clusters. They all have ingress controllers. They're very good ingress controllers that are very clever at what they do when you're building in Google Cloud.

I'm also running Consul as a Service Mesh. It's a very good service mesh, and I particularly enjoy injecting specific services across resources in a secure, mTLS-ed manner.

I want to leverage the proxy that is injected into these pods to secure the connection between the GCE Ingress controller, and the pods - not exactly an uncommon pattern.

Lets say I have a deployment manifest of an API like so:

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: prodigy-api
  labels:
    app: prodigy-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: prodigy-api
  template:
    metadata:
      annotations:
        consul.hashicorp.com/connect-service: "prodigy-api"
        consul.hashicorp.com/connect-inject: "true"
        consul.hashicorp.com/connect-service-port: "http"
      name: prodigy-api
      labels:
        app: prodigy-api
    spec:
      containers:
        - image: registry.gitlab.com/cto-uk/prodigy-api:<VERSION>
          name: prodigy
          ports:
            - containerPort: 8000
              name: http
          readinessProbe:
            httpGet:
              scheme: HTTP
              port: http
              path: /health

I can expose this deployment with in a pattern that allows GCE to expose it for all its Layer 7 HTTP(S) magic:

service.yaml

apiVersion: v1
kind: Service
metadata:
  annotations:
    beta.cloud.google.com/backend-config: '{"ports":{"80":"default"}}'
    cloud.google.com/neg: '{"exposed_ports": {"80":{}}}'
  name: prodigy-api
spec:
  externalTrafficPolicy: Cluster
  ports:
    - name: prodigy-api
      port: 80
      protocol: TCP
      targetPort: 8000
  selector:
    app: prodigy-api
  type: NodePort

Note the cloud.google.com/neg annotation - which means that GKE will create a Google NEG for the deployment, which can then be leveraged by the HTTP(S) Load Balancer (these concepts can get confusing all in I know)

I can then create a said HTTP(S) Load Balancer in the form of a GCE Ingress Controller:

ingress.yaml

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: prodigy-api
  annotations:
    kubernetes.io/ingress.allow-http: "false"
    kubernetes.io/ingress.class: "gce"
spec:
  tls:
    - secretName: cto-tls
    - hosts:
        - api.example.com
  rules:
    - host: api.example.com
      http:
        paths:
          - backend:
              serviceName: prodigy-api
              servicePort: 80
            path: /*

The issue

How do I go about securing the connection between the GCE LB, and the pods themselves?

The pods have the Consul sidecar attached, and the Envoy proxy.

Service Sync means that I have a first class Kubernetes Service available to reference in the Ingress manifest (it even has the cloud.google.com/neg annotation!) but it has no selector and thus no port, so the manifest is incomplete.

Any advice on how to put this together, or if I'm barking up the wrong tree entirely. TIA

[lkysow]

Hi!
For the ingress controller to talk to the connect-injected pods it needs to have an ssl certificate generated by Consul. I don't think it supports pulling that certificate from Consul so you'd need to either:

1. Use the ambassador ingress controller (which can request the right certs from Consul)
2. Keep the gce ingress controller but have it route to consul's ingress gateway which would then route to the connect pods
3. Use Consul's Ingress gateway on the edge (but it doesn't support Ingress objects)

Let us know if any of these options may work for you and we can go into more detail. See also https://www.consul.io/docs/k8s/connect/ingress-gateways, https://www.consul.io/docs/connect/gateways/ingress-gateway, https://www.consul.io/docs/k8s/connect/ambassador

[ghost]

Yeah so using another Ingress controller isn't really an option for us; we're far deeper in the Google Cloud stack than we are Consul, and the GCE Ingress Object works really well for us.

Of those 3 options then what we're left with is to front all of our services with another ingress controller, and have the GCE controller route to that; however that kinda defeats the point of the GCE ingress controller.

NEGs allow us to get much better performance out of our deployments -- tightly coupling the integration of HTTP(S) load balancing and all of its benefits -- with our backends.

Personally I think a much more flexible solution which would enable anyone to bring their own Ingress object should they so choose, would be to drop down some of the functionality of the Ingress into the sidecars.

Google can connect to secured backends, which Envoy provides in the form of the sidecar. Creating a policy in Consul which can be bound to the pod with the usual annotations that allows north-south traffic using the usual L4/L7 techniques means we're getting the best of both worlds here

[lkysow]

Do you know if there's a way to tell GCE which client certificate to use for its requests (between GCE and consul sidecars)? Then maybe we could have a process that is requesting a certificate for a service called "ingress-gateway" and supplying/updating that to the GCE controller config. Then you could create an intention from ingress-gateway => your-service.

The reason we don't support non TLS traffic into the connect pods is that this would bypass the whole secure service mesh.

[ghost]

You can't specify which client certificate to use for the HTTPS request from GCE, however thats not to say we couldn't build a strong cryptographic identity around the one we have now (especially if it's an inbound request)

The GCE annotations mean that the scheme is enforced, so it should just come down to how to define an intention of some design that allows this traffic

[lkysow]

I talked a bit more with the team. This functionality doesn't exist right now (intention allowing ingress given a certain ca cert) so we'd need to open up an issue in hashicorp/consul to ask for it.

For right now the only solutions I can think of are the ones I listed in my first comment.

[david-yu]

Going to close this since we now have general guidance on how to use Consul K8s with Ingress Controllers here: https://www.consul.io/docs/k8s/connect/ingress-controllers.

If you have follow up feedback or issues please file a new issue!

[blake]

This requires changes in Consul in order to support this type of deployment. We are using hashicorp/consul#10626 to track this feature request.

[david-yu]

For those following along, @blake is correct! The the docs I linked above do not address this feature request, and address the use case where you are using Consul service mesh with ingress proxies that are deployed in-cluster. This GH issue is requesting support for ingress proxies that are external to the cluster, such as a cloud-hosted ingress. We are tracking this in Consul Core via hashicorp/consul#10626