GKE Ingress controller incompatible ingress created with garden deploy

[worldofgeese]

Bug

Garden creates ingresses for module type container incompatible with GKE clusters using the default GKE ingress controller which results in an error when trying to reach the created load balancer's IP: response 404 (backend NotFound), service rules for the path non-existent.

Current Behavior

Given module type container with the following declaration:

kind: Module
type: container
name: vscode-remote-try-python
dockerfile: Dockerfile
services:
  - name: frontend
    ports: 
      - name: http
        containerPort: 5000
        servicePort: 80
    ingresses:
      - port: http

and a project file:

kind: Project
name: vscode-remote-try-python
environments:
  - name: remote
providers:
  - name: kubernetes
    context: gke_devrel-348008_europe-north1_gke-autopilot-north1-demo-1
    defaultHostname: vscode-remote-try-python.devrel.demo.garden
    deploymentRegistry:
      hostname: "docker.io"
      namespace: "worldofgeese"

Garden will create the following ingress declaration:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    garden.io/generated: "true"
    garden.io/manifest-hash: a238bf58e700e09f5ef8559c737f0aeb3114434d9ec74199378f9fe810ec72dc
    garden.io/version: v-0b9668f3fd
    ingress.kubernetes.io/backends: '{"k8s-be-32038--57c5a8cb84a64a0c":"HEALTHY","k8s1-57c5a8cb-vscode-remote-try-python-defa-frontend-8-61461ea1":"HEALTHY"}'
    ingress.kubernetes.io/force-ssl-redirect: "false"
    ingress.kubernetes.io/forwarding-rule: k8s2-fr-bh145e40-vscode-remote-try-python-def-frontend-toort30l
    ingress.kubernetes.io/target-proxy: k8s2-tp-bh145e40-vscode-remote-try-python-def-frontend-toort30l
    ingress.kubernetes.io/url-map: k8s2-um-bh145e40-vscode-remote-try-python-def-frontend-toort30l
  labels:
    module: vscode-remote-try-python
    service: frontend
  name: frontend-0
  namespace: vscode-remote-try-python-default
spec:
  rules:
  - host: vscode-remote-try-python.devrel.demo.garden
    http:
      paths:
      - backend:
          service:
            name: frontend
            port:
              number: 80
        path: /
        pathType: Prefix

Google's own example documentation for creating container-native ingresses uses a different key to declare a backend service and service path:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: neg-demo-ing
spec:
  defaultBackend:
    service:
      name: neg-demo-svc # Name of the Service targeted by the Ingress
      port:
        number: 80 # Should match the port used by the Service

Expected behavior

garden deploy should create an ingress resource compatible with the default GKE ingress controller.

Workaround

If one replaces the Garden's rules key with defaultBackend, a user should be able to access the service directly by IP:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    garden.io/generated: "true"
    garden.io/manifest-hash: a238bf58e700e09f5ef8559c737f0aeb3114434d9ec74199378f9fe810ec72dc
    garden.io/version: v-0b9668f3fd
    ingress.kubernetes.io/backends: '{"k8s-be-32038--57c5a8cb84a64a0c":"HEALTHY","k8s1-57c5a8cb-vscode-remote-try-python-defa-frontend-8-61461ea1":"HEALTHY"}'
    ingress.kubernetes.io/force-ssl-redirect: "false"
    ingress.kubernetes.io/forwarding-rule: k8s2-fr-bh145e40-vscode-remote-try-python-def-frontend-toort30l
    ingress.kubernetes.io/target-proxy: k8s2-tp-bh145e40-vscode-remote-try-python-def-frontend-toort30l
    ingress.kubernetes.io/url-map: k8s2-um-bh145e40-vscode-remote-try-python-def-frontend-toort30l
  labels:
    module: vscode-remote-try-python
    service: frontend
  name: frontend-0
  namespace: vscode-remote-try-python-default
spec:
  defaultBackend:
    service:
      name: frontend
      port:
        number: 80
  # rules:
  # - host: vscode-remote-try-python.devrel.demo.garden
  #   http:
  #     paths:
  #     - backend:
  #         service:
  #           name: frontend
  #           port:
  #             number: 80
  #       path: /
  #       pathType: Prefix

Your environment

• OS: Windows 11 running Ubuntu via Windows Subsystem for Linux 2
• How I'm running Kubernetes: GKE Autopilot v1.22.10-gke.600

garden version 0.12.43

[twelvemo]

This would work well for the GKE specific ingress controller, but not for every other ingress controller. A defaultBackend is conventionally configured on the ingress controller and not on every ingress resource.
https://kubernetes.io/docs/concepts/services-networking/ingress/#default-backend
Garden users would not be able to configure different paths with rules when we only specify defaultBackend. See https://kubernetes.io/docs/concepts/services-networking/ingress/#ingress-rules
If this should be implemented in Garden we would need to add a stanza to specify a defaultBackend instead of rules. Personally i think this is a very specific usecase that warrants the use of a kubernetes module and not a container module.

[worldofgeese]

My gut feeling is Garden should "just work" with Big Cloud without any additional configuration by a user. They're large enough to each have their own idiosyncrasies and these idiosyncrasies should be abstracted away from the user as much as possible so time to productive is as close to zero as we can manage. I don't have any specific technical suggestions to provide as to how we might implement this but philosophically I think it's important a solution is invisible and presents a generic, agnostic interface.

[stale]

This issue has been automatically marked as stale because it hasn't had any activity in 90 days. It will be closed in 14 days if no further activity occurs (e.g. changing labels, comments, commits, etc.). Please feel free to tag a maintainer and ask them to remove the label if you think it doesn't apply. Thank you for submitting this issue and helping make Garden a better product!

[stale]

This issue has been automatically marked as stale because it hasn't had any activity in 90 days. It will be closed in 14 days if no further activity occurs (e.g. changing labels, comments, commits, etc.). Please feel free to tag a maintainer and ask them to remove the label if you think it doesn't apply. Thank you for submitting this issue and helping make Garden a better product!

[10ko]

We are not going to work on this, please use the kubernetes action type if you encounter errors.