[FEATURE]: Ability to use kustomizations with remote sources

[stefreak]

Feature Request

I would like to import Kubernetes manifests using the remote sources, and then apply kustomizations on top of them.

Background / Motivation

Sometimes it is useful to import a kubernetes manifest from another git repository: For example when deploying the grafana helm chart, it would be nice to be able to deploy kubernetes grafana dashboards from https://github.com/prometheus-operator/kube-prometheus/blob/main/manifests/grafana-dashboardDefinitions.yaml

What should the user be able to do?

Currently this fails:
Files:

garden.yml
kustomize-add-annotation
kustomize-add-annotation/kustomization.yaml

garden.yml

kind: Project
apiVersion: garden.io/v1
name: repro
environments:
 - name: default
providers:
 - name: local-kubernetes
 
---
kind: Deploy
type: kubernetes
name: test
source:
  repository:
    url: https://github.com/prometheus-operator/kube-prometheus.git#0198f98f682f1d7e40b1dd5615437a916107f1aa
spec:
  kustomization:
    path: ./kustomize-add-annotation
  files:
    - manifests/grafana-dashboardDefinitions.yaml

with the error message:

✖ deploy.test          → Failed resolving status for Deploy type=kubernetes name=test (took 0.16 sec). Here is the output:

Failed resolving kustomize manifests: Command "/Users/steffen/.garden/tools/kustomize/e1db5bee834d3301/kustomize build ./kustomize-add-annotation" failed with code 1:

Error: evalsymlink failure on './kustomize-add-annotation' : lstat /Users/steffen/repro/.garden/sources/action/deploy.test--481d97f2d9/kustomize-add-annotation: no such file or directory

The reason for that is that the kustomization.yaml is not part of the action's context under .garden/sources/action/deploy.test--<version> because we are using remote sources.

This means there is no way to run kustomization with remote sources (This is true also for Garden 0.12 – using the dependencies.copy feature has no effect for service deployments)

Why do they want to do this? What problem does it solve?

it would be nice if I could use garden to get Kubernetes manifests from another git repository, and then add kustomizations on top of them. Otherwise I need to import the source code into my git repository, or create helm charts which might not be available for my use case.

Suggested Implementation(s)

There would be two options here:

1. We allow copyFrom in any action kinds. Then I could have a type=exec kind=Build action that imports the kubernetes manifest using remote sources, and import the manifest I need into the deploy action using copyFrom. That way the kustomization.yaml would be also part of the action's context.
2. We allow inline kustomization; could look like this:

kind: Deploy
type: kubernetes
name: test
source:
  repository:
    url: https://github.com/prometheus-operator/kube-prometheus.git#0198f98f682f1d7e40b1dd5615437a916107f1aa
spec:
  kustomization:
    # Add grafana_dashboard annotation
    inline:
      apiVersion: kustomize.config.k8s.io/v1beta1
      kind: Kustomization
      resources:
        - manifests/grafana-dashboardDefinitions.yaml
      commonLabels:
        grafana_dashboard: "1"
  files:
    - manifests/grafana-dashboardDefinitions.yaml

A big benefit of the second approach would be that it would make it possible to use Garden template variables and function inside the kustomization yaml.

How important is this feature for you/your team?

🌵 Not having this feature makes using Garden painful

[stefreak]

Found a workaround in theory, but it does not work due to #4506

# Get prometheus-operator Grafana default dashboards
kind: Build
type: exec
name: github-kube-prometheus
source:
  repository:
    url: https://github.com/prometheus-operator/kube-prometheus.git#3fff8b56097778f1491edba968aee3cfbd5652ef
---
# Combine the dashboards we need with the kustomization we need
kind: Build
type: exec
name: grafana-default-dashboards-with-kustomization
dependencies:
 - build.github-kube-prometheus
copyFrom:
  - build: github-kube-prometheus
    sourcePath: manifests/grafana-dashboardDefinitions.yaml
    targetPath: grafana-prometheus-default.yaml
include:
 - kustomize-add-label/**/*
 - grafana-prometheus-default.yaml
---
# Deploy the default dashboards after applying the kustomization
kind: Deploy
type: kubernetes
name: grafana-default-dashboards
description: Additional grafana dashboards
dependencies:
 - build.grafana-default-dashboards-with-kustomization
build: grafana-default-dashboards-with-kustomization
spec:
  namespace: monitoring
  kustomize:
    path: ./kustomize-add-label/
  files:
    - grafana-prometheus-default.yaml

[stefreak]

I tested that this is possible now after #4506 has been merged.

I rewrote the example above, making it a little bit easier to understand:

# Combine remote source with kustomization
kind: Build
type: exec
name: remote-source
dependencies:
 - build.kustomization
copyFrom:
  - build: kustomization
    sourcePath: kustomization
    targetPath: kustomization
source:
  repository:
    url: https://github.com/prometheus-operator/kube-prometheus.git#3fff8b56097778f1491edba968aee3cfbd5652ef
---

# Build action makes the kustomization available in `copyFrom` for the `remote-source` action
kind: Build
type: exec
name: kustomization
include:
 - kustomization/**/*.yaml
---

# Deploy the default dashboards after applying the kustomization
kind: Deploy
type: kubernetes
name: grafana-default-dashboards
description: Additional grafana dashboards
build: remote-source
spec:
  namespace: monitoring
  kustomize:
    path: ./kustomization/
  files:
    - manifests/grafana-dashboardDefinitions.yaml

With #5082 resolved we have a more elegant option to accomplish the same thing (altering existing manifests from remote sources):

# Deploy the default dashboards after applying the kustomization
kind: Deploy
type: kubernetes
name: grafana-default-dashboards
description: Additional grafana dashboards

source:
  repository:
    url: https://github.com/prometheus-operator/kube-prometheus.git#3fff8b56097778f1491edba968aee3cfbd5652ef

spec:
  namespace: monitoring
  files:
    - manifests/grafana-dashboardDefinitions.yaml

  # Adds a grafana_dashboard label to the grafana-dashboard-workload-total ConfigMap defined in the remote source
  patchResources:
    - name: grafana-dashboard-workload-total
      kind: ConfigMap
      patch:
        metadata:
          labels:
            grafana_dashboard: "1"

patchResources is available in the bonsai-edge pre-release of Garden already, and will be available in stable Garden releases in versions 0.13.18 and onwards.

[stefreak]

I've tested that applying patches on top of kustomizations works; So those two approaches can even be combined.

I think that this issue is therefore fully resolved for now.