Using Datree with Kustomize

[exocode]

Is your feature request related to a problem? Please describe.

I try to implement DaTree into my Gitlab-CI. I primarly use Kustomize for my deployments. So my "root"-manifests are sparely written and are completed by "Kustomization" for the different stages like "staging" and "production".

So all my deployments are failing. Base, staging and production. But the "kustomized", rendered ones are definitely fullfil all rules.

Describe the solution you'd like
When I have Kustomize declared, it Datree should recognize it and only check the "rendered" ones. Maybe by specifiying "kustomization.yaml" files itself (filtering by names)

Additional context
DEMO:

base-deployment.yml

That file fails, because it does not apply resources

apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
  namespace: default
      containers:
        - name: frontend
          imagePullPolicy: Always
          image: registry.gitlab.com/app/frontend/master
          livenessProbe:
            httpGet:
              path: /healthz
              port: http
          name: sealed-secrets-controller
          ports:
            - containerPort: 8080
              name: http
          readinessProbe:
            httpGet:
              path: /healthz
              port: http
          securityContext:
            readOnlyRootFilesystem: true
            runAsNonRoot: true
            runAsUser: 1001

          resources: {} #### Have a look, it's empty, get's "filled" by staging-deployment

staging-deployment.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
    spec:
      containers:
        - name: frontend
          imagePullPolicy: Always
          image: registry.gitlab.com/app/frontend/master:0.0.1
          resources:
            requests:
              memory: "512Mi"
              cpu: "500m"
            limits:
              memory: "512Mi"
              cpu: "500m"

How to tell DaTree that it should only check "rendered" Kustomize manifests?

This is my current gitlab-ci.yml file which runs the tests:

test:
  stage: test
  script:
    - curl https://get.datree.io | /bin/bash
    - datree test **/*.yaml **/*.yml

[hadar-co]

@exocode thanks, great catch!

What do you think about a Kustomize plugin to address this? We are planning to write one and would love your input.

In the meantime I believe this workaround should resolve your issue:
$ kustomize build [resources] | datree test -

Please let us know if it did :)

[exocode]

Hi @hadar-co , I am not that great Kustomize expert, I am an average user.

with your hint, I can do something like this:

for filename in ./**/overlays/**/kustomization.yml; do eval "datree test ${filename}"; done

EDIT: That is a working example:

for filename in ./**/overlays/**/kustomization.yml; do eval "kustomize build $(dirname ${filename})|datree test -"; done

This will find all my kustomization.yaml files and pass it to datree... quite easy

I am sure, I dont have to explain Kustomize :-) I am sure you are more experienced than me :-)

But what would be nice is, that

• when a kustomization.yaml file is found or declared
• it looks to it's parent dependend kustomization.yams files, renders it and runs datree against
• all other yaml files within the same folders where kustomization.yaml-files are located are not explicitly tested alone.

base
  kustomization.yaml <-- should NOT be scanned, because it's mentioned in /overlays/staging/kustomization.yaml
  deployment.yaml <--  should NOT be scanned by datree, because it's declared in ./kustomization.yaml
overlays
  staging
    kustomization.yaml <-- SHOULD be rendered and passed to datree, because not mentioned in any other kustomization.yaml
    deployment.yaml <-- should NOT be scanned by datree, because it's declared in ./kustomization.yaml

[eyarz]

I wonder if the user should pass Datree's CLI a path to the chart/dir that contains the kustomization.yaml (like we do with Helm) or the path to the kustomization.yaml itself (like @exocode maintained).

Also, Kustomize has different versions, so we need to research if they behave differently.

[exocode]

@eyarz as an idea to handle the folder structure, Checkov has published it's kustomize support today. They have made a docs page, which explains how they treat that:
https://github.com/bridgecrewio/checkov/blob/master/docs/7.Scan%20Examples/Kustomize.md

For the versioning I have too low experience. I am too new to Kustomize to answer that question

[eyarz]

This is helpful, thank :)
I think we have enough info to start work on it - moving to Selected for development.
@adifay, please share a draft for this capability so someone can grab it.

[adifayer]

Goal 🥅

A new test command that supports kustomize mechanism- first, it builds the configuration files, then the policy check validates the build output (rendered yaml files)

User Flows 🏊‍♀️

User runs kustomize test command on a directory with kustomization.yaml file
Kustomize build command takes place
The policy check validates the kustomize build output
User executes kustomize test command on a directory w/o kustomization.y(a)ml file → Indicative error message is displayed
error: kustomization.y(a)ml file not found in the directory [path]
User executes datree test on kustomization.y(a)ml / Kustomization file w/o kustomize test command-
The file is recognized as a kustomize file
Indicative error message is displayed with a link to kustomize docs (will be provided by product team)

Thumb rule ☝🏻

Whenever there is unanswered product flow/question, look at the kustomize tool behavior, and use it (kustomize build)

[eyarz]

in production:
https://hub.datree.io/kustomize-support