Add collectors to Scheduling Gate Integration Tests

[RealAnna]

Given this generic example on how to add a collector to kuttl tests #1834, we should add collectors in the scheduling gate integration test that:

• collect logs of lifecycle-operator
• run a kubectl describe for Workload Version, App Version, job

DoD

• manually verify that when the test fails, the logs and describes are visible in the shell

[sudiptob2]

interested

[sudiptob2]

Hi @RealAnna
I am trying to run the existing scheduling gates integration test but getting an error. Are there any additional steps that I need to take to run tests locally? I followed the following steps. Attached logs as well. scheduling-gate-text-logs.txt

1. development code is deployed in a local kind cluster using make build-deploy-dev-environment RELEASE_REGISTRY=docker.io/sudiptob2 TAG=main
2. run make integration-test-scheduling-gates-local

[RealAnna]

Hi @sudiptob2 scheduling gates are disabled by default. To test it you need to use kind with k8s image 1.27 and you can install the latest keptn using helm https://lifecycle.keptn.sh/docs/install/install/#basic-installation setting the helm value of scheduling gate to true

[sudiptob2]

I tried the scheduling gate as per your suggestion but unfortunately, I'm still hitting the same error when running ./test/scheduling-gates on my local cluster. My Kubernetes version is 1.27.x.
These tests should pass right?

scheduling-gate-text-logs.txt

As I investigated further, the following portion of the test is causing failure. Ref here. I tried removing this assert and the test works fine. Any thoughts on what might be causing this?

---
apiVersion: v1
kind: Pod
metadata:
  labels:
    app: test
  annotations:
    keptn.sh/scheduling-gate-removed: "true"
status:
  phase: Running
spec:
  schedulerName: default-scheduler

Mentioning @odubajDT as well, I was wondering if you could also offer some guidance since you added this test.

[RealAnna]

Hi @sudiptob2 as I see you installed only the scheduler, however the scheduling gate tests lifecycle controller. Please install the entire keptn using the installation instructions and then run the test again.

[sudiptob2]

Hi @RealAnna
Actually scheduling-gates is installed in the keptn-lifecycle-toolkit-system namespace, whereas other operators are installed in keptn-system namespace. That's why in the above screenshot only one pod is shown.
Should it be installed in the same namespace?

[RealAnna]

Hi @RealAnna Actually scheduling-gates is installed in the keptn-lifecycle-toolkit-system namespace, whereas other operators are installed in keptn-system namespace. That's why in the above screenshot only one pod is shown. Should it be installed in the same namespace?

I am not sure what you mean by scheduling gates installatiion, however in general all keptn component need to be installed in the same namespace. I would suggest to delete the kind cluster and start from scratch.

[sudiptob2]

Hi @RealAnna, Basically certificate-operator, lifecycle-operator and metrics-operator are installed in keptn-system namespace when I run the following make command.

make build-deploy-dev-environment RELEASE_REGISTRY=docker.io/sudiptob2 TAG=main 

However for scheduler it looks for keptn-lifecycle-toolkit-system namespace and fails to install. If I create the keptn-lifecycle-toolkit-system namespace manually and then run the make command then scheduler is installed in the keptn-lifecycle-toolkit-system namespace. I think this is because keptn-lifecycle-toolkit-system is mentioned in this deployment file.

I tried in a fresh kind cluster and got the above result. Could you please see if I missing any steps here? Here is a screenshot of the output.

[RealAnna]

@sudiptob2 this test will not work by installing keptn using the make file since by default the scheduling gates are disabled. You need to use an official helm chart and to pass it a value such as:

lifecycleOperator: 
   schedulingGatesEnabled: true

something like helm upgrade --install keptn kepn/keptn --set lifecycleOperator.schedulingGatesEnabled=true