[K8s testing] Support use of Kustomize

[constanca-m]

What does this issue try to achieve?

Updating kube state metrics version on testing at most one time.

What is the problem?

Right now, the current kube state metrics version being used for Kubernetes integration testing is 2.3.0.
However, this version is no longer aligned with the Kubernetes version being supported, which range between
1.23.x-1.26.x. This implies that, at least, there should be support for 2.4.2 version - and this version should keep being incremented at the same time Kubernetes versions are released and supported by the integration.

Example of a problem the current being used version is causing (also documented here):

DEBUG found 0 hits in metrics-kubernetes.state_cronjob-ep data stream: index_not_found_exception: no such index [metrics-kubernetes.state_cronjob-ep] Status=404

This causes our tests to fail and makes it impossible to merge PRs on our Kubernetes integration.

The problem is that changing this version, doesn't apply changing only one tag. We need to:

• Update tag on five different files (the ones related to kube state metrics), and, on top of that:
• We need to update other things, such as permissions.

This can be seen on this PR.

This is, of course, not ideal and this process should not require such high maintenance. Since, Kustomize is already being used (see from this PR), the same approach should be followed.

Steps this would require

We would need to:

1. In elastic package repository Change flag -f on kubectl to -k in here to allow (and enforce) the use of Kustomize;

• PR: [K8s testing] Support use of Kustomize #1125

2. In integrations repository Update files in a way that a new kustomization.yaml file is deployed, that will simultaneously apply kube-state-metrics from the original repository. Same approach as shown in this merged PR;

• PR: [Kubernetes] Update system testing manifests integrations#5227

3. Document this (maybe in this file).

Making this change will let us update kube-state-version by changing just one word: the reference to the branch we are targeting. And if we always want to use the latest version we won't even have to do that.

Which packages does this affect?

As of the moment this issue is created, the only package that will be affected by this change will be Kubernetes.
The reason for this, is that only integrations with this kind of folder _dev/deploy/k8s are considered on Kubernetes testing. There are three integrations with one: Kubernetes, Cloud Defend and Cloud Security Posture. However, as documented here, since the last two have the .empty file, only the Kubernetes integration makes use of this test.

[gsantoro]

I like this approach, I could benefit of a kustomization.yaml file to enforce an order of execution for 2 manifests used for system tests for nginx-ingress-controller as part of this issue

[gsantoro]

Maybe since we are there we should look into how they are using:

• https://github.com/elastic/elastic-package/blob/main/internal/kubectl/kube-state-metrics-single.yaml
• https://github.com/elastic/elastic-package/blob/main/internal/kubectl/kube-state-metrics-multiple.yaml

[constanca-m]

Maybe since we are there we should look into how they are using:

* https://github.com/elastic/elastic-package/blob/main/internal/kubectl/kube-state-metrics-single.yaml

* https://github.com/elastic/elastic-package/blob/main/internal/kubectl/kube-state-metrics-multiple.yaml

I don't really understand where those files are being used, but since they relate to the same thing, I agree we should look into that as well @gsantoro

[gsantoro]

I couldn't find them referenced anywhere but we should ask around.

[tetianakravchenko]

1. I don't think we should enforce use of Kustomize, if usage of manifests stored in _dev/deploy/k8s covers all needs and there is no reason to use kustomization.yml - this should be supported

2. In elastic package repository Change flag -f on kubectl to -k in here to allow (and enforce) the use of Kustomize;

if you are referring to this line - -k doesn't seem to support working with stdin
now it applies a kubernetes manifest provided as stdin, similar to running: cat manifest.yaml | kubectl apply -f - -o yaml

[gsantoro]

1. I agree that we should support kustomize as optional. I was thinking that if we were to add some extra logic that only support kustomize if there is a kustomization.yml in that directory that would be better.
2. the alternative there could be something along the line

if test -f "./kustomization.yml"; then
  kustomize build | kubectl apply -f - -o yaml
else
  cat manifest.yaml | kubectl apply -f - -o yaml
fi

[constanca-m]

I was thinking in forcing a kustomization file only for the custom directory. Since we use elastic agent files similarly as a user would use through Kibana, I don't think it is a good idea to change that without a discussion @gsantoro @tetianakravchenko . And since we only use the custom directory for the state_* datastreams, we know that we need to deploy kube state metrics and there is no need to maintain those files.

[gsantoro]

1. yes, we should discuss it with the elastic-package team either way
2. when you say we only use the custom directoy for the state_* datastream do you mean this directory? packages/kubernetes/data_stream/state_cronjob/_dev/test/system. since we also have system tests at packages/kubernetes/data_stream/proxy/_dev/test/system.
3. Also FYI, at packages/kubernetes/_dev/deploy/k8s there is more than just the kube-state-metrics service.
4. Also FYI, I could use the kustomization.yaml in the system tests to fix this issue Craft a system test for nginx_ingress_controller integration that works with k8s integrations#4874 where I need to fix the order of two manifests for that to work.

[constanca-m]

1. yes, we should discuss it with the elastic-package team either way

   2. when you say we only use the custom directoy for the state_* datastream do you mean this directory? packages/kubernetes/data_stream/state_cronjob/_dev/test/system. since we also have system tests at packages/kubernetes/data_stream/proxy/_dev/test/system.

   3. Also FYI, at packages/kubernetes/_dev/deploy/k8s there is more than just the kube-state-metrics service.

   4. Also FYI, I could use the kustomization.yaml in the system tests to fix this issue Craft a system test for nginx_ingress_controller integration that works with k8s integrations#4874 where I need to fix the order of two manifests for that to work.

2. No, sorry, I should have made it explicit. I mean this one packages/kubernetes/_dev/deploy/k8s
3. Yes, I noticed, I was thinking in putting them as resources on the kustomization.yaml
4. Ok, as soon as I can make it work locally, I will look into that.

Edit: making it clear, the manifests file deploying anything related to elastic agent will be left as they are now, and they will keep being deployed with the -f flag. The only ones that will use kustomize, will be the custom resources for our package (packages/kubernetes/_dev/deploy/k8s) that are used by all state_* datastreams.

@gsantoro

[gsantoro]

I'm not 100% sure which resources for elastic-agent you are talking about. either way, let's have a look once you make the change and have all the system tests working.

[tetianakravchenko]

@constanca-m I also don't fully follow you here, regarding this part:

the manifests file deploying anything related to elastic agent will be left as they are now, and they will keep being deployed with the -f flag. The only ones that will use kustomize, will be the custom resources for our package (packages/kubernetes/_dev/deploy/k8s) that are used by all state_* datastreams.

but as mentioned above - let's have a look once you make the change