Helm Operator doesn't work with Kube Prometheus Stack with additionalPrometheusRulesMap yaml list

[perk-sumo]

Bug Report

Hi!
Helm Operator doesn't seem to work with Kube Prometheus Stack when there is a list used inside the additionalPrometheusRulesMap configuration.
The same config works just fine when used with helm v3.2.4 without the operator.

Without the additionalPrometheusRulesMap key the helm operator works as expected.

What did you do?

1. Created the Helm Operator for the Kube Prometheus Stack

$ operator-sdk_linux_amd64 init --plugins helm --helm-chart prometheus-community/kube-prometheus-stack --domain example.com --group helm-chart --version v1 --kind Promhelm

2. Prepared the following CR

apiVersion: helm-chart.example.com/v1
kind: Promhelm
metadata:
  name: promhelm-sample
spec:
  additionalPrometheusRulesMap:
    some-rules:
      groups:
        - name: foo
          rules:
            - expr: bar
              record: baz
  alertmanager:
    enabled: false
  grafana:
    enabled: false
    defaultDashboardsEnabled: false
  prometheusOperator:
    admissionWebhooks:
      enabled: false
    tls:
      enabled: false

3. Deployed the CR

kubectl apply -f /prometheus-helm-operator/config/samples/promhelm.yaml -n promhelm --validate=false

What did you expect to see?

I should see the Helm chart deployed.
I should see applied k8s objects in the operator logs.
I should be able to edit / delete the CR.

What did you see instead? Under which circumstances?

The Helm chart has been deployed and Prometheus seems to be working fine. I can uninstall it without the operator using the external helm command.

In the operator logs I don't see the applied k8s objects.

Also in the operator logs I can see the following error line:

E0311 12:05:49.494839       1 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.4/tools/cache/reflector.go:156: Failed to watch /v1, Kind=List: failed to list /v1, Kind=List: the server could not find the requested resource

I cannot change nor delete the previously applied CR.

Environment

Operator type:

/language helm

Kubernetes cluster type:

microk8s

$ operator-sdk version

operator-sdk version: "v1.3.2", commit: "5dd883dc90085da6a6e75bb05443b4d491a39143", kubernetes version: "1.19.4", go version: "go1.15.5", GOOS: "linux", GOARCH: "amd64"

$ kubectl version

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.15", GitCommit:"73dd5c840662bb066a146d0871216333181f4b64", GitTreeState:"clean", BuildDate:"2021-01-13T13:22:41Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.15", GitCommit:"73dd5c840662bb066a146d0871216333181f4b64", GitTreeState:"clean", BuildDate:"2021-01-13T13:14:05Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

$ helm search repo prometheus-community/kube-prometheus-stack

NAME                                      	CHART VERSION	APP VERSION	DESCRIPTION
prometheus-community/kube-prometheus-stack	14.0.1       	0.46.0     	kube-prometheus-stack collects Kubernetes manif...

Possible Solution

Additional context

[hickeyma]

@perk-sumo Thanks for raising the issue.

The prometheus-community/kube-prometheus-stack chart uses a Kind: List in additionalPrometheusRules.

It would seem that this kind is not used very often in Kubernetes and therefore in Helm charts as well. From this explanation, it says: "List is not actually a "resource". Each resource can have an associated list type, PodList or CronJobList, but those are not actually resources. And each of those lists is represented in yaml by kind: List." There is also an open issue in Kubernetes to document it.

What this means is that the Helm operator would need to be updated to handle Lists. I will look to push a PR when I get a chance.

Thanks to the great feedback on this issue from @joelanford on slack.

[estroz]

Interesting, TIL! Sounds like ansible-operator may have to handle this case as well.

/kind feature

[hickeyma]

@perk-sumo As a short term workaround, if you didn't set additionalPrometheusRulesMap in the values file, it should work as expected. This is because it won't need to use additionalPrometheusRules then as if statement will be false.

[hickeyma]

@estroz Do you think that this should be re-categorized to a bug?

[estroz]

Yes.

/remove-kind feature
/kind bug

[hickeyma]

Thanks @estroz