Operator monitoring ports mismatch

[sblargaba]

Hello,
I'm running into some issues with the monitoring port of the operator:

I'm using the kiali operator helm chart to deploy, which defines a Deployment object exposing a port on

ports:
  - name: http-metrics
    containerPort: 8080

In the same namespace I see a Service and ServiceMonitor with the following configuration

• Service:

  - name: http-metrics
    port: 8383
    protocol: TCP
    targetPort: 8383
  - name: cr-metrics
    port: 8686
    protocol: TCP
    targetPort: 8686
  

• ServiceMonitor:

  endpoints:
  - bearerTokenSecret:
      key: ""
    port: http-metrics
  - bearerTokenSecret:
      key: ""
    port: cr-metrics
  

It is my understanding (from #2805) that these two are created by the operator itself

Here are my questions:

1. Why aren't the service and servicemonitor part of the helm chart? Why is the operator deploying it's own resources?
   When looking at the prometheus operator, it doesn't deploy by itself any resource related to the operator, but only ones for the prometheus instances
2. Why is there no metadata used to identify the operator as the owner of those resources?
   Even now I'm not 100% sure the source of those Service and Servicemonitor is the operator, despite spending quite some time trying to dig trough the deployed resources first, and the operator source code afterwards.
   It would be great if there was an easier way to identify the operator as the owner of the deployed resources
3. How can I fix/disable the servicemonitor and stop having target down alerts on my monitoring system?

Thank you for your help!

[jmazzitelli]

You didn't say what version of operator you have or what the full installation commands you used to install it or if you have a customized kiali operator of your own or if you are using a published helm chart from kiali.org. So I can't tell you much. But what I can tell you is that it sounds like you have an old operator from some non-latest helm chart. You say:

In the same namespace I see a Service and ServiceMonitor with the following configuration

Service:
- name: http-metrics
  port: 8383
  protocol: TCP
  targetPort: 8383
- name: cr-metrics
  port: 8686
  protocol: TCP
  targetPort: 8686

But those ports are old - http-metrics has changed and the cr-metrics one was completely removed. See commit: kiali/helm-charts@c17ed65#diff-1346ff7b057e784bdfdb84a46c4cc0f45f1f478c7fe0b9b9d36a9f872ededc59L76-R76

Now I just installed the latest published kiali-operator using the helm-chart via:

kubectl create ns kiali-operator
helm install --namespace kiali-operator --repo https://kiali.org/helm-charts kiali-operator kiali-operator

And here is what I see:

$ kubectl get -n kiali-operator services
No resources found in kiali-operator namespace.

$ kubectl get -n kiali-operator deployments -l app=kiali-operator -o jsonpath='{.items[0].spec.template.spec.containers[0].ports}' | jq
[
  {
    "containerPort": 8080,
    "name": "http-metrics",
    "protocol": "TCP"
  }
]

$ kubectl get -n kiali-operator pods -l app=kiali-operator -o jsonpath='{.items[0].spec.containers[0].ports}' | jq
[
  {
    "containerPort": 8080,
    "name": "http-metrics",
    "protocol": "TCP"
  }
]

IIRC, ServiceMonitor objects are created by the operator SDK itself (which Kiali is based on). You can see that in the issue you linked to where you can see the operator SDK logs that message (note: that is an SDK log message - Kiali itself isn't doing that - its under the covers):

{"level":"info","ts":1589913367.0101564,"logger":"cmd","msg":"Could not create ServiceMonitor object","Namespace":"kiali-operator","error":"servicemonitors.monitoring.coreos.com \"kiali-operator-metrics\" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>"}

But I don't see that servicemonitor (or service) resources in my installation - I wonder if these were created in the older version of the operator SDK that was used (Kiali operator recently moved up to the 1.x version of the SDK - perhaps this behavior changed from the previous 0.17)?

Not knowing what you have installed/running or how you installed it, that's as much as I can say.

[sblargaba]

Sorry for the lack of information: I'm deploying with the kiali-operator chart version 1.34.1 from https://kiali.org/helm-charts
The issue seems most likely caused by a lack of cleanup when we automatically updated to a new chart (and operator) version.

Simply deleting the resources solved the problem.
Thank you for your patience!