OCPBUGS-29247: ibm-vpc-block-csi-driver is missing sidecar metrics

[RomanBednar]

cc @openshift/storage

Verification

Before patch

No sidecar metric endpoints are present:

$ oc -n openshift-monitoring exec -ti -c prometheus prometheus-k8s-0 sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.

sh-4.4$ export token=$(cat /run/secrets/kubernetes.io/serviceaccount/token)

sh-4.4$ curl -k -H "Authorization: Bearer $token" 'https://ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc:9202/metrics'
curl: (6) Could not resolve host: ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc
sh-4.4$ curl -k -H "Authorization: Bearer $token" 'https://ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc:9203/metrics'
curl: (6) Could not resolve host: ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc
sh-4.4$ curl -k -H "Authorization: Bearer $token" 'https://ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc:9204/metrics'
curl: (6) Could not resolve host: ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc
sh-4.4$ curl -k -H "Authorization: Bearer $token" 'https://ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc:9205/metrics'
curl: (6) Could not resolve host: ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc
sh-4.4$ curl -k -H "Authorization: Bearer $token" 'https://ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc:9206/metrics'
curl: (6) Could not resolve host: ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc

After patch

Metric endpoints are set and metric data available:

$ oc -n openshift-monitoring exec -ti -c prometheus prometheus-k8s-0 sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.

sh-4.4$ export token=$(cat /run/secrets/kubernetes.io/serviceaccount/token)

sh-4.4$ for port in {9202..9206}; do curl -k -H "Authorization: Bearer $token" "https://ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc:{$port}/metrics" | tail -n 1;done
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 34614    0 34614    0     0  1469k      0 --:--:-- --:--:-- --:--:-- 1469k
workqueue_work_duration_seconds_count{name="volumes"} 0
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 10045    0 10045    0     0   613k      0 --:--:-- --:--:-- --:--:--  613k
process_start_time_seconds 1.70904193396e+09
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  9863    0  9863    0     0   601k      0 --:--:-- --:--:-- --:--:--  601k
process_start_time_seconds 1.70904193341e+09
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  7599    0  7599    0     0   494k      0 --:--:-- --:--:-- --:--:--  494k
process_start_time_seconds 1.70904193438e+09
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  6239    0  6239    0     0   358k      0 --:--:-- --:--:-- --:--:--  358k

Check ServiceMonitor presence:

$ oc -n openshift-cluster-csi-drivers get servicemonitor/ibm-vpc-block-csi-driver-controller-monitor -o yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  creationTimestamp: "2024-02-27T13:46:26Z"
  generation: 1
  name: ibm-vpc-block-csi-driver-controller-monitor
  namespace: openshift-cluster-csi-drivers
  resourceVersion: "41925"
  uid: 121a5cb7-01c6-4d87-bead-3c7f814c82d7
spec:
  endpoints:
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    interval: 30s
    path: /metrics
    port: provisioner-m
    scheme: https
    tlsConfig:
      caFile: /etc/prometheus/configmaps/serving-certs-ca-bundle/service-ca.crt
      serverName: ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    interval: 30s
    path: /metrics
    port: attacher-m
    scheme: https
    tlsConfig:
      caFile: /etc/prometheus/configmaps/serving-certs-ca-bundle/service-ca.crt
      serverName: ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    interval: 30s
    path: /metrics
    port: resizer-m
    scheme: https
    tlsConfig:
      caFile: /etc/prometheus/configmaps/serving-certs-ca-bundle/service-ca.crt
      serverName: ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    interval: 30s
    path: /metrics
    port: snapshotter-m
    scheme: https
    tlsConfig:
      caFile: /etc/prometheus/configmaps/serving-certs-ca-bundle/service-ca.crt
      serverName: ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    interval: 30s
    path: /metrics
    port: driver-m
    scheme: https
    tlsConfig:
      caFile: /etc/prometheus/configmaps/serving-certs-ca-bundle/service-ca.crt
      serverName: ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc
  jobLabel: component
  selector:
    matchLabels:
      app: ibm-vpc-block-csi-driver-controller-metrics

[openshift-ci-robot]

@RomanBednar: This pull request references Jira Issue OCPBUGS-29247, which is invalid:

• expected the bug to target the "4.16.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

Verification

Before patch

No sidecar metric endpoints are present:

$ oc -n openshift-monitoring exec -ti -c prometheus prometheus-k8s-0 sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.

sh-4.4$ export token=$(cat /run/secrets/kubernetes.io/serviceaccount/token)

sh-4.4$ curl -k -H "Authorization: Bearer $token" 'https://ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc:9202/metrics'
curl: (6) Could not resolve host: ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc
sh-4.4$ curl -k -H "Authorization: Bearer $token" 'https://ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc:9203/metrics'
curl: (6) Could not resolve host: ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc
sh-4.4$ curl -k -H "Authorization: Bearer $token" 'https://ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc:9204/metrics'
curl: (6) Could not resolve host: ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc
sh-4.4$ curl -k -H "Authorization: Bearer $token" 'https://ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc:9205/metrics'
curl: (6) Could not resolve host: ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc
sh-4.4$ curl -k -H "Authorization: Bearer $token" 'https://ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc:9206/metrics'
curl: (6) Could not resolve host: ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc

---

After patch

Metric endpoints are set and metric data available:

$ oc -n openshift-monitoring exec -ti -c prometheus prometheus-k8s-0 sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.

sh-4.4$ export token=$(cat /run/secrets/kubernetes.io/serviceaccount/token)

sh-4.4$ for port in {9202..9206}; do curl -k -H "Authorization: Bearer $token" "https://ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc:{$port}/metrics" | tail -n 1;done
 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                Dload  Upload   Total   Spent    Left  Speed
100 34614    0 34614    0     0  1469k      0 --:--:-- --:--:-- --:--:-- 1469k
workqueue_work_duration_seconds_count{name="volumes"} 0
 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                Dload  Upload   Total   Spent    Left  Speed
100 10045    0 10045    0     0   613k      0 --:--:-- --:--:-- --:--:--  613k
process_start_time_seconds 1.70904193396e+09
 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                Dload  Upload   Total   Spent    Left  Speed
100  9863    0  9863    0     0   601k      0 --:--:-- --:--:-- --:--:--  601k
process_start_time_seconds 1.70904193341e+09
 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                Dload  Upload   Total   Spent    Left  Speed
100  7599    0  7599    0     0   494k      0 --:--:-- --:--:-- --:--:--  494k
process_start_time_seconds 1.70904193438e+09
 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                Dload  Upload   Total   Spent    Left  Speed
100  6239    0  6239    0     0   358k      0 --:--:-- --:--:-- --:--:--  358k

Check ServiceMonitor presence:

$ oc -n openshift-cluster-csi-drivers get servicemonitor/ibm-vpc-block-csi-driver-controller-monitor -o yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
 creationTimestamp: "2024-02-27T13:46:26Z"
 generation: 1
 name: ibm-vpc-block-csi-driver-controller-monitor
 namespace: openshift-cluster-csi-drivers
 resourceVersion: "41925"
 uid: 121a5cb7-01c6-4d87-bead-3c7f814c82d7
spec:
 endpoints:
 - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
   interval: 30s
   path: /metrics
   port: provisioner-m
   scheme: https
   tlsConfig:
     caFile: /etc/prometheus/configmaps/serving-certs-ca-bundle/service-ca.crt
     serverName: ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc
 - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
   interval: 30s
   path: /metrics
   port: attacher-m
   scheme: https
   tlsConfig:
     caFile: /etc/prometheus/configmaps/serving-certs-ca-bundle/service-ca.crt
     serverName: ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc
 - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
   interval: 30s
   path: /metrics
   port: resizer-m
   scheme: https
   tlsConfig:
     caFile: /etc/prometheus/configmaps/serving-certs-ca-bundle/service-ca.crt
     serverName: ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc
 - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
   interval: 30s
   path: /metrics
   port: snapshotter-m
   scheme: https
   tlsConfig:
     caFile: /etc/prometheus/configmaps/serving-certs-ca-bundle/service-ca.crt
     serverName: ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc
 - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
   interval: 30s
   path: /metrics
   port: driver-m
   scheme: https
   tlsConfig:
     caFile: /etc/prometheus/configmaps/serving-certs-ca-bundle/service-ca.crt
     serverName: ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc
 jobLabel: component
 selector:
   matchLabels:
     app: ibm-vpc-block-csi-driver-controller-metrics

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: RomanBednar

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [RomanBednar]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci]

@RomanBednar: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-ibmcloud-csi	7294dd0	link	false	/test e2e-ibmcloud-csi
ci/prow/e2e-ibmcloud-csi-extended	7294dd0	link	false	/test e2e-ibmcloud-csi-extended

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[RomanBednar]

/jira refresh

[openshift-ci-robot]

@RomanBednar: This pull request references Jira Issue OCPBUGS-29247, which is invalid:

• expected the bug to target the "4.16.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[RomanBednar]

/jira refresh

[openshift-ci-robot]

@RomanBednar: This pull request references Jira Issue OCPBUGS-29247, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.16.0) matches configured target version for branch (4.16.0)
• bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

No GitHub users were found matching the public email listed for the QA contact in Jira (wduan@redhat.com), skipping review request.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[jsafrane]

/lgtm

[openshift-ci-robot]

@RomanBednar: Jira Issue OCPBUGS-29247: All pull requests linked via external trackers have merged:

• openshift/ibm-vpc-block-csi-driver-operator#112

Jira Issue OCPBUGS-29247 has been moved to the MODIFIED state.

In response to this:

cc @openshift/storage

Verification

Before patch

No sidecar metric endpoints are present:

$ oc -n openshift-monitoring exec -ti -c prometheus prometheus-k8s-0 sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.

sh-4.4$ export token=$(cat /run/secrets/kubernetes.io/serviceaccount/token)

sh-4.4$ curl -k -H "Authorization: Bearer $token" 'https://ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc:9202/metrics'
curl: (6) Could not resolve host: ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc
sh-4.4$ curl -k -H "Authorization: Bearer $token" 'https://ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc:9203/metrics'
curl: (6) Could not resolve host: ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc
sh-4.4$ curl -k -H "Authorization: Bearer $token" 'https://ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc:9204/metrics'
curl: (6) Could not resolve host: ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc
sh-4.4$ curl -k -H "Authorization: Bearer $token" 'https://ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc:9205/metrics'
curl: (6) Could not resolve host: ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc
sh-4.4$ curl -k -H "Authorization: Bearer $token" 'https://ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc:9206/metrics'
curl: (6) Could not resolve host: ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc

After patch

Metric endpoints are set and metric data available:

$ oc -n openshift-monitoring exec -ti -c prometheus prometheus-k8s-0 sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.

sh-4.4$ export token=$(cat /run/secrets/kubernetes.io/serviceaccount/token)

sh-4.4$ for port in {9202..9206}; do curl -k -H "Authorization: Bearer $token" "https://ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc:{$port}/metrics" | tail -n 1;done
 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                Dload  Upload   Total   Spent    Left  Speed
100 34614    0 34614    0     0  1469k      0 --:--:-- --:--:-- --:--:-- 1469k
workqueue_work_duration_seconds_count{name="volumes"} 0
 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                Dload  Upload   Total   Spent    Left  Speed
100 10045    0 10045    0     0   613k      0 --:--:-- --:--:-- --:--:--  613k
process_start_time_seconds 1.70904193396e+09
 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                Dload  Upload   Total   Spent    Left  Speed
100  9863    0  9863    0     0   601k      0 --:--:-- --:--:-- --:--:--  601k
process_start_time_seconds 1.70904193341e+09
 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                Dload  Upload   Total   Spent    Left  Speed
100  7599    0  7599    0     0   494k      0 --:--:-- --:--:-- --:--:--  494k
process_start_time_seconds 1.70904193438e+09
 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                Dload  Upload   Total   Spent    Left  Speed
100  6239    0  6239    0     0   358k      0 --:--:-- --:--:-- --:--:--  358k

Check ServiceMonitor presence:

$ oc -n openshift-cluster-csi-drivers get servicemonitor/ibm-vpc-block-csi-driver-controller-monitor -o yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
 creationTimestamp: "2024-02-27T13:46:26Z"
 generation: 1
 name: ibm-vpc-block-csi-driver-controller-monitor
 namespace: openshift-cluster-csi-drivers
 resourceVersion: "41925"
 uid: 121a5cb7-01c6-4d87-bead-3c7f814c82d7
spec:
 endpoints:
 - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
   interval: 30s
   path: /metrics
   port: provisioner-m
   scheme: https
   tlsConfig:
     caFile: /etc/prometheus/configmaps/serving-certs-ca-bundle/service-ca.crt
     serverName: ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc
 - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
   interval: 30s
   path: /metrics
   port: attacher-m
   scheme: https
   tlsConfig:
     caFile: /etc/prometheus/configmaps/serving-certs-ca-bundle/service-ca.crt
     serverName: ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc
 - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
   interval: 30s
   path: /metrics
   port: resizer-m
   scheme: https
   tlsConfig:
     caFile: /etc/prometheus/configmaps/serving-certs-ca-bundle/service-ca.crt
     serverName: ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc
 - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
   interval: 30s
   path: /metrics
   port: snapshotter-m
   scheme: https
   tlsConfig:
     caFile: /etc/prometheus/configmaps/serving-certs-ca-bundle/service-ca.crt
     serverName: ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc
 - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
   interval: 30s
   path: /metrics
   port: driver-m
   scheme: https
   tlsConfig:
     caFile: /etc/prometheus/configmaps/serving-certs-ca-bundle/service-ca.crt
     serverName: ibm-vpc-block-csi-driver-controller-metrics.openshift-cluster-csi-drivers.svc
 jobLabel: component
 selector:
   matchLabels:
     app: ibm-vpc-block-csi-driver-controller-metrics

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-bot]

[ART PR BUILD NOTIFIER]

This PR has been included in build ose-ibm-vpc-block-csi-driver-operator-container-v4.16.0-202403222013.p0.g9cc8f14.assembly.stream.el9 for distgit ose-ibm-vpc-block-csi-driver-operator.
All builds following this will include this PR.

[openshift-merge-robot]

Fix included in accepted release 4.16.0-0.nightly-2024-03-23-062612