feat!: Add dedicated -metrics service

[sbernauer]

Description

Part of stackabletech/issues#692

The metrics Service was missed, as we skipped the listener rollout to OPA.

Also add a test that actually scrapes Prometheus metrics.

See changelog for detailed changes.

k get svc -A | grep -P 'simple-zk|test-opa'                                                                                kind-kind
default                  simple-zk-server                    ClusterIP   10.96.56.205   <none>        2282/TCP                 82m
default                  simple-zk-server-default-headless   ClusterIP   None           <none>        2888/TCP,3888/TCP        82m
default                  simple-zk-server-default-metrics    ClusterIP   None           <none>        9505/TCP,7000/TCP        82m
kuttl-test-causal-bass   test-opa-server                     ClusterIP   10.96.164.60   <none>        8081/TCP                 52s
kuttl-test-causal-bass   test-opa-server-default-headless    ClusterIP   None           <none>        8081/TCP                 52s
kuttl-test-causal-bass   test-opa-server-default-metrics     ClusterIP   None           <none>        8081/TCP                 52s

Definition of Done Checklist

• Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
• Please make sure all these things are done and tick the boxes

Author

• Changes are OpenShift compatible
• CRD changes approved
• CRD documentation for all fields, following the style guide.
• Helm chart can be installed and deployed operator works
• Integration tests passed (for non trivial changes)
• Changes need to be "offline" compatible

Reviewer

• Code contains useful comments
• Code contains useful logging statements
• (Integration-)Test cases added
• Documentation added or updated. Follows the style guide.
• Changelog updated
• Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

• Feature Tracker has been updated
• Proper release label has been added
• Add type/deprecation label & add to the deprecation schedule
• Add type/experimental label & add to the experimental features tracker

[maltesander]

LGTM!

[sbernauer]

Not merging now, waiting for the SDP 25.7 release team to update the changelog in main

[sbernauer]

Started https://testing.stackable.tech/view/02%20Operator%20Tests%20(custom)/job/opa-operator-it-custom/26/

[maltesander]

LGTM!

[maltesander]

LGTM!

[sbernauer]

All tests passed locally:

--- PASS: kuttl (190.84s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/resources_opa-latest-1.4.2_openshift-false (20.14s)
        --- PASS: kuttl/harness/aas-user-info_opa-latest-1.4.2_openshift-false (76.00s)
        --- PASS: kuttl/harness/logging_opa-1.4.2_openshift-false (65.10s)
        --- PASS: kuttl/harness/smoke_opa-1.0.1_openshift-false (99.10s)
        --- PASS: kuttl/harness/logging_opa-1.0.1_openshift-false (100.43s)
        --- PASS: kuttl/harness/cluster-operation_opa-latest-1.4.2_openshift-false (36.03s)
        --- PASS: kuttl/harness/smoke_opa-1.4.2_openshift-false (59.13s)
        --- PASS: kuttl/harness/keycloak-user-info_opa-latest-1.4.2_keycloak-23.0.1_openshift-false (114.83s)
PASS

[maltesander]

LGTM!

[sbernauer]

Release notes

Added

• Add a dedicated per-rolegroup -metrics Service, which can be used to get Prometheus metrics
• Expose more Prometheus metrics, such as successful or failed bundle loads and information about the OPA environment

Changed

• BREAKING: The per-rolegroup services now only serves the HTTP port and has a -headless suffix to better indicate their purpose and to be consistent with other operators ([feat!: Add dedicated -metrics service #748]).