fix(controller): add missing workqueue metrics (#16315)

[agaudreault]

Description

Closes #12241 #16315

I believe this should be cherry-picked in 2.9 and 2.10 since it seems to be a regression.

My guess is that another package was calling workqueue.SetProvider() after the controller/metrics package, overriding the custom config. To not depend on the loading order, I think it is fair to use the default workqueue metrics.

Impacts

• Uses default bucket for workqueue metrics
• Adds new metrics emitted by the controller-runtime

Checklist:

• Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this does not need to be in the release notes.
• The title of the PR states what changed and the related issues number (used for the release note).
• The title of the PR conforms to the Toolchain Guide
• I've included "Closes [ISSUE #]" or "Fixes [ISSUE #]" in the description to automatically close the associated issue.
• I've updated both the CLI and UI to expose my feature, or I plan to submit a second PR with them.
• Does this PR require documentation updates?
• I've updated documentation as required by this PR.
• I have signed off all my commits as required by DCO
• I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
• My build is green (troubleshooting builds).
• My new feature complies with the feature status guidelines.
• I have added a brief description of why this PR is necessary and/or what this PR solves.
• Optional. My organization is added to USERS.md.
• Optional. For bug fixes, I've indicated what older releases this fix should be cherry-picked into (this may or may not happen depending on risk/complexity).

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (52ffd7d) 49.25% compared to head (3464c96) 49.27%.
Report is 12 commits behind head on master.

❗ Current head 3464c96 differs from pull request most recent head 378bea5. Consider uploading reports for the commit 378bea5 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #17013      +/-   ##
==========================================
+ Coverage   49.25%   49.27%   +0.02%     
==========================================
  Files         274      273       -1     
  Lines       48170    48122      -48     
==========================================
- Hits        23724    23714      -10     
+ Misses      22099    22064      -35     
+ Partials     2347     2344       -3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[rumstead]

Does this PR address the metrics on the applicationset controller as well or just the application controller?

[agaudreault]

Does this PR address the metrics on the applicationset controller as well or just the application controller?

@rumstead the applicationset uses the default metrics from the Kubernetes controller so it does not have the issue.
See https://github.com/agaudreault-jive/argo-cd/blob/8f84ad1b0c0b0216e65e99bf4b7d796f3fef5fdd/cmd/argocd-applicationset-controller/commands/applicationset_controller.go#L118

[rumstead]

Does this PR address the metrics on the applicationset controller as well or just the application controller?

@rumstead the applicationset uses the default metrics from the Kubernetes controller so it does not have the issue. See https://github.com/agaudreault-jive/argo-cd/blob/8f84ad1b0c0b0216e65e99bf4b7d796f3fef5fdd/cmd/argocd-applicationset-controller/commands/applicationset_controller.go#L118

Yup understood where the metrics are coming from. Your PR is closing #12241 which is talking about the applicationset missing the metrics and having a similar issue as described for the application controller (I noted on the ticket the applicationset controller has been missing since 2.4.* ).

It isn't clear to me if this PR restores the missing metrics for the application controller. I suggest we leave the #12241 issue open if it doesn't.

[leoluz]

This PR is replacing custom Argo CD controller metrics (original PR #8318) with the ones provided by the sigs.k8s.io/controller-runtime/pkg/metrics package. Does this package provides a one-to-one match with the ones provided by Argo CD originally. If so, can you please provide evidence?

[leoluz]

Please double check that we are not duplicating metrics computation by introducing "sigs.k8s.io/controller-runtime/pkg/metrics" in the controller metrics.

[leoluz]

LGTM

[gdsoumya]

LGTM! I faced similar issues with the recent argocd versions and was debugging it and came to the same conclusion. Thanks for fixing this!

[agaudreault]

After testing to make sure no metrics were lost, these are the added metrics. The new version of controller-runtime fix the cardinality concern of rest_client. Other "extra" metrics have low cardinality.

Caveat: sometimes when testing in a local cluster, process metrics like process_cpu_seconds_total don't show up in the /metrics endpoint, but when running in live environment they are there, so I assume they are collected properly.

# HELP certwatcher_read_certificate_errors_total Total number of certificate read errors
# TYPE certwatcher_read_certificate_errors_total counter
certwatcher_read_certificate_errors_total
# HELP certwatcher_read_certificate_total Total number of certificate reads
# TYPE certwatcher_read_certificate_total counter
certwatcher_read_certificate_total
# HELP rest_client_requests_total Number of HTTP requests, partitioned by status code, method, and host.
# TYPE rest_client_requests_total counter
rest_client_requests_total{code="200",host="10.43.0.1:443",method="GET"}
rest_client_requests_total{code="200",host="10.43.0.1:443",method="PATCH"}
rest_client_requests_total{code="200",host="dev.argocd.svc:6443",method="GET"}
rest_client_requests_total{code="200",host="production-1.argocd.svc:6443",method="GET"}
rest_client_requests_total{code="200",host="production-2.argocd.svc:6443",method="GET"}
rest_client_requests_total{code="200",host="staging.argocd.svc:6443",method="GET"}
rest_client_requests_total{code="201",host="10.43.0.1:443",method="POST"}
# HELP workqueue_adds_total Total number of adds handled by workqueue
# TYPE workqueue_adds_total counter
workqueue_adds_total{name="app_operation_processing_queue"}
workqueue_adds_total{name="app_reconciliation_queue"}
workqueue_adds_total{name="project_reconciliation_queue"}
# HELP workqueue_depth Current depth of workqueue
# TYPE workqueue_depth gauge
workqueue_depth{name="app_operation_processing_queue"}
workqueue_depth{name="app_reconciliation_queue"}
workqueue_depth{name="project_reconciliation_queue"}
# HELP workqueue_longest_running_processor_seconds How many seconds has the longest running processor for workqueue been running.
# TYPE workqueue_longest_running_processor_seconds gauge
workqueue_longest_running_processor_seconds{name="app_operation_processing_queue"}
workqueue_longest_running_processor_seconds{name="app_reconciliation_queue"}
workqueue_longest_running_processor_seconds{name="project_reconciliation_queue"}
# HELP workqueue_queue_duration_seconds How long in seconds an item stays in workqueue before being requested
# TYPE workqueue_queue_duration_seconds histogram
workqueue_queue_duration_seconds_bucket{name="app_operation_processing_queue",le="1e-08"}
workqueue_queue_duration_seconds_bucket{name="app_operation_processing_queue",le="1e-07"}
workqueue_queue_duration_seconds_bucket{name="app_operation_processing_queue",le="1e-06"}
workqueue_queue_duration_seconds_bucket{name="app_operation_processing_queue",le="9.999999999999999e-06"}
workqueue_queue_duration_seconds_bucket{name="app_operation_processing_queue",le="9.999999999999999e-05"}
workqueue_queue_duration_seconds_bucket{name="app_operation_processing_queue",le="0.001"}
workqueue_queue_duration_seconds_bucket{name="app_operation_processing_queue",le="0.01"}
workqueue_queue_duration_seconds_bucket{name="app_operation_processing_queue",le="0.1"}
workqueue_queue_duration_seconds_bucket{name="app_operation_processing_queue",le="1"}
workqueue_queue_duration_seconds_bucket{name="app_operation_processing_queue",le="10"}
workqueue_queue_duration_seconds_bucket{name="app_operation_processing_queue",le="+Inf"}
workqueue_queue_duration_seconds_sum{name="app_operation_processing_queue"}
workqueue_queue_duration_seconds_count{name="app_operation_processing_queue"}
workqueue_queue_duration_seconds_bucket{name="app_reconciliation_queue",le="1e-08"}
workqueue_queue_duration_seconds_bucket{name="app_reconciliation_queue",le="1e-07"}
workqueue_queue_duration_seconds_bucket{name="app_reconciliation_queue",le="1e-06"}
workqueue_queue_duration_seconds_bucket{name="app_reconciliation_queue",le="9.999999999999999e-06"}
workqueue_queue_duration_seconds_bucket{name="app_reconciliation_queue",le="9.999999999999999e-05"}
workqueue_queue_duration_seconds_bucket{name="app_reconciliation_queue",le="0.001"}
workqueue_queue_duration_seconds_bucket{name="app_reconciliation_queue",le="0.01"}
workqueue_queue_duration_seconds_bucket{name="app_reconciliation_queue",le="0.1"}
workqueue_queue_duration_seconds_bucket{name="app_reconciliation_queue",le="1"}
workqueue_queue_duration_seconds_bucket{name="app_reconciliation_queue",le="10"}
workqueue_queue_duration_seconds_bucket{name="app_reconciliation_queue",le="+Inf"}
workqueue_queue_duration_seconds_sum{name="app_reconciliation_queue"}
workqueue_queue_duration_seconds_count{name="app_reconciliation_queue"}
workqueue_queue_duration_seconds_bucket{name="project_reconciliation_queue",le="1e-08"}
workqueue_queue_duration_seconds_bucket{name="project_reconciliation_queue",le="1e-07"}
workqueue_queue_duration_seconds_bucket{name="project_reconciliation_queue",le="1e-06"}
workqueue_queue_duration_seconds_bucket{name="project_reconciliation_queue",le="9.999999999999999e-06"}
workqueue_queue_duration_seconds_bucket{name="project_reconciliation_queue",le="9.999999999999999e-05"}
workqueue_queue_duration_seconds_bucket{name="project_reconciliation_queue",le="0.001"}
workqueue_queue_duration_seconds_bucket{name="project_reconciliation_queue",le="0.01"}
workqueue_queue_duration_seconds_bucket{name="project_reconciliation_queue",le="0.1"}
workqueue_queue_duration_seconds_bucket{name="project_reconciliation_queue",le="1"}
workqueue_queue_duration_seconds_bucket{name="project_reconciliation_queue",le="10"}
workqueue_queue_duration_seconds_bucket{name="project_reconciliation_queue",le="+Inf"}
workqueue_queue_duration_seconds_sum{name="project_reconciliation_queue"}
workqueue_queue_duration_seconds_count{name="project_reconciliation_queue"}
# HELP workqueue_retries_total Total number of retries handled by workqueue
# TYPE workqueue_retries_total counter
workqueue_retries_total{name="app_operation_processing_queue"}
workqueue_retries_total{name="app_reconciliation_queue"}
workqueue_retries_total{name="project_reconciliation_queue"}
# HELP workqueue_unfinished_work_seconds How many seconds of work has been done that is in progress and hasn't been observed by work_duration. Large values indicate stuck threads. One can deduce the number of stuck threads by observing the rate at which this increases.
# TYPE workqueue_unfinished_work_seconds gauge
workqueue_unfinished_work_seconds{name="app_operation_processing_queue"}
workqueue_unfinished_work_seconds{name="app_reconciliation_queue"}
workqueue_unfinished_work_seconds{name="project_reconciliation_queue"}
# HELP workqueue_work_duration_seconds How long in seconds processing an item from workqueue takes.
# TYPE workqueue_work_duration_seconds histogram
workqueue_work_duration_seconds_bucket{name="app_operation_processing_queue",le="1e-08"}
workqueue_work_duration_seconds_bucket{name="app_operation_processing_queue",le="1e-07"}
workqueue_work_duration_seconds_bucket{name="app_operation_processing_queue",le="1e-06"}
workqueue_work_duration_seconds_bucket{name="app_operation_processing_queue",le="9.999999999999999e-06"}
workqueue_work_duration_seconds_bucket{name="app_operation_processing_queue",le="9.999999999999999e-05"}
workqueue_work_duration_seconds_bucket{name="app_operation_processing_queue",le="0.001"}
workqueue_work_duration_seconds_bucket{name="app_operation_processing_queue",le="0.01"}
workqueue_work_duration_seconds_bucket{name="app_operation_processing_queue",le="0.1"}
workqueue_work_duration_seconds_bucket{name="app_operation_processing_queue",le="1"}
workqueue_work_duration_seconds_bucket{name="app_operation_processing_queue",le="10"}
workqueue_work_duration_seconds_bucket{name="app_operation_processing_queue",le="+Inf"}
workqueue_work_duration_seconds_sum{name="app_operation_processing_queue"}
workqueue_work_duration_seconds_count{name="app_operation_processing_queue"}
workqueue_work_duration_seconds_bucket{name="app_reconciliation_queue",le="1e-08"}
workqueue_work_duration_seconds_bucket{name="app_reconciliation_queue",le="1e-07"}
workqueue_work_duration_seconds_bucket{name="app_reconciliation_queue",le="1e-06"}
workqueue_work_duration_seconds_bucket{name="app_reconciliation_queue",le="9.999999999999999e-06"}
workqueue_work_duration_seconds_bucket{name="app_reconciliation_queue",le="9.999999999999999e-05"}
workqueue_work_duration_seconds_bucket{name="app_reconciliation_queue",le="0.001"}
workqueue_work_duration_seconds_bucket{name="app_reconciliation_queue",le="0.01"}
workqueue_work_duration_seconds_bucket{name="app_reconciliation_queue",le="0.1"}
workqueue_work_duration_seconds_bucket{name="app_reconciliation_queue",le="1"}
workqueue_work_duration_seconds_bucket{name="app_reconciliation_queue",le="10"}
workqueue_work_duration_seconds_bucket{name="app_reconciliation_queue",le="+Inf"}
workqueue_work_duration_seconds_sum{name="app_reconciliation_queue"}
workqueue_work_duration_seconds_count{name="app_reconciliation_queue"}
workqueue_work_duration_seconds_bucket{name="project_reconciliation_queue",le="1e-08"}
workqueue_work_duration_seconds_bucket{name="project_reconciliation_queue",le="1e-07"}
workqueue_work_duration_seconds_bucket{name="project_reconciliation_queue",le="1e-06"}
workqueue_work_duration_seconds_bucket{name="project_reconciliation_queue",le="9.999999999999999e-06"}
workqueue_work_duration_seconds_bucket{name="project_reconciliation_queue",le="9.999999999999999e-05"}
workqueue_work_duration_seconds_bucket{name="project_reconciliation_queue",le="0.001"}
workqueue_work_duration_seconds_bucket{name="project_reconciliation_queue",le="0.01"}
workqueue_work_duration_seconds_bucket{name="project_reconciliation_queue",le="0.1"}
workqueue_work_duration_seconds_bucket{name="project_reconciliation_queue",le="1"}
workqueue_work_duration_seconds_bucket{name="project_reconciliation_queue",le="10"}
workqueue_work_duration_seconds_bucket{name="project_reconciliation_queue",le="+Inf"}
workqueue_work_duration_seconds_sum{name="project_reconciliation_queue"}
workqueue_work_duration_seconds_count{name="project_reconciliation_queue"}

[leoluz]

/cherry-pick release-2.10

[leoluz]

/cherry-pick release-2.9

[gcp-cherry-pick-bot]

Cherry-pick failed with Merge error ff7192bfc56c6a3c249af45e8eacfe5aa47dbcd7 into temp-cherry-pick-1cbc64-release-2.9

[Z3r0Sum]

@leoluz it doesn't look like this made it into 2.9, was that still the plan?

[agaudreault]

it doesn't look like this made it into 2.9, was that still the plan?

@Z3r0Sum the fix could not be cherry picked because it fails with the kubernetes api updates from v0.24 to v0.26 that happened since 2.9. It is not ideal to change the kubernetes dep versions in a patch, especially that this is mostly fixed upstream and https://github.com/kubernetes-sigs/controller-runtime/blob/release-0.14/go.mod uses 0.26, while 2.9 uses the controller runtime 0.11.

There are no plans to cherry-pick this PR because I don't have enough confidence it wont break 2.9, but another fix could be implemented specifically for 2.9. If you have a cluster running 1.25, you can give a try to this PR.