Add controller to enable metrics/alerts for argocd instances #54
Conversation
|
Hi @jopit. Thanks for your PR. I'm waiting for a redhat-developer member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
@wtam2018 @bigkevmcd Could you also please take a look at this PR? |
|
/ok-to-test @jopit added to org. |
| namespace.Labels = make(map[string]string) | ||
| } | ||
| namespace.Labels["openshift.io/cluster-monitoring"] = "true" | ||
| err = r.client.Update(context.TODO(), &namespace) |
There was a problem hiding this comment.
Service account should have update on namespaces ( note to self )
There was a problem hiding this comment.
@sbose78 do I need to add namespaces to the deploy/olm-catalog/gitops-operator/manifests/gitops-operator.clusterserviceversion.yaml file?
There was a problem hiding this comment.
There was a problem hiding this comment.
That should be fine.
| "Namespace", readRoleBinding.Namespace, "Name", readRoleBinding.Name) | ||
| err = r.client.Create(context.TODO(), readRoleBinding) | ||
| if err != nil { | ||
| reqLogger.Info("Error creating a new read role binding", |
There was a problem hiding this comment.
Should this be reqLogger.Error(..) ?
| } | ||
|
|
||
| // Create role to grant read permission to the openshift metrics stack | ||
| err = r.createReadRoleIfAbsent(request.Namespace, reqLogger) |
There was a problem hiding this comment.
Could have used the view cluster role but that probably isn't a good idea since view cluster role grants a lot more.
There was a problem hiding this comment.
Also set ownerReferences to the corresponding ArgoCD CR ?
| // Create ServiceMonitor for ArgoCD application metrics | ||
| serviceMonitorLabel := fmt.Sprintf("%s-metrics", request.Name) | ||
| serviceMonitorName := request.Name | ||
| err = r.createServiceMonitorIfAbsent(request.Namespace, serviceMonitorName, serviceMonitorLabel, reqLogger) |
There was a problem hiding this comment.
Also set ownerReferences to the corresponding ArgoCD CR ?
| } | ||
|
|
||
| // Create role binding to grant read permission to the openshift metrics stack | ||
| err = r.createReadRoleBindingIfAbsent(request.Namespace, reqLogger) |
There was a problem hiding this comment.
Also set ownerReferences to the corresponding ArgoCD CR ?
| // Create ServiceMonitor for ArgoCD API server metrics | ||
| serviceMonitorLabel = fmt.Sprintf("%s-server-metrics", request.Name) | ||
| serviceMonitorName = fmt.Sprintf("%s-server", request.Name) | ||
| err = r.createServiceMonitorIfAbsent(request.Namespace, serviceMonitorName, serviceMonitorLabel, reqLogger) |
There was a problem hiding this comment.
Also set ownerReferences to the corresponding ArgoCD CR ?
| // Create ServiceMonitor for ArgoCD repo server metrics | ||
| serviceMonitorLabel = fmt.Sprintf("%s-repo-server", request.Name) | ||
| serviceMonitorName = fmt.Sprintf("%s-repo-server", request.Name) | ||
| err = r.createServiceMonitorIfAbsent(request.Namespace, serviceMonitorName, serviceMonitorLabel, reqLogger) |
There was a problem hiding this comment.
Also set ownerReferences to the corresponding ArgoCD CR ?
| } | ||
|
|
||
| // Create alert rule | ||
| err = r.createPrometheusRuleIfAbsent(request.Namespace, reqLogger) |
There was a problem hiding this comment.
Also set ownerReferences to the corresponding ArgoCD CR ?
| return &ArgoCDMetricsReconciler{client: mgr.GetClient(), scheme: mgr.GetScheme()} | ||
| } | ||
|
|
||
| func (r ArgoCDMetricsReconciler) Reconcile(request reconcile.Request) (reconcile.Result, error) { |
There was a problem hiding this comment.
This should be func (r *ArgoCDMetricsReconciler) Reconcile...
| return err | ||
| } | ||
|
|
||
| func (r ArgoCDMetricsReconciler) createReadRoleBindingIfAbsent(namespace string, reqLogger logr.Logger) error { |
There was a problem hiding this comment.
This should be func (r *ArgoCDMetricsReconciler) createReadRoleBindingIfAbsent...
| s.AddKnownTypes(monitoringv1.SchemeGroupVersion, &monitoringv1.PrometheusRule{}) | ||
| return s | ||
| } | ||
|
|
There was a problem hiding this comment.
Comsider using gotest.tools assert package
https://pkg.go.dev/gotest.tools/v3@v3.0.3/assert
| @@ -0,0 +1,351 @@ | |||
| package argocdmetrics | |||
There was a problem hiding this comment.
This file should probaby be renamed to argocd_metrics_controller.go.
| "Namespace", request.Namespace) | ||
| return reconcile.Result{}, err | ||
| } | ||
| _, exists := namespace.Labels["openshift.io/cluster-monitoring"] |
There was a problem hiding this comment.
Use constant for openshift.io/cluster-monitoring?
|
@jopit Can you please provide the steps for how to verify the changes |
|
@sbose78 @wtam2018 @amitkrout I believe I've addressed all of the review comments, could you all please take another look at this PR? |
| return reconcile.Result{}, err | ||
| } | ||
|
|
||
| const clusterMonitoringLabel = "openshift.io/cluster-monitoring" |
There was a problem hiding this comment.
This const could be promoted to package scope so that the test module uses it as well.
There was a problem hiding this comment.
I don't think it's a good practice to share constants between test and production code. If the constant got accidentally changed in the production code, the test would still pass, but the code would fail when running for real.
| reqLogger.Info("Creating new read role", | ||
| "Namespace", readRole.Namespace, "Name", readRole.Name) | ||
|
|
||
| // Set the ArgoCD instance as the owner and controller |
There was a problem hiding this comment.
The comment is a bit confusing. Do you mean "the owner of controlled 'readRole`"?
I think this comment can be removed.
|
@jopit added a testing note for QE / @dewan-ahmed @amitkrout Could you list the resources created that one should expect to be gone when an ArgoCD CR is removed ? |
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: wtam2018 The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
@sbose78 @dewan-ahmed @amitkrout I've added the list of resources that should be deleted when an argocd instance is deleted to the initial comment of this PR. |
|
Successfully validated following the instructions in the PR. attached screenshot and logs for reference. cc: @amitkrout
dewanahmed@dahmed-mac gitops-operator % oc get argocd -n openshift-gitops |
Adds a new controller to watch for the creation of ArgoCD instances and to add the configuration to enable the built-in metrics stack to scrape metrics from the new ArgoCD. Also adds an alert rule to report ArgoCD applications that are out of sync.
How to test by running the operator locally
Login to an openshift cluster from the command line
Download the operator-sdk binary from here: https://github.com/operator-framework/operator-sdk/releases/tag/v0.17.2
Check out the argocd-metrics branch from https://github.com/jopit/gitops-operatorCheck out the master branch (since the change has been merged)
cd gitops-operator
go mod vendor
kubectl apply -f deploy/crds
operator-sdk run --local --watch-namespace ""
Create a new argocd application in the openshift-gitops namespace with auto-sync turned off, do not sync the app. It may take several minutes (I've seen it take over five minutes) but eventually an alert should show up on the overview page in the openshift admin console.
You can also go to the Monitoring>Metrics page in the admin console and enter a PromQL query, for example
sum(argocd_app_info{dest_namespace=~"default",health_status!=""}) by (sync_status)Testing note by Shoubhik
Please validate that deleting up an ArgoCD instance also removes all associated rolebindings/roles and monitoring objects that were created.
Resources to be automatically deleted
Where
%sis the name of the argocd instance.