feat: Add ignore-resources-tracking annotation to ignore resources update #18343

kahoulei · 2024-05-21T21:42:13Z

This is the enhancement request to allow user ignore any resource update.

Previously we have a feature ignoreResourceUpdatesEnabled which allow user to ignore certain part of the resource manifest. But that feature does not fulfill the use case of newly created object.

When an dependent object is created, there is no old manifest to compare and therefore we always refresh in such case. This is wasting a lot of compute cycle if a cluster has a lot of dependent objects (e.g. jobs and pods created by cronjobs).

Instead, this PR introduces an annotation argocd.argoproj.io/ignore-resources-tracking so that user can use it on a specific resource and argocd can completely ignore it.

Also, this annotation will only work behind the same ignoreResourceUpdatesEnabled flag.

Checklist:

kahoulei · 2024-05-21T21:44:49Z

cc @agaudreault

agaudreault · 2024-05-31T18:24:18Z

controller/cache/cache.go

@@ -67,6 +68,9 @@ const (

 	// EnvClusterCacheRetryUseBackoff is the env variable to control whether to use a backoff strategy with the retry during cluster cache sync
 	EnvClusterCacheRetryUseBackoff = "ARGOCD_CLUSTER_CACHE_RETRY_USE_BACKOFF"
+
+	// AnnotationIgnoreResourcesTracking is a Kubernetes annotation for a Kubernetes resource to ignore any resources tracking
+	AnnotationIgnoreResourcesTracking = "argocd.argoproj.io/ignore-resources-tracking"


I would keep the same terminology. resource tracking means it does not show up in Argo UI and this would lead to confusion

Suggested change

AnnotationIgnoreResourcesTracking = "argocd.argoproj.io/ignore-resources-tracking"

AnnotationIgnoreResourcesUpdate = "argocd.argoproj.io/ignore-resources-update"

agaudreault · 2024-05-31T19:26:43Z

controller/cache/cache.go

+	// annotations stores all the ObjectRef annotations
+	annotations map[string]string


I think we can save memory here and just have a bool instead. annotations like kubectl.kubernetes.io/last-applied-configuration can end up consuming a lot if you consider the number of resources argo watches

agaudreault · 2024-05-31T19:28:21Z

controller/cache/cache.go

@@ -342,6 +349,14 @@ func skipAppRequeuing(key kube.ResourceKey) bool {
 }

 func skipResourceUpdate(oldInfo, newInfo *ResourceInfo) bool {
+	if val, ok := newInfo.annotations[AnnotationIgnoreResourcesTracking]; ok {


I think you should ignore only if the health status did not change.

agaudreault · 2024-05-31T19:34:35Z

controller/cache/cache.go

@@ -549,7 +564,7 @@ func (c *liveStateCache) getCluster(server string) (clustercache.ClusterCache, e
 					"name":        ref.Name,
 					"api-version": ref.APIVersion,
 					"kind":        ref.Kind,
-				}).Debug("Ignoring change of object because none of the watched resource fields have changed")
+				}).Debugf("Ignoring change of object because none of the watched resource fields have changed or annotation %v is set to true", AnnotationIgnoreResourcesTracking)


I don't think the debug message adds much information. I would either leave it as-is, or perhaps add a log field with the values of the annotation. This way we can query logs to know what was ignored because of annotations.

Suggested change

}).Debugf("Ignoring change of object because none of the watched resource fields have changed or annotation %v is set to true", AnnotationIgnoreResourcesTracking)

}).Debug("Ignoring change of object because none of the watched resource fields have changed")

agaudreault · 2024-05-31T19:52:44Z

@kahoulei Something I was thinking but had not the time to complete was to do the same, but with an "allow" annotation instead of a "ignore". Basically, argocd.argoproj.io/apply-resources-updates: "true". When this annotation is specified, it performs the same logic as we have today: generating the hash based on configuration.

The advantage was that you dont have to always ignore the full resources, but you can laverage the ignoreResourceUpdates configurations.

kahoulei · 2024-06-04T19:13:02Z

@agaudreault thanks for the review.

If I understand correctly, for the resource that we want to ignore we should set argocd.argoproj.io/apply-resources-updates: "false". Is it correct?

If argocd.argoproj.io/apply-resources-updates: "true", we will just compare the generated hash as is.

agaudreault · 2024-06-04T19:57:03Z

@agaudreault thanks for the review.

If I understand correctly, for the resource that we want to ignore we should set argocd.argoproj.io/apply-resources-updates: "false". Is it correct?

If argocd.argoproj.io/apply-resources-updates: "true", we will just compare the generated hash as is.

Correct, if the annotation is there and is true, we would generate a hash and compare it later on. If the annotation is false or missing, then we do nothing (like what we have today)

kahoulei · 2024-06-04T22:30:22Z

If the annotation is false or missing, then we do nothing (like what we have today)

I thought we will generate hash if ignoreResourceUpdatesEnabled is true. So if the annotation is missing, we will generate the hash anyway? (for backward compatibility)

see https://github.com/argoproj/argo-cd/blob/master/controller/cache/cache.go#L507-L514

kahoulei · 2024-06-05T18:44:16Z

@agaudreault PTAL if you get a chance. Thanks!

agaudreault · 2024-06-05T19:18:54Z

controller/cache/info.go

+		if k == AnnotationApplyResourcesUpdate {
+			value, err := strconv.ParseBool(v)
+			if err != nil {
+				value = false
+			}
+			res.applyResourcesUpdate = value
+		}


The code can be simplified by just checking this at the end of shouldHashManifest method.
If the shouldHashManifest return true because the annotations is present for an untracked resource, then it will take the same codepath as current behavior for tracked resources.

No need to update the ResourceInfo, store booleans or annotations.

@agaudreault updated with your suggestion.

One additional change in skipResourceUpdate: since we are not going to generate hash and therefore the logic of isSameManifest need to change. So PTAL.

We simply cannot generate a hash by default or skip the resources updates for all the resources. When the annotation is not defined, the hash should not be generated. If you generate a hash by default for all resource updates, the controller will use a lot more CPU generating hashes than what you will save skipping reconcile.

I also think we should not go in a direction where non-argocd resources must have argocd annotations.

That is why I think the best approach is:

Generate the hash only if annotation is present and true.

(already done) Skip the resource updates if the hashes exist and the hashes match OR if the resource does not belong to any application

…date Signed-off-by: kahoulei <kahou.lei@okta.com>

Signed-off-by: kahoulei <kahou.lei@okta.com>

codecov · 2024-06-06T20:02:10Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 47.65%. Comparing base (355ec75) to head (e4b3837).
Report is 75 commits behind head on master.

❗ Current head e4b3837 differs from pull request most recent head bec8ffe

Please upload reports for the commit bec8ffe to get more accurate results.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #18343      +/-   ##
==========================================
+ Coverage   45.01%   47.65%   +2.64%     
==========================================
  Files         354      332      -22     
  Lines       48018    46835    -1183     
==========================================
+ Hits        21613    22317     +704     
+ Misses      23597    21669    -1928     
- Partials     2808     2849      +41

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

kahoulei · 2024-06-07T17:03:59Z

@agaudreault I have added unit tests and they all passed. But the e2e tests are failing and looks like they are not related. Can you help me to take a look? (Sorry I am still not familiar with the e2e tests yet)

controller/cache/cache.go

agaudreault · 2024-06-10T12:28:22Z

controller/cache/cache.go

@@ -348,20 +352,37 @@ func skipResourceUpdate(oldInfo, newInfo *ResourceInfo) bool {
 		return false
 	}
 	isSameHealthStatus := (oldInfo.Health == nil && newInfo.Health == nil) || oldInfo.Health != nil && newInfo.Health != nil && oldInfo.Health.Status == newInfo.Health.Status
-	isSameManifest := oldInfo.manifestHash != "" && newInfo.manifestHash != "" && oldInfo.manifestHash == newInfo.manifestHash
+	isSameManifest := oldInfo.manifestHash == newInfo.manifestHash


This condition should not change. If a resource does not have a hash, that means the annotation was not there (or not true), therefore we should not ignore updates.

Ok, I misunderstood your intention. So you want is the dependent object set argocd.argoproj.io/apply-resources-update=true and then use the existing ignore resources config to skip refresh. I will update the PR.

controller/cache/cache.go

controller/cache/cache_test.go

docs/operator-manual/reconcile.md

docs/user-guide/annotations-and-labels.md

Signed-off-by: kahoulei <kahou.lei@okta.com>

kahoulei · 2024-06-12T22:13:35Z

@agaudreault can you take a look if anything else I need to change?

agaudreault

Logic looks good. Most comments are about terminology and user documentation to polish.

agaudreault · 2024-06-13T16:25:05Z

controller/cache/cache.go

+	// AnnotationApplyResourcesUpdate when set to true on a resource that is not tracked under an app, argocd will generate a
+	// hash and apply `ignoreResourceUpdate` configuration on it. If the annotation is set to false (or not presented) on a resource
+	// that is not tracked under an app, ignoreResourceUpdates configuration will not be applied.


Leaking implementation details on the API.

Suggested change

// AnnotationApplyResourcesUpdate when set to true on a resource that is not tracked under an app, argocd will generate a

// hash and apply `ignoreResourceUpdate` configuration on it. If the annotation is set to false (or not presented) on a resource

// that is not tracked under an app, ignoreResourceUpdates configuration will not be applied.

// AnnotationApplyResourcesUpdate when set to true on an untracked resource,

// argo will apply `ignoreResourceUpdates` configuration on it.

agaudreault · 2024-06-13T16:25:56Z

controller/cache/cache.go

+	// AnnotationApplyResourcesUpdate when set to true on a resource that is not tracked under an app, argocd will generate a
+	// hash and apply `ignoreResourceUpdate` configuration on it. If the annotation is set to false (or not presented) on a resource
+	// that is not tracked under an app, ignoreResourceUpdates configuration will not be applied.
+	AnnotationApplyResourcesUpdate = "argocd.argoproj.io/apply-resources-update"


Now that the code is written, I think it will be much easier for users to have the same terminology with the existing feature. wdyt?

Suggested change

AnnotationApplyResourcesUpdate = "argocd.argoproj.io/apply-resources-update"

AnnotationIgnoreResourceUpdates = "argocd.argoproj.io/ignore-resource-updates"

@agaudreault I think AnnotationApplyResourcesUpdate is more logical to me. AnnotationIgnoreResourceUpdates seems the invert of what we are doing and therefore we need to flip our logic if we rename it.

agaudreault · 2024-06-13T16:26:41Z

controller/cache/cache.go

 	// Best      - Only hash for resources that are part of an app or their dependencies
 	// (current) - Only hash for resources that are part of an app + all apps that might be from an ApplicationSet
 	// Orphan    - If orphan is enabled, hash should be made on all resource of that namespace and a config to disable it
 	// Worst     - Hash all resources watched by Argo
-	return appName != "" || (gvk.Group == application.Group && gvk.Kind == application.ApplicationKind)
+	isTrackedResources := appName != "" || (gvk.Group == application.Group && gvk.Kind == application.ApplicationKind)


Only one resource

Suggested change

isTrackedResources := appName != "" || (gvk.Group == application.Group && gvk.Kind == application.ApplicationKind)

isTrackedResource := appName != "" || (gvk.Group == application.Group && gvk.Kind == application.ApplicationKind)

agaudreault · 2024-06-13T16:35:11Z

docs/operator-manual/reconcile.md

+
+## Tracking Dependent Resources
+
+Dependent resources by default are not being tracked. Therefore, we cannot generate any hash of those objects and utilize 
+the `ignoreResourceUpdates` configuration.
+
+If you want to track the dependent object and apply the `ignoreResourceUpdates` configuration, you can add 
+`argocd.argoproj.io/apply-resources-update=true` annotation in the dependent resources manifest:


I think the concepts of tracked resources and watched resources here are mixed up. Also the user facing documentation is leaking the implementation.

Suggested change

## Tracking Dependent Resources

Dependent resources by default are not being tracked. Therefore, we cannot generate any hash of those objects and utilize

the `ignoreResourceUpdates` configuration.

If you want to track the dependent object and apply the `ignoreResourceUpdates` configuration, you can add

`argocd.argoproj.io/apply-resources-update=true` annotation in the dependent resources manifest:

## Ignoring updates for untracked resources

ArgoCD will only apply `ignoreResourceUpdates` configuration to tracked resources of an application. This means dependant resources, such as a `ReplicaSet` and `Pod` created by a `Deployment`, will not ignore any updates and trigger a reconcile of the application for any changes.

If you want to apply the `ignoreResourceUpdates` configuration to an untracked resource, you can add the

`argocd.argoproj.io/ignore-resource-updates=true` annotation in the dependent resources manifest.

agaudreault · 2024-06-13T16:36:33Z

docs/operator-manual/reconcile.md

+          restartPolicy: OnFailure
+```
+
+Then you can update `argocd-cm` configMap to ignore the dependent resources:


Suggested change

Then you can update `argocd-cm` configMap to ignore the dependent resources:

The resource updates will be ignored based on your the `ignoreResourceUpdates` configuration in the `argocd-cm` configMap:

agaudreault · 2024-06-13T16:37:24Z

docs/operator-manual/reconcile.md

+Note: If you set `argocd.argoproj.io/apply-resources-update: "false"`, no hash will be generated and `ignoreResourceUpdates`
+cannot be applied on those resources.


Implementation details.

Suggested change

Note: If you set `argocd.argoproj.io/apply-resources-update: "false"`, no hash will be generated and `ignoreResourceUpdates`

cannot be applied on those resources.

Signed-off-by: kahoulei <kahou.lei@okta.com>

kahoulei · 2024-06-17T15:45:37Z

@agaudreault I updated the doc. Thanks.

Signed-off-by: kahoulei <kahou.lei@okta.com>

agaudreault · 2024-06-18T13:20:18Z

controller/cache/cache.go

+
+	// AnnotationIgnoreResourcesUpdate when set to true on an untracked resource,
+	// argo will apply `ignoreResourceUpdates` configuration on it.
+	AnnotationIgnoreResourcesUpdate = "argocd.argoproj.io/ignore-resources-update"


Let's make the name consistent. Make sure to validate all references

Suggested change

AnnotationIgnoreResourcesUpdate = "argocd.argoproj.io/ignore-resources-update"

AnnotationIgnoreResourceUpdates = "argocd.argoproj.io/ignore-resource-updates"

Signed-off-by: kahoulei <kahou.lei@okta.com>

agaudreault

Code LGTM. Waiting for a before/after graph on an instance with real load before merging

Signed-off-by: kahoulei <kahou.lei@okta.com>

ronaknnathani · 2024-06-21T15:30:41Z

Glad to see that this feature is being added. We have a case where we run large k8s clusters (~5k nodes) with tens of thousands of pods. Pod updates result in significant reconciliation activity and we are looking to leverage this feature.

A quick question about this change - if let's say updates on /status are ignored for an object like Pods, would the UI not show the latest Pod state in that case or would this only skip the Application reconciliation?

kahoulei · 2024-06-24T14:28:03Z

A quick question about this change - if let's say updates on /status are ignored for an object like Pods, would the UI not show the latest Pod state in that case or would this only skip the Application reconciliation?

@ronaknnathani the behavior is same as ignoreResourceUpdates feature.

kahoulei · 2024-06-27T04:27:04Z

Discussed with @agaudreault about load testing. I don't have environment to test it currently. If someone know how to load test it, please leave a comment here so that we can merge it.

kahoulei requested a review from a team as a code owner May 21, 2024 21:42

kahoulei requested a review from a team as a code owner May 22, 2024 15:02

jfatta approved these changes May 29, 2024

View reviewed changes

agaudreault suggested changes May 31, 2024

View reviewed changes

agaudreault reviewed Jun 5, 2024

View reviewed changes

kahoulei added 12 commits June 6, 2024 12:31

feat: Add ignore-resources-tracking annotation to ignore resources up…

78644c9

…date Signed-off-by: kahoulei <kahou.lei@okta.com>

add doc

a22404e

Signed-off-by: kahoulei <kahou.lei@okta.com>

update annotation doc

5157dce

Signed-off-by: kahoulei <kahou.lei@okta.com>

refactor annotation usage base on comment feedback

782621e

Signed-off-by: kahoulei <kahou.lei@okta.com>

update annotation

62dc5dc

Signed-off-by: kahoulei <kahou.lei@okta.com>

do not store boolean in resourceInfo

e0acf88

Signed-off-by: kahoulei <kahou.lei@okta.com>

typo

ba861c1

Signed-off-by: kahoulei <kahou.lei@okta.com>

update logic

a0fe22a

Signed-off-by: kahoulei <kahou.lei@okta.com>

refactor

7128bb0

Signed-off-by: kahoulei <kahou.lei@okta.com>

add comment

5f70557

Signed-off-by: kahoulei <kahou.lei@okta.com>

add tests

d9f7dfa

Signed-off-by: kahoulei <kahou.lei@okta.com>

update doc

140c3d8

Signed-off-by: kahoulei <kahou.lei@okta.com>

kahoulei force-pushed the kahou/ignore-resources-tracking-16415 branch from 42fb0bb to 140c3d8 Compare June 6, 2024 19:31

Merge branch 'master' into kahou/ignore-resources-tracking-16415

cfa2baa

agaudreault reviewed Jun 10, 2024

View reviewed changes

kahoulei added 2 commits June 10, 2024 09:30

update code base on comment feedback

b1fe0b9

Signed-off-by: kahoulei <kahou.lei@okta.com>

update annotation doc

e51e61a

Signed-off-by: kahoulei <kahou.lei@okta.com>

kahoulei and others added 2 commits June 10, 2024 10:36

Merge branch 'master' into kahou/ignore-resources-tracking-16415

295eab3

fix goimport

0cc1287

Signed-off-by: kahoulei <kahou.lei@okta.com>

kahoulei force-pushed the kahou/ignore-resources-tracking-16415 branch from 8f0225d to 0cc1287 Compare June 10, 2024 19:15

fix golint

0d94782

Signed-off-by: kahoulei <kahou.lei@okta.com>

kahoulei requested a review from agaudreault June 11, 2024 00:22

kahoulei and others added 2 commits June 10, 2024 17:27

update comments

9da4335

Signed-off-by: kahoulei <kahou.lei@okta.com>

Merge branch 'master' into kahou/ignore-resources-tracking-16415

b3abf14

agaudreault suggested changes Jun 13, 2024

View reviewed changes

update docs

06589c5

Signed-off-by: kahoulei <kahou.lei@okta.com>

update annotation name

c29e0fb

Signed-off-by: kahoulei <kahou.lei@okta.com>

kahoulei force-pushed the kahou/ignore-resources-tracking-16415 branch from 7c0577c to c29e0fb Compare June 18, 2024 03:21

agaudreault reviewed Jun 18, 2024

View reviewed changes

rename annotation

e4b3837

Signed-off-by: kahoulei <kahou.lei@okta.com>

agaudreault approved these changes Jun 18, 2024

View reviewed changes

lint check

bec8ffe

Signed-off-by: kahoulei <kahou.lei@okta.com>

ronaknnathani mentioned this pull request Jun 21, 2024

Provide an option to ignore dependents resources update which are created a resource tracked by argocd #16415

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add ignore-resources-tracking annotation to ignore resources update #18343

feat: Add ignore-resources-tracking annotation to ignore resources update #18343

kahoulei commented May 21, 2024 •

edited

Loading

kahoulei commented May 21, 2024 •

edited

Loading

agaudreault May 31, 2024 •

edited

Loading

agaudreault May 31, 2024

agaudreault May 31, 2024

agaudreault May 31, 2024

agaudreault commented May 31, 2024 •

edited

Loading

kahoulei commented Jun 4, 2024

agaudreault commented Jun 4, 2024 •

edited

Loading

kahoulei commented Jun 4, 2024 •

edited

Loading

kahoulei commented Jun 5, 2024

agaudreault Jun 5, 2024

kahoulei Jun 5, 2024

agaudreault Jun 5, 2024 •

edited

Loading

codecov bot commented Jun 6, 2024 •

edited

Loading

kahoulei commented Jun 7, 2024

agaudreault Jun 10, 2024

kahoulei Jun 10, 2024 •

edited

Loading

kahoulei commented Jun 12, 2024

agaudreault left a comment

agaudreault Jun 13, 2024

agaudreault Jun 13, 2024

kahoulei Jun 17, 2024

agaudreault Jun 13, 2024

agaudreault Jun 13, 2024

agaudreault Jun 13, 2024

agaudreault Jun 13, 2024

kahoulei commented Jun 17, 2024

agaudreault Jun 18, 2024

agaudreault left a comment •

edited

Loading

ronaknnathani commented Jun 21, 2024

kahoulei commented Jun 24, 2024

kahoulei commented Jun 27, 2024

	AnnotationIgnoreResourcesTracking = "argocd.argoproj.io/ignore-resources-tracking"
	AnnotationIgnoreResourcesUpdate = "argocd.argoproj.io/ignore-resources-update"

		// annotations stores all the ObjectRef annotations
		annotations map[string]string

	}).Debugf("Ignoring change of object because none of the watched resource fields have changed or annotation %v is set to true", AnnotationIgnoreResourcesTracking)
	}).Debug("Ignoring change of object because none of the watched resource fields have changed")

	AnnotationApplyResourcesUpdate = "argocd.argoproj.io/apply-resources-update"
	AnnotationIgnoreResourceUpdates = "argocd.argoproj.io/ignore-resource-updates"

	isTrackedResources := appName != "" \|\| (gvk.Group == application.Group && gvk.Kind == application.ApplicationKind)
	isTrackedResource := appName != "" \|\| (gvk.Group == application.Group && gvk.Kind == application.ApplicationKind)

	Then you can update `argocd-cm` configMap to ignore the dependent resources:
	The resource updates will be ignored based on your the `ignoreResourceUpdates` configuration in the `argocd-cm` configMap:

		Note: If you set `argocd.argoproj.io/apply-resources-update: "false"`, no hash will be generated and `ignoreResourceUpdates`
		cannot be applied on those resources.

feat: Add ignore-resources-tracking annotation to ignore resources update #18343

Are you sure you want to change the base?

feat: Add ignore-resources-tracking annotation to ignore resources update #18343

Conversation

kahoulei commented May 21, 2024 • edited Loading

kahoulei commented May 21, 2024 • edited Loading

agaudreault May 31, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

agaudreault commented May 31, 2024 • edited Loading

kahoulei commented Jun 4, 2024

agaudreault commented Jun 4, 2024 • edited Loading

kahoulei commented Jun 4, 2024 • edited Loading

kahoulei commented Jun 5, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

agaudreault Jun 5, 2024 • edited Loading

Choose a reason for hiding this comment

codecov bot commented Jun 6, 2024 • edited Loading

Codecov Report

kahoulei commented Jun 7, 2024

Choose a reason for hiding this comment

kahoulei Jun 10, 2024 • edited Loading

Choose a reason for hiding this comment

kahoulei commented Jun 12, 2024

agaudreault left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kahoulei commented Jun 17, 2024

Choose a reason for hiding this comment

agaudreault left a comment • edited Loading

Choose a reason for hiding this comment

ronaknnathani commented Jun 21, 2024

kahoulei commented Jun 24, 2024

kahoulei commented Jun 27, 2024

kahoulei commented May 21, 2024 •

edited

Loading

kahoulei commented May 21, 2024 •

edited

Loading

agaudreault May 31, 2024 •

edited

Loading

agaudreault commented May 31, 2024 •

edited

Loading

agaudreault commented Jun 4, 2024 •

edited

Loading

kahoulei commented Jun 4, 2024 •

edited

Loading

agaudreault Jun 5, 2024 •

edited

Loading

codecov bot commented Jun 6, 2024 •

edited

Loading

kahoulei Jun 10, 2024 •

edited

Loading

agaudreault left a comment •

edited

Loading