feat(controller): Enhanced TTL controller scalability #4736
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Alex Collins <alex_collins@intuit.com>
alexec
commented
Dec 14, 2020
| @@ -10,8 +10,7 @@ You cannot horizontally scale the controller. | |||
|
|
|||
| You can scale the controller vertically: | |||
|
|
|||
| - If you have workflows with many steps, increase `--pod-workers`. | |||
There was a problem hiding this comment.
I don't recommend changing --pod-workers anymore. It has no impact in any testing I've done.
alexec
commented
Dec 14, 2020
| @@ -129,9 +130,6 @@ func (c *Controller) enqueueWF(obj interface{}) { | |||
| log.Warnf("'%v' is not an unstructured", obj) | |||
| return | |||
| } | |||
| if un.GetDeletionTimestamp() != nil { | |||
There was a problem hiding this comment.
redundant now it is checked by FilterFunc on line 52
alexec
commented
Dec 14, 2020
| @@ -187,9 +192,14 @@ func (c *Controller) deleteWorkflow(key string) error { | |||
|
|
|||
| err = c.wfclientset.ArgoprojV1alpha1().Workflows(wf.Namespace).Delete(wf.Name, &metav1.DeleteOptions{PropagationPolicy: commonutil.GetDeletePropagation()}) | |||
| if err != nil { | |||
| return err | |||
| if apierr.IsNotFound(err) { | |||
There was a problem hiding this comment.
I've not seen this error apart from initial testing - prevented by line 155 I think, but a good bonus check
Signed-off-by: Alex Collins <alex_collins@intuit.com>
alexec
changed the title
alexec
commented
Dec 15, 2020
| }, | ||
| Handler: cache.ResourceEventHandlerFuncs{ | ||
| AddFunc: controller.enqueueWF, | ||
| UpdateFunc: func(old, new interface{}) { | ||
| controller.enqueueWF(new) | ||
| }, | ||
| DeleteFunc: controller.enqueueWF, |
There was a problem hiding this comment.
this is invoked after the WF is deleted - so it is not needed - removing reduces the number of API requests by 1
alexec
commented
Dec 15, 2020
alexec
commented
Dec 15, 2020
alexec
commented
Dec 15, 2020
alexec
commented
Dec 15, 2020
Signed-off-by: Alex Collins <alex_collins@intuit.com>
Signed-off-by: Alex Collins <alex_collins@intuit.com>
sarabala1979
approved these changes
Dec 15, 2020
| @@ -100,6 +101,7 @@ func NewRootCommand() *cobra.Command { | |||
| command.Flags().StringVar(&logLevel, "loglevel", "info", "Set the logging level. One of: debug|info|warn|error") | |||
| command.Flags().IntVar(&glogLevel, "gloglevel", 0, "Set the glog logging level") | |||
| command.Flags().IntVar(&workflowWorkers, "workflow-workers", 32, "Number of workflow workers") | |||
| command.Flags().IntVar(&workflowTTLWorkers, "workflow-ttl-workers", 4, "Number of workflow TTL workers") | |||
There was a problem hiding this comment.
Thanks for adding this. This was my proposal during the prod incident. But We don't want to make too many changes
simster7
pushed a commit
that referenced
this pull request
Jan 4, 2021
Signed-off-by: Alex Collins <alex_collins@intuit.com>
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Alex Collins alex_collins@intuit.com
Checklist: