Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add parameters to support change work queue #841

Merged
merged 2 commits into from
Jun 29, 2020

Conversation

cwdsuzhou
Copy link
Contributor

Fix #840

@cwdsuzhou
Copy link
Contributor Author

tests seem unhappy since #834 merged

@@ -198,8 +217,9 @@ func NewPodController(cfg PodControllerConfig) (*PodController, error) {
ready: make(chan struct{}),
done: make(chan struct{}),
recorder: cfg.EventRecorder,
k8sQ: workqueue.NewNamedRateLimitingQueue(workqueue.DefaultControllerRateLimiter(), "syncPodsFromKubernetes"),
deletionQ: workqueue.NewNamedRateLimitingQueue(workqueue.DefaultControllerRateLimiter(), "deletePodsFromKubernetes"),
k8sQ: workqueue.NewNamedRateLimitingQueue(controllerRateLimiter(cfg.WorkQueueRetryQPS), "syncPodsFromKubernetes"),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Were the previous defaults 10?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeap, it is defined 10 in the original client-go

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

btw, what do you recommend?

In our environment, we have changed it to 100 now to accelerate deleting pods.

@sargun
Copy link
Contributor

sargun commented Jun 23, 2020

tests should be fixed by #842 -- the problem is that K8s just release 1.17.7. This broke the validation script.

@sargun
Copy link
Contributor

sargun commented Jun 24, 2020

btw, what do you recommend?

@cwdsuzhou
Copy link
Contributor Author

Before this works, we should wait for the PR virtual-kubelet/node-cli#22 in node-cli to be merged

@sargun
Copy link
Contributor

sargun commented Jun 25, 2020

@cwdsuzhou I'll hold off on the merge. But, this to me looks like it should work, and fallback to 10.

@cpuguy83
Copy link
Contributor

Currently we do not expose anything about the queue, or even that we use a queue to the package API. This changes that.

@cpuguy83
Copy link
Contributor

I wonder if it would be better to expose the queue to the config instead of the queue config.
The downside here is we are stuck with the workqueue.RatelimitingInterface in that case... not that it's particularly bad, just need to be careful about what we expose.

@cwdsuzhou
Copy link
Contributor Author

@cwdsuzhou I'll hold off on the merge. But, this to me looks like it should work, and fallback to 10.

yes, this should work

@cwdsuzhou
Copy link
Contributor Author

cwdsuzhou commented Jun 26, 2020

I wonder if it would be better to expose the queue to the config instead of the queue config.
The downside here is we are stuck with the workqueue.RatelimitingInterface in that case... not that it's particularly bad, just need to be careful about what we expose.

This is better. But I prefer to exposing the workqueue.RateLimiter since the origin one just implemented this.

@cwdsuzhou
Copy link
Contributor Author

@cpuguy83 @sargun

PR updated, PTAL thanks

@cwdsuzhou cwdsuzhou changed the title Add parameters to support change work queue qps Add parameters to support change work queue Jun 26, 2020
@cwdsuzhou
Copy link
Contributor Author

@cpuguy83 @sargun

PR updated, PTAL thanks

I will close virtual-kubelet/node-cli#22 and re-open a new one when this PR has been merged.

@sargun
Copy link
Contributor

sargun commented Jun 26, 2020

@cpuguy83 Are you okay with this, or do you want to take a different approach of queue / queue configuration / queue configuration policy?

Copy link
Contributor

@cpuguy83 cpuguy83 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is good. LGTM

@cpuguy83 cpuguy83 merged commit bd977cb into virtual-kubelet:master Jun 29, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

DefaultControllerRateLimiter may cause slowly update
3 participants