Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Up kubernetes/client-go QPS and Burst config #640

Closed
julianvmodesto opened this issue Oct 17, 2017 · 4 comments

Comments

Projects
None yet
2 participants
@julianvmodesto
Copy link
Contributor

commented Oct 17, 2017

kubernetes/client-go uses rate-limiting to prevent spamming the apiserver.

We had to rapidly scale our backend up, however Voyager Operator stopped updating the HAProxy ConfigMap's list of backend Endpoints, and unfortunately the ConfigMap didn't have a majority of our endpoint. Due to kubernetes/client-go throttling, the ConfigMap wasn't updated in an adequate time period so it was quicker for us to delete the Voyager Operator pod.

Voyager Operator logs showed many entries of the form request.go:638] Throttling request took 130.848105ms, ... showing that apiserver calls were being throttled client-side, including hooks for PUTs to update the HAProxy ConfigMap.

We can use the QPS and Burst config to prevent throttling to this extent https://godoc.org/k8s.io/client-go/rest#Config

@tamalsaha

This comment has been minimized.

Copy link
Member

commented Oct 18, 2017

ConfigMaps are watched / updated from operator pod and watched/read from the HAProxy pods. Should not this mean that both of those flags are configurable? Currently we use inCluster.Config. So, I am guessing this is just using the default values.

I don't see how the deployment flags can be configured. One option is annotation, the other option is to pass from the operator pods (if using non-default setting).

@julianvmodesto

This comment has been minimized.

Copy link
Contributor Author

commented Oct 18, 2017

Looks like the client config will be built from the cluster config, but it seems like it's primarily getting auth.

Could we set it directly on the prometheus-operator client in run.go? As for what the config should be, the default Burst is 10 and QPS is 5. I looked at Prometheus Operator, and they use rest.InClusterConfig, but they also set Burst to 100 and QPS to 100. Nginx Ingress Controller also uses rest.InClusterConfig and overrides to even higher values, of Burst to 1e6 and QPS to 1e6 -- this makes sense and is probably closer to what we want since Endpoints may be changing rapidly if there are many Ingresses watching many Services with many Endpoints.

@julianvmodesto

This comment has been minimized.

Copy link
Contributor Author

commented Oct 18, 2017

Ahh i see what you're saying -- yes it seems both kubeClients should be configured w/ a higher Burst and QPS.

This looks like a good pattern to use inClusterConfig w/ overrides:
https://github.com/coreos/prometheus-operator/blob/9d68ecf289d711c66bef39d2f83429265abc6986/pkg/k8sutil/k8sutil.go#L79-L97

@tamalsaha

This comment has been minimized.

Copy link
Member

commented Oct 18, 2017

appscode/voyager:4.0.0-rc.17 has this pr.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.