Skip to content
This repository has been archived by the owner on Feb 27, 2023. It is now read-only.

Discovery: Make queue parameters configurable #140

Open
alexbrand opened this issue Jun 5, 2018 · 5 comments
Open

Discovery: Make queue parameters configurable #140

alexbrand opened this issue Jun 5, 2018 · 5 comments
Labels
discoverer kind/feature Categorizes issue or PR as related to a new feature. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete.

Comments

@alexbrand
Copy link
Contributor

Is this a BUG REPORT, PERFORMANCE REPORT or FEATURE REQUEST?:
FEATURE REQUEST

What happened:
During performance testing, we noticed that the discoverers were taking a long time to drain the discovery queue when testing with a large number of services (>1k). The issue is that we are using the default client-go rate limited queue, which limits us to 10 items per second.

What you expected to happen:
Ideally, the rate limit should be configurable so that it can be adjusted according to the environment.

@alexbrand alexbrand self-assigned this Jun 5, 2018
@alexbrand alexbrand added kind/feature Categorizes issue or PR as related to a new feature. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. discoverer labels Jun 5, 2018
@alexbrand
Copy link
Contributor Author

After further investigation, I realized that what is actually limiting us is the rate at which the API server can create services.

The API server we are using for testing can handle approximately 4 services per second. I arrived at this number using time and kubectl. This means that discovering 1000 services will take approximately 4 minutes, which is pretty much what we are seeing in our performance tests.

Lowering the priority on this for now.

cc: @stevesloka @rosskukulinski

@alexbrand alexbrand added priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. and removed priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels Jun 7, 2018
@alexbrand alexbrand removed their assignment Jun 7, 2018
@rosskukulinski
Copy link
Contributor

Are we creating Services one by one or are we using ServerList type?

@alexbrand
Copy link
Contributor Author

One by one. There is no API endpoint that takes a list of services for creation AFAIK.

@rosskukulinski
Copy link
Contributor

@alexbrand is this the same as #150 which was merged?

@alexbrand
Copy link
Contributor Author

@rosskukulinski This is different. This issue is about the queue we use to put all the services and endpoints that we have found in the backend cluster, before they are created in the Gimbal cluster. Currently, we are using the default rate limiter for this queue.

Issue #150 was about the Kubernetes client, which also has a built-in rate limiter to control the number of QPS.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
discoverer kind/feature Categorizes issue or PR as related to a new feature. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete.
Projects
None yet
Development

No branches or pull requests

2 participants