carbonserver: introduce new max-inflight-requests and no-service-when-index-is-not-ready configs #443
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bom-d-van
force-pushed
the
carbonserver/rate-limiting
branch
from
f0ba6b9
to
ae3366e
Compare
…-index-is-not-ready configs carbonserver.max-inflight-requests: Maximum inflight requests allowed in go-carbon. Default is 0 which means no limit. If limit is reached, go-carbon/carbonserver returns 429 (Too Many Requests). This option would be handy as if there are too many long running requests piling up, it would slows down data ingestion in persister and slows down index building (especially during restart). carbonserver.no-service-when-index-is-not-ready: Returns 503 (Service Unavailable) when go-carbon is still building the first index cache (trie/trigram) after restart. With trie or trigram index, carbonserver falls back to filepath.Glob when index is not ready. And filepath.Glob is expensive and not scaled enough to support some heavy queries (manifested as high usage and frequent GC). Thus it is possible that the read requests is so heavy that it not only slows down index building, it also hinders persister from flushing down data to disk and causing cache.size overflow. By default, it is disabled. It is recommend to enable this flag together with "file-list-cache" (depends on the number of metrics, indexing building with "file-list-cache" is usually much faster than re-scanning the whole file tree after restart). These two configs could be used to stop carbonserver from starving persister and index building goroutines.
bom-d-van
force-pushed
the
carbonserver/rate-limiting
branch
from
ae3366e
to
29ff325
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
carbonserver.max-inflight-requests:
carbonserver.no-service-when-index-is-not-ready:
These two configs could be used to stop carbonserver from starving persister
and indexing goroutines.