carbonserver: adds /admin/info endpoint for returning internal info #433
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
currently only cache info are returned. it could be used by other programs like buckytool to check if the operations are hurting go-carbon and causing disruptions of the service.
bom-d-van
added a commit
to go-graphite/buckytools
that referenced
this pull request
Oct 22, 2021
…trol
The original sync mechanism is a bit too simple, as it is controllyed by only workers.
It is hard to balance both efficiency and reliability. Setting the worker count too low,
rebalancing becomes too slow; setting it too high, buckyd might take away too much
resources.
To meet both criteria, this changes introduced a go-carbon health check and a automatic
sync rate adjustments based on metrics per second per node.
bucky rebalance -f -offload -ignore404 \
-graphite-ip-to-hostname \
-graphite-metrics-prefix carbon.minutely.buckytool.rebalance.$cluster.dst.$to_location.src.$from_location \
-graphite-endpoint 127.0.0.1:3002 \
-go-carbon-health-check=$enable_go_carbon_health_check \
-go-carbon-health-check-interval 5 \
-go-carbon-port 8080 \
-go-carbon-protocol http \
-go-carbon-cache-threshold 0.5 \
-sync-speed-up-interval $sync_speed_up_interval \
-metrics-per-second $sync_metrics_per_second \
-h ${seed_node}:4242 \
-workers $workers \
-timeout $timeout
Using the above example for explanation:
-go-carbon-health-check=true: asks buckytools to check go-carbon cache usage, which indicates if there is capacity issues.
-sync-speed-up-interval: means increased sync rate (metrics per second) every specified seconds.
-metrics-per-second: means the initial sync rate. Should be set to a lower value like 5 - 10 initially while using -sync-speed-up-interval.
Whats more, with go-carbon health check, buckytools also does automatic sync rate easing. Could be disabled by `-no-random-easing=false`.
Full flags could be found with `bucky rebalance -help`.
Besides go-carbon health check, this commits also introduce exporting some internal sync rate metrics to graphite
for monitoring. Could be enabled by -graphite-metrics-prefix.
By default -go-carbon-health-check is disabled, which means buckytools fallback to the original sync behaviour.
Relevant go-carbon PR: go-graphite/go-carbon#433
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
currently only cache info are returned. it could be used by other programs like buckytool
to check if the operations are hurting go-carbon and causing disruptions of the
service.
this is mostly an early setup. more stat and info could be added and returned in the future.