Blocking bad metrics through statsD or Graphite Carbon relay

[kira510]

Hello Team,

I have set up a statsD node that sends metrics to a cluster wide carbon relay server(only carbon relay installed on a linux server). And this carbon relay server forwards the metrics to 3 graphite nodes(version 0.9.15). And finally i render the graphs on grafana.

Users are sending data that takes a lot of time to be rendered on grafana since they send numerous number of sub-metrics for single data source name (for example: Test.numerous.*.count , * is a huge number of metrics within Test) .

Could you please advise:

1. How to track the data source as it is very difficult to know who is sending the data?
2. How to block metrics based on certain pattern at statsD or cluster wide carbon relay level?

Thanks in advance,
Kiran

[wolfzhaoshuai]

For question 2, I think carbon-c-relay can help you. Carbon-c-relay block, transfer, and aggregate metrics based on pre-defined patterns.

[deniszh]

Hi @kira510,
Sorry for late response, I missed your question somehow.
First, it's quite hard to track connections in graphite cluster, unfortunately. You can enable access logging and you will see metrics and source IP of the requestor in the log.
For the second question - yes, what @wolfzhaoshuai said. Graphite has blacklist feature too, but for cluster-wide blocking, you may prefer do it on relay level. 3rd party carbon-c-relay or graphite-ng relays are able to do that.

[kira510]

Hi @deniszh @wolfzhaoshuai ,

Thanks for the suggestions. You may close this now.

Regards,