Filter invalid lines.

[piotr1212]

Checks the number of fields in a line, there should be three fields. Fields are
separated by spaces, count the spaces and forward only lines which have two
spaces in them.

It's been a while since I wrote C... please check carefully.

[grobian]

so,the problem is you want the relay to discard lines that do not follow the <metric> <value> <timestamp>\n pattern?

[piotr1212]

Yes, that is correct. I'm currently at a large organisation with an "open" Graphite for other teams to use. A lot of them send badly formatted metrics, I would like to filter them out on the relay instead of sending them to the downstream carbon-cache's.

[grobian]

Hmm, I see the usecase here. We have the same setup ("open" Graphite) but rely on the stores dealing with random crap people seem to produce. I guess performance-wise this doesn't make much difference, visibility wise, I think it needs a new counter metricsInvalid or something to check we're not dropping like mad unexpectedly. I need to think this through a bit.

[piotr1212]

I eventuall want to log the invalid lines someway, not sure yet how as I want to avoid disk run full when someone messes up and starts sending millions invalid lines every second (things like that happen..).

The reason why I would prefer this in the relay instead of the stores is that the relay has the source IP of the offender (and could log that), where the stores only see source ip of the relay.

[grobian]

Heh, now you actually request the same as issue #78, and the source IP is currently hard to get at the level where we'd log.

[piotr1212]

I'm more interested in logging wrongly formatted lines, the other request is for correct lines based on non-wanted metricname

Take for example the following lines:

1. metric.somepath.aa 42 1438155337
2. metric.somepath.aa 42 nonsense
3. unwanted.path.bb 42 1438155337

with #78 you can configure that 1 and 2 will be sent to dest1 and 3 to be logged. My intention is to only drop/log line 2.
These lines could be discarded before the routing logic occurs, thus making it easier to include an IP. I actually hacked some code to see if it would work, and seems to do so. I expect to have some more time next week to work on this.

[grobian]

Something like this https://github.com/Skyscanner/carbon-relay-ng/commit/69eafe322140b59d728cdc0be2e04b36f7552fa3 makes me feel like it should be an option whether or not to do validation.

[piotr1212]

Sorry for the long delay. I've started a new project at a new customer and my focus got a bit changed by that. I don't expect to have time to finish it soon. :(
Work this far is https://github.com/piotr1212/carbon-c-relay/commits/filter2 which implements:

• basic filtering
• logging invalid lines
• logging ip addresses for upd and tcp socket both ipv4 and ipv6
• configurable
  I'm not really satisfied by the filtering part, after I made it configurable it made the code more complex and less efficient.

Anyway, I don't mind if you close this one and leave unimplemented.

[piotr1212]

Sorry for late response. Thank you for reviewing and commenting my code. The suggested regex would be great! Unfortunately I still don't expect to have time to rework this soon ... :(

[grobian]

don't worry, this is on my todo stack for implementing, I hope to be able to reuse your intial patch.

[ravibhooshan]

Is this Value filtering been implemented in latest release.

[grobian]

nope

[grobian]

I've introduced a validate clause to match rules