Question about behavior of file cluster

[JosephFY]

We are trying to achieve a goal of documenting the metrics that doesn't match any rule instead of just sending them to a blackhole cluster.

To avoid having the file being so big, we want to do some logger rotation of the file to keep for certain number of days. We tried testing by sending metrics that doesn't match any role and it landed in the file as expected BUT if we were to remove/rename that file it looks like the process wont recreate the file (just like an append concept when doing cat somefile >> another file ) until the service gets restarted which means the metrics doesn't gets recorded and we are not sure where they will land eventually !.

So the question is; is this is intended behavior? and if it is, is there any workaround or away of approaching what we want ? the configuration portion is as below

cluster dropped_metrics file ip /carbon-c-relay/blackhole/dropped_metrics.metric;
match * send to dropped_metrics ;

[grobian]

Hey, in short, yes, this is intended behaviour. I quickly looked to see if SIGHUP can do it, but it won't. Your best way to do it is either restart the relay, or change the file in the config file, and SIGHUP the relay. Then it will for sure close and open the new file.

[JosephFY]

I kinda found a work around solution for this but I have not implemented it yet, my idea is to cat the file to a newer file attaching the date to it and then truncate -s 0 blackholed_metrics_file which it will empty the file in place without deleting/recreating it so the relay wont see it gone.
The only down side I can see of this is that I have to put it a cronjob which not bad but just one more thing to think about in upgrades, recovery, maintenance ...etc

I wish that the file cluster would take a date in it's name, then it will create a file on a daily basis. I think this will be useful in my case and other case as well when metrics data are so important and duplicate records should be retained for recovery and backup.

@grobian what do you think ?

[grobian]

Problem with a date in the name is that I need to check on every write whether I have to open a new file, which isn't that efficient. It's much "friendlier" to respond to a signal like SIGHUP in a way that tools like logrotate can just deal with everything.