Best practice for configuring huge alert rules?

[zxwing]

My system allows users to create alarms dynamically which are essentially implemented by Prometheus's alert rules. I have two ways to do this:

1. put all rules in a big single file
2. create rule files for every single rule and list them in rule_files

reload the prometheus by kill -1.

Which approach will you recommend for a high concurrent environment with huge rules? for example, 10000 people create 10000 rules concurrently. The #1 may cause a very large file that blows up memory while #2 may lead to a long for...loop.

The DNSMASQ used to have the performance issue in the aforementioned scenario; they fixed it using inotify with rules per file after I reported the issue.

I know I have #3 that groups rules in files with decent size; I just want to know if prometheus has been designed for this scenario.

[brian-brazil]

I presume there's only a very small number of potential types of alerts users can create, likely less than 10.

The way to approach this is to create those 10 alerts, and have the thresholds and notification targets coming from 10 metrics (one per alert type).

This unittest should point you in the right direction: https://github.com/prometheus/prometheus/blob/master/promql/testdata/operators.test#L322

[zxwing]

Thanks

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.