You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
Currently the PrometheusRules api endpoint, at the worst case, will list all the rules from all rules. For a cluster with hundreds of rulers one api call will trigger hundreds of GRPC calls, one for each ruler, and if one of these GRPC calls fail we return 5xx. So as the number of rulers increase, the chance of one grpc call failing causing the entire list rules call to fail also increases.
Is your feature request related to a problem? Please describe.
Currently the
PrometheusRules
api endpoint, at the worst case, will list all the rules from all rules. For a cluster with hundreds of rulers one api call will trigger hundreds of GRPC calls, one for each ruler, and if one of these GRPC calls fail we return 5xx. So as the number of rulers increase, the chance of one grpc call failing causing the entire list rules call to fail also increases.Describe the solution you'd like
There should be a retry mechanism on the grpc call to make the endpoint more robust. https://github.com/cortexproject/cortex/blob/master/pkg/ruler/ruler.go#L907
The text was updated successfully, but these errors were encountered: