Pagination support for Ruler and Alertmanager resource listing APIs

[alvinlin123]

Is your feature request related to a problem? Please describe.

There are uses cases from my customers that needs large amount of rules, alerts, and silences. Lager means more than 10K+ with an upper bound limited by underlying infra.

Currently resource listing APIs in Ruler and Alert Manager returns everything in one call which is not scalable because 1) marshalling the large amount of structs into JSON is CPU and memory intensive and 2) transferring large amount of JSON is time consuming and occupies value resources like memory and network bandwidth for too long.

Describe the solution you'd like
I would like to have pagination support for any APIs in Ruler and Alert Manager that list resuorces.

Describe alternatives you've considered

• I told the customer to not create too many stuff and, they rejected this proposal with good reasons.
• I thought about just limit amount of data returned and have customer leverage API filter parameters, but that is not customer friendly nor solving the problem 100%.

There are existing issues/proposals in Prometheus to support pagination:

• [Feature Request] Pagination support for rule listing APIs prometheus/prometheus#12120
• [proposal] additional API endpoints to retrieve rules information prometheus/prometheus#12320
• Proposal: API spec for paginated version of getAlertGroups and getAlerts prometheus/alertmanager#3361

My ask

I think we can partner with Mimir in grafana/mimir#5657 in influencing having Prometheus maintainers to make a decision on an API spec. I also wonder if Cortex can support pagination behind a feature flag before Prometheus does.