AutoComplete/Template Variables: Prometheus auto-complete and template variables can be very slow to load on large clusters

[harry671003]

What happened?

When a Prometheus compatible data source such as Cortex is used with millions of time series, the Grafana auto-complete and template variables can be very slow to load.

Screen.Recording.2024-05-14.at.1.49.09.PM.mov

What did you expect to happen?

Auto complete is expected to be much quicker to be useful to the users of Grafana.

Did this work before?

No response

How do we reproduce it?

Video recording attached.

Is the bug inside a dashboard panel?

No response

Environment (with versions)?

Grafana: Amazon Managed Grafana
OS: Mac OS X
Browser: Firefox

Grafana platform?

Other

Datasource(s)?

Prometheus

[harry671003]

Recently Prometheus has introduced a new LIMIT parameter for the metadata APIs.
prometheus/prometheus#13396

Also see the HTTP reference:

• https://prometheus.io/docs/prometheus/latest/querying/api/#finding-series-by-label-matchers
• https://prometheus.io/docs/prometheus/latest/querying/api/#getting-label-names
• https://prometheus.io/docs/prometheus/latest/querying/api/#querying-label-values

Would it be a good idea to pass a configurable limit (eg: 10K) when calling these APIs in auto-complete?

[harry671003]

Duplicate of other issue?

• Prometheus: Use limit parameter to manage metric names payloads #85444
• Prometheus: Improve Code Mode autocomplete performance for large numbers of metric names #84612