Grafana dashboard for Contour doesn't work with lots of virtualhost

[rosskukulinski]

What steps did you take and what happened:

I created many IngressRoute objects, each with a unique virtualhost. Viewing the Contour dashboard for valid and invalid IngressRoutes did not work (grafana put a ! icon indicating something was wrong with the graph query).

What did you expect to happen:

Grafana graphs for invalid and valid ingressroutes should work regardless of the number of virtualhosts or namespace.

Anything else you would like to add:

The issue seems to be with the Grafana variables we have, Vhost and namespace. They support Grafana's "All" value, which by default lists every possible option. This is bad for prometheus:

By default the All value will include all options in combined expression. This can become very long and can have performance problems. Many times it can be better to specify a custom all value, like a wildcard regex. To make it possible to have custom regex, globs or lucene syntax in the Custom all value option it is never escaped so you will have to think avbout what is a valid value for your data source.

We should have a custom ALL value of .* for these (and any other) variables in our dashboards that might have high cardinality.