vmui: add explore tab for exploration of metrics, which belong to a particular job/instance

[valyala]

Is your feature request related to a problem? Please describe.

vmui doesn't provide functionality for exploration of metrics, which belong to a particular job (aka a set of scrape targets with identical metrics) or instance (aka scrape target).

Describe the solution you'd like

It would be great if vmui would provide an explore tab with the following functionality:

1. A time range selector similar to the selector used in the Graph tab for selecting arbitrary time range.
2. A drop-down list of job label values available on the selected time range. This list can be obtained via /api/v1/label/job/values API, which accepts also a time range via start and end query args in the same way as /api/v1/query_range API does.
3. A drop-down list of instance label values available on the selected time range. This list must be updated with values for the selected job by issuing a query to /api/v1/label/instance/values?match[]={job="selected_job_here"}&start=...&end=... - see API docs for details.
4. vmui should load the list of all the metric names for the selected job and print it as a sorted list with one metric name per each line. The list of metric names must be obtained from /api/v1/label/__name__/values?match={job="selected_job_here"}.
5. Every metric name in the printed list must be pre-pended with a clickable + icon. When user clicks on the icon, then the corresponding graph for this metric on the selected time range must be drawn below the metric name. The following query must be used for the graph depending on whether job / instance is selected and depending on the metric name:
   a) if metric name ends with _total, _sum or _count, then it should be automatically wrapped into rate() (e.g. we assume that the given metric is a counter ).
   b) if job is selected while instance isn't selected, then the query from step a) should be wrapped into sum(...) without (job)

Such a functionality should simplify exploration of metrics collected from Prometheus-compatible scrape targets.

Describe alternatives you've considered

To build a separate dashboard with all the metrics exported by each job in Grafana. This isn't a trivial task.

[valyala]

See additional enhancements for this feature at #3470 (comment)

[valyala]

The explore metrics tab has been added in the commit 4deea60 . This commit will be included in the next release of VictoriaMetrics. In the meantime it is possible to play with it at https://play.victoriametrics.com/select/accounting/1/6a716b0f-38bc-4856-90ce-448fd713e3fe/prometheus/graph/#/metrics

[valyala]

@Loori-R , I played a bit with the Explore metrics tab at VictoriaMetrics playground. It has been appeared that the idea with the expandable list of metric names isn't the best one. It isn't very convenient to use when the job exposes more than 10 metrics, and this is the most frequent case in practice where a typical job exports a few hundreds of metrics.

It would be better showing the metrics list in the same way as the list of job and instance labels with the ability to quickly locate the needed metric name via search for metrics with the particular substring. The only distinction is that every metric in the list must contain a checkbox in front of it. If the checkbox is checked, then the corresponding graph for this metric should appear below. The checkbox can be checked in the following ways:

• by clicking it
• by clicking the metric name in the list
• by selecting the metric name in the list with up/down keyboard keys and then pressing enter or space on the keyboard

The checkbox can be unchecked in the same ways as by checking it.

The title of per-metric graph should contain the following information and controls:

• The metric name itself, so users could select and copy the name into clipboard. This functionality already exists in the current implementation.
• The rate() checkbox, so users could add/remove rate() to the metric. This functionality already exists in the current implementation.
• The x icon for closing the graph.

The show only opened metrics toggle should be removed, since it won't have sense after implementing the functionality above.

The initial focus should be put at the job input list after opening the Explore metrics tab.

It would be great also to have the ability to manually change the order of opened graphs. This is useful for placing the related graphs closer to each other. Probably, every title of the graph should contain also the ^ and v buttons for moving the graph up/down in the list of opened graphs.

Additionally, it would be great to have an ability to change the vertical size of opened graphs. This should simplify comparison of multiple opened graphs. Probably, to add a drop-down list with the name Graph size at the top of Explore metrics tab with the following items: small, medium, big. The default value should be medium - in this case the vertical graph sizes should equal to 40% of vertical screen space. The small value should reduce the vertical graph size to 20% of the vertical screen space, while the big should increase the vertical graph size to 80% of vertical screen space.

The graph legend should be hidden at Explore metrics page, since it doesn't contain useful information, while it steals non-trivial amounts of scarce vertical space.

[leeoniya]

with the ability to quickly locate the needed metric name via search for metrics with the particular substring

i wrote a lib for similar purpose to use in Grafana. maybe it's interesting: https://github.com/leeoniya/uFuzzy

[valyala]

The Explore metrics tab is available in VMUI starting from v1.86.0. See, for example, https://play.victoriametrics.com/select/accounting/1/6a716b0f-38bc-4856-90ce-448fd713e3fe/prometheus/graph/?g0.range_input=30m&g0.end_input=2023-01-11T02%3A06%3A09&g0.step_input=4s&g0.relative_time=last_30_minutes#/metrics . Closing the feature request as done.