datatrails: persist search terms and use them to reduce retrieved metric names

[darrenjaneczek]

What is this feature?

Creates a metric search terms variable.
This variable is stored as a URL parameter, and sent to the prometheus request for metric names.

A limit is added to the prometheus request to prevent an excessive response of metric names, which could lead to a time out. The user is encouraged to add adhoc filters or additional search terms to reduce the number of results returned.

Why do we need this feature?

Three features:

• store and persist search terms on trail steps
• use search terms to reduce number of metrics returned
• set a limit to returned metric names (depends on prometheus version)

Limitations:

• The types of regular expressions available to the prometheus match[] parameter excludes "lookahead". Instead, we construct a list of expressions out of the search terms, and requests n of those terms to be present in the metric name (where n is the number of search terms). This will result in false positiives, such as "go hello" matching "go_gone" since "go" appears twice. The same javascript regular expression is used as before, which would filter out "go_gone" since it lacks "hello". Unfortunately, the limit will include these false positives.
• Certain (I assume older) versions of prometheus do not consider the "limit" parameter when requesting metric names.

Special notes for your reviewer:

• When comparing files, enable the "ignore whitespace option"

Please check that:

• It works as expected from a user's perspective.
• If this is a pre-GA feature, it is behind a feature toggle.
• The docs are updated, and if this is a notable improvement, it's added to our What's New doc.