datatrails: persist search terms and use them to reduce retrieved metric names #87884

darrenjaneczek · 2024-05-15T02:40:06Z

What is this feature?

Creates a metric search terms variable.
This variable is stored as a URL parameter, and sent to the prometheus request for metric names.

A limit is added to the prometheus request to prevent an excessive response of metric names, which could lead to a time out. The user is encouraged to add adhoc filters or additional search terms to reduce the number of results returned.

Why do we need this feature?

Three features:

store and persist search terms on trail steps
use search terms to reduce number of metrics returned
set a limit to returned metric names (depends on prometheus version)

Limitations:

The types of regular expressions available to the prometheus match[] parameter excludes "lookahead". Instead, we construct a list of expressions out of the search terms, and requests n of those terms to be present in the metric name (where n is the number of search terms). This will result in false positiives, such as "go hello" matching "go_gone" since "go" appears twice. The same javascript regular expression is used as before, which would filter out "go_gone" since it lacks "hello". Unfortunately, the limit will include these false positives.
Certain (I assume older) versions of prometheus do not consider the "limit" parameter when requesting metric names.

Special notes for your reviewer:

Please check that:

It works as expected from a user's perspective.
If this is a pre-GA feature, it is behind a feature toggle.
The docs are updated, and if this is a notable improvement, it's added to our What's New doc.

torkelo · 2024-06-03T13:53:11Z

public/app/features/trails/MetricSelect/MetricSearchTermsVariable.ts

+export interface MetricsSearchTermsVariableState extends SceneVariableState {
+  terms: string[];
+}
+
+export class MetricSearchTermsVariable
+  extends SceneObjectBase<MetricsSearchTermsVariableState>


Can you explain why we needs to be a new type of variable, why not just use a CustomVariable?

I'll see if I can get everything I really need via CustomVariable.

What I need is a variable that stores a list of search terms that has been typed in, but interpolates these differently:

When persisting the search terms in the URL, I just want to interpolate this as a simple comma-separated list (for URL readability -- not a hard requirement, any simple format would do)

When requesting a list of metric names that match these search terms, I want to this variable to be interpolated into a regular expression that the request to Prometheus will make use of

I was struggling to find a way to override the formatting as I do in getValue(). It seems I can provide value with a formatter, but it seems I would have to provide a formatter function each time I set state value.

Looking more closely at CustomVariable, it just looks like a list of preset custom options you could select one of more of. If that's the case, it might not be what I'm looking for.

Maybe I can extend the TextBoxVariable instead. Maybe I can even render it on the MetricSelectScene directly, simplifying code.

@torkelo
I have simplified much of the code here by extending TextBoxVariable in 2ecb62a.

torkelo · 2024-06-04T17:29:01Z

public/app/features/trails/MetricSelect/util.ts

+  );
+}
+
+export function createJSRegExpFromSearchTerms(searchTerms: string[]) {


what don't get is why this searchTerms is not simply the plain searchQuery string, you can do searchQuery.split(' ' ) split here and build the regex from the terms, same inside createPromRegExp.

Then no need for the custom variable, custom URL sync, etc.

(technically no need for variable, just a state prop that is URL synced). But a variable could be easier as you get the URL sync for free, not sure

what don't get is why this searchTerms is not simply the plain searchQuery string

No longer any reason. I am going to change it to be so.

then no need for the custom variable

Hmm, yes... Here is how it's used currently.

const match = sceneGraph.interpolate(trail, '{${filters}${metricSearch:filter}}');

Arguably, I could construct this query parameter without metricSearch being an interpolated variable.

custom URL sync, etc.

Yeah, that's already gone. Much less to worry about as full string.

But a variable could be easier as you get the URL sync for free

We could have searchQuery be a state prop on DataTrail, to ensure that it persists when metric select scene gets wiped. It shouldn't be hard to urlSync this alongside metric in DataTrail. ~~Hoping it's just a matter of appending the key to the list.~~

torkelo · 2024-06-04T17:35:36Z

public/app/features/trails/MetricSelect/MetricSearchTermsVariable.ts

+export class MetricSearchTermsVariable extends TextBoxVariable {
+  constructor(initialState: Partial<{ value: string }>) {
+    super({
+      value: '',
+      hide: VariableHide.hideVariable,
+      name: VAR_METRIC_SEARCH_TERMS,
+      ...initialState,
+    });
+  }
+
+  getValue(): VariableValue {
+    const filtersVar = sceneGraph.lookupVariable(VAR_FILTERS, this);
+
+    const { value } = this.state;
+    let separator = ',';
+
+    if (filtersVar instanceof AdHocFiltersVariable) {
+      const value = filtersVar.getValue()?.valueOf();
+      if (!value) {
+        separator = '';
+      }
+    }
+
+    return {
+      formatter: (format: Format) => {
+        switch (format) {
+          case 'filter': {
+            const terms = deriveSearchTermsFromInput(value);
+            const regex = createPromRegExp(terms);
+            const result = !terms?.length ? '' : `${separator}__name__=~"${regex}"`;
+            return result;
+          }
+          default:
+            return value;
+        }
+      },
+    };
+  }
+}


just not sure I understand what is going on here.

I think it would be easier to just represent / store it as a search string, but parse it into terms and build the regex on the fly (in a function called from _refreshMetricNames), think representing this as a new type of variable with custom formatter could just be complicating things.

Seems to be the case. It was an interesting exploration into making variables, but on the most part things are much cleaner with a metricSearch state property on DataTrail.

Here is the new implementation of the content you highlighted here;

https://github.com/grafana/grafana/pull/87884/files#r1626618400

darrenjaneczek · 2024-06-04T21:23:26Z

public/app/features/trails/MetricSelect/MetricSelectScene.tsx

+  private async _refreshMetricNames() {
+    const trail = getTrailFor(this);
+    const timeRange: RawTimeRange | undefined = trail.state.$timeRange?.state;
+
+    if (!timeRange) {
+      return;
+    }
+
+    const matchTerms = [];
+
+    const filtersVar = sceneGraph.lookupVariable(VAR_FILTERS, this);
+    const hasFilters = filtersVar instanceof AdHocFiltersVariable && filtersVar.getValue()?.valueOf();
+    if (hasFilters) {
+      matchTerms.push(sceneGraph.interpolate(trail, '${filters}'));
+    }
+
+    const metricSearchRegex = createPromRegExp(trail.state.metricSearch);
+    if (metricSearchRegex) {
+      matchTerms.push(`__name__=~"${metricSearchRegex}"`);
+    }
+
+    const match = `{${matchTerms.join(',')}}`;


Here is how we build the match part of the query now.

torkelo

Nice , I think this looking good now m, wdyt?

torkelo · 2024-06-05T05:16:26Z

public/app/features/trails/DataTrail.tsx

@@ -223,6 +229,8 @@ export class DataTrail extends SceneObjectBase<DataTrailState> {
      stateUpdate.topScene = new MetricSelectScene({});
    }

+    stateUpdate.metricSearch = values.metricSearch?.toString() || undefined;


This you need the same logic as above (only set this if it's a string or clear it if it's null )

it's due to a strange behavior of url sync , values are only passed to this function if the URL values differ from state (so if they are the same but metric is different then this will be called with values having metric value but not metricSearch). Something I have been wanting to change but not come up with a solution for yet grafana/scenes#551

This you need the same logic as above (only set this if it's a string or clear it if it's null )

Ok.
Fixed in 3f5905f

* main: datatrails: persist search terms and use them to reduce retrieved metric names (#87884) CI: Make pkg/build its own module, remove unused Grafana modules in go.mo… (#89243) Fix typo in dashboard-variables destinations and also use Grafana Cloud docs if they exist (#89244) Tooltip: Add tooltip support to Histogram (#89196) Use ref URIs with the reference style links (#89204) Move RecentlyDeleted into browse-dashboards (#89214) Datasource Config: Return error object on failed updates (#88661)

johannes-petzold-snow · 2024-07-16T15:49:42Z

We believe this change is triggering 414 Request-URI Too Large errors due to very long Prometheus request URLs with "match" request parameter introduced by this change, see #90452.

Assuming this is indeed the issue, is there a way to opt out of the functionality introduced by this change, or any other workaround? Thanks in advance.

johannes-petzold-snow · 2024-07-29T15:13:23Z

Hi @darrenjaneczek, any chance you could comment on the 414 Request-URI Too Large issue? This is currently blocking us from upgrading to Grafana 11.x and may be affecting anyone who uses Prometheus datasources with large number of metrics. I can help figure out more detailed repro steps if needed. Thanks in advance!

darrenjaneczek self-assigned this May 15, 2024

grafana-delivery-bot bot added this to the 11.1.x milestone May 15, 2024

grafana-pr-automation bot added the area/frontend label May 15, 2024

darrenjaneczek changed the title ~~Datatrails/variable search terms~~ datatrails: persist search terms and use them to reduce retrieved metric names May 23, 2024

darrenjaneczek mentioned this pull request May 24, 2024

datatrails: refactor: move related code to trails/MetricSelect subdir #88262

Merged

3 tasks

darrenjaneczek requested review from itsmylife and NWRichmond May 24, 2024 13:33

darrenjaneczek force-pushed the datatrails/variable-search-terms branch 3 times, most recently from 50f08ea to 0a7345c Compare May 28, 2024 13:12

darrenjaneczek requested a review from bohandley May 28, 2024 14:53

darrenjaneczek force-pushed the datatrails/variable-search-terms branch from e45aa29 to 5915eb8 Compare May 31, 2024 13:26

darrenjaneczek marked this pull request as ready for review May 31, 2024 13:26

darrenjaneczek requested a review from torkelo as a code owner May 31, 2024 13:26

darrenjaneczek added no-backport Skip backport of PR no-changelog Skip including change in changelog/release notes labels May 31, 2024

darrenjaneczek requested a review from grafanabot as a code owner May 31, 2024 13:50

torkelo reviewed Jun 3, 2024

View reviewed changes

darrenjaneczek requested a review from torkelo June 4, 2024 17:21

darrenjaneczek force-pushed the datatrails/variable-search-terms branch from eb16622 to 2ecb62a Compare June 4, 2024 17:24

torkelo reviewed Jun 4, 2024

View reviewed changes

darrenjaneczek commented Jun 4, 2024

View reviewed changes

darrenjaneczek requested a review from torkelo June 4, 2024 21:25

torkelo approved these changes Jun 5, 2024

View reviewed changes

darrenjaneczek added 5 commits June 6, 2024 09:54

fix: store the metric search terms in a variable

d881df8

fix: adding filters and limits to __name__ request

64a2efb

No empty filters

de0676a

fix

4693d20

Increase the number of metric names

be12a66

darrenjaneczek added 11 commits June 6, 2024 09:54

move metric select scene

d54b598

fix: message & tooltip

14d5713

fix: lint

ec81a51

fix: remove commented junk

3c945dc

fix: acknolwedge worsering to betterer

3e4ed32

fix: simplify by extending TextBoxVariable

231370a

fix: use common function getSearchTermsVariable

30bf344

fix: switch to metricQuery state property

f12341f

fix: console

b932704

fix: similar logic for updating metricSearch

bf10f0c

fix: remove stale code

9b64f41

darrenjaneczek force-pushed the datatrails/variable-search-terms branch from 1af5712 to 9b64f41 Compare June 6, 2024 13:54

darrenjaneczek and others added 2 commits June 14, 2024 11:23

Merge branch 'main' into datatrails/variable-search-terms

9e4b698

fix: increase limit to 20000

b53a7ca

darrenjaneczek merged commit 79540a2 into main Jun 14, 2024
17 checks passed

darrenjaneczek deleted the datatrails/variable-search-terms branch June 14, 2024 16:36

grafana-delivery-bot bot modified the milestones: 11.1.x, 11.2.x Jun 14, 2024

johannes-petzold-snow mentioned this pull request Jul 15, 2024

Dashboards: 414 Error "Request-URI Too Large" Popups After 11.x Grafana upgrade #90452

Open

aangelisc modified the milestones: 11.2.x, 11.2.0 Aug 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

datatrails: persist search terms and use them to reduce retrieved metric names #87884

datatrails: persist search terms and use them to reduce retrieved metric names #87884

darrenjaneczek commented May 15, 2024 •

edited

Loading

torkelo Jun 3, 2024 •

edited

Loading

darrenjaneczek Jun 3, 2024

darrenjaneczek Jun 3, 2024

darrenjaneczek Jun 3, 2024 •

edited

Loading

darrenjaneczek Jun 4, 2024

torkelo Jun 4, 2024 •

edited

Loading

darrenjaneczek Jun 4, 2024 •

edited

Loading

torkelo Jun 4, 2024 •

edited

Loading

darrenjaneczek Jun 4, 2024 •

edited

Loading

darrenjaneczek Jun 4, 2024

torkelo left a comment

torkelo Jun 5, 2024

torkelo Jun 5, 2024 •

edited

Loading

darrenjaneczek Jun 5, 2024

johannes-petzold-snow commented Jul 16, 2024

johannes-petzold-snow commented Jul 29, 2024

datatrails: persist search terms and use them to reduce retrieved metric names #87884

datatrails: persist search terms and use them to reduce retrieved metric names #87884

Conversation

darrenjaneczek commented May 15, 2024 • edited Loading

torkelo Jun 3, 2024 • edited Loading

Choose a reason for hiding this comment

darrenjaneczek Jun 3, 2024

Choose a reason for hiding this comment

darrenjaneczek Jun 3, 2024

Choose a reason for hiding this comment

darrenjaneczek Jun 3, 2024 • edited Loading

Choose a reason for hiding this comment

darrenjaneczek Jun 4, 2024

Choose a reason for hiding this comment

torkelo Jun 4, 2024 • edited Loading

Choose a reason for hiding this comment

darrenjaneczek Jun 4, 2024 • edited Loading

Choose a reason for hiding this comment

torkelo Jun 4, 2024 • edited Loading

Choose a reason for hiding this comment

darrenjaneczek Jun 4, 2024 • edited Loading

Choose a reason for hiding this comment

darrenjaneczek Jun 4, 2024

Choose a reason for hiding this comment

torkelo left a comment

Choose a reason for hiding this comment

torkelo Jun 5, 2024

Choose a reason for hiding this comment

torkelo Jun 5, 2024 • edited Loading

Choose a reason for hiding this comment

darrenjaneczek Jun 5, 2024

Choose a reason for hiding this comment

johannes-petzold-snow commented Jul 16, 2024

johannes-petzold-snow commented Jul 29, 2024

darrenjaneczek commented May 15, 2024 •

edited

Loading

torkelo Jun 3, 2024 •

edited

Loading

darrenjaneczek Jun 3, 2024 •

edited

Loading

torkelo Jun 4, 2024 •

edited

Loading

darrenjaneczek Jun 4, 2024 •

edited

Loading

torkelo Jun 4, 2024 •

edited

Loading

darrenjaneczek Jun 4, 2024 •

edited

Loading

torkelo Jun 5, 2024 •

edited

Loading