Issue using label filter prior to unwrapping the same field

[owen-d]

# This works
sum(sum_over_time({job="loki-ops/query-frontend"}
  |= "metrics.go"
  | logfmt
  | subqueries!=0
  | unwrap subqueries
  [5m]
))

# but this does not parse correctly
sum(sum_over_time({job="loki-ops/query-frontend"}
  |= "metrics.go"
  | logfmt
  | subqueries!="0"
  | unwrap subqueries
  [5m]
))

Note the string !="0" does not work, but the numeric version !=0 does. I suspect we're doing some incorrect type conversion/etc somewhere.

[kavirajk]

quickly checked. Looks like this issue is only when using "!=" operation in label filters. works fine for "=" with both string and numeric values.

And using other binary operations like "<=" or ">=" returns decent error message.

syntax error: unexpected STRING, expecting BYTES or NUMBER or DURATION (<nil>)

Still investigating why is this case for "!=".

[cyriltovena]

I think the query should be

sum(sum_over_time({job="loki-ops/query-frontend"}
  |= "metrics.go"
  | logfmt
  | subqueries!=""
  | unwrap subqueries
  [5m]
))

but I'm not sure why

sum(sum_over_time({job="loki-ops/query-frontend"}
  |= "metrics.go"
  | logfmt
  | subqueries!=0
  | unwrap subqueries
  [5m]
))

work you might have been lucky ? or we do have an issue because empty label should error when converting to a number.

[owen-d]

or we do have an issue because empty label should error when converting to a number.

This is what I suspect is the problem too

[kavirajk]

work you might have been lucky ?

The problem is 100% reproducible btw.

The core of the problem is when trying to extract the metric sample from a logline via unwrap, and if that label doesn't exist on the logline, the this error is set.

it comes from here. https://github.com/grafana/loki/blob/kavirajk%2Flogql-label-check-bug/pkg/logql/log/metrics_extraction.go#L181-L182

Two options here.

1. We can fix this only on during metrics extraction (if label doesn't exist, assume to have value of 0 as sample value)
2. Or we ignore log lines that doesn't contain label during != and !~ like in this PR fix(logql): Make LabelSampleExtractor work as expected when label doesn't exist in logline #6766

Before rejecting the option (2). Please have a look at the below screen shots. Is it ok to have two different responses for these queries.

Q1

{job="loki-ops/query-frontend"}
  |= "metrics.go"
  | logfmt
  | query_type="labels"
  | subqueries!=0

Q2

{job="loki-ops/query-frontend"}
  |= "metrics.go"
  | logfmt
  | query_type="labels"
  | subqueries!="0"

Log queries

With unwrap

I don't mind going with option (1). But just wanted to make sure the above behavior accepted from user point of view.

@cyriltovena @owen-d

[cyriltovena]

So the problem is difference in behaviour between the filter label and the unwrap operation. Unwrap is stricter.

The problem in using sample value 0 for empty label is that you will be fine for sum_over_time it works, but it might not for quantile or min/max_over_time.

May be we discard empty values ? If that is possible.

Before rejecting the option (2). Please have a look at the below screen shots. Is it ok to have two different responses for these queries.

Then don't return the log result so I think so.

[kavirajk]

May be we discard empty values ? If that is possible.

Yes that's possible. We can skip empty values during metrics extraction when using unwrap without changing any of the existing label filter behavior.

Also the more I think about it, subqueries!="0" and subqueries!=0 are two different label filters and should be treated as such. And former can indeed match log lines that doesn't have subqueries label at all (we are assuming it has default empty string substring=" when doesn't exist).

I have change the PR like it's mentioned here.