Loki: Option to add derived fields based on labels #76162

5cat · 2023-10-07T08:01:09Z

What is this feature?
Loki only derives fields by a regex matcher, this limits its usage when functions such as line_formatter is used on top of the logs.
Some users already have logs parsed in json or logfmt structure which are detected as fields in loki.
This pull request allows the mapping between detected fields values and derived values by matching the fields' names.
~~Currently the feature is behind lokiDerivedFieldsFromLabels feature toggle.~~

Who is this feature for?

This feature is for users who want to have a minimal readable log lines while still having the ability to jump to traces via derived fields.
This feature is for users who prefer to offload the extraction of fields in their queries rather than being limited to what is regex can do.

Which issue(s) does this PR fix?:

Fixes #73598

Special notes for your reviewer:

I did my best to pattern match my way through this, and I dont understand the full implications of my changes so I have added a feature toggle for this.
I did my best to maintain backward compatibility and avoid any refactoring, wanted to keep my changes minimal.

Please check that:

It works as expected from a user's perspective.
If this is a pre-GA feature, it is behind a feature toggle.
The docs are updated, and if this is a notable improvement, it's added to our What's New doc.

Loki only derives fields by a regex matcher, this limits its usage when functions such as `line_formatter` is used on top of the logs. Some users already have logs parsed in json or logfmt structure which are detected as fields in loki. This pull request allows the mapping between detected fields values and derived values by matching the fields' names. Currently the feature is behind `lokiEnableNameMatcherOption` feature toggle.

svennergr · 2023-10-09T13:41:18Z

Hi @5cat,

thanks a lot for your contribution. We will look into this and provide feedback in the upcoming days.

While we do this, are you sure this PR will fix #73598?

Thanks,
@svennergr

5cat · 2023-10-09T16:27:39Z

Hey @svennergr, yes it fixes the issue.

For example, in the past if you did add line_format in your query, the regex method for deriving the fields will not work since it operates on the displayed line, which is already edited by the line_format function.
With this pull request, having loki parse the logs and detect the field (with functions such as json logfmt unpack etc) is enough.

I called this matching option "Name matcher", since it matches the fields based on their names. which can be enabled from the provisioned datasource.yml via the enableNameMatcher option (which i made sure it is backward compatible) assuming you have toggled lokiEnableNameMatcherOption feature in grafana.ini:

  - name: gdev-loki
    type: loki
    access: proxy
    url: http://localhost:3100
    jsonData:
      derivedFields:
        - name: "traceID"
          enableNameMatcher: true
          url: "$${__value.raw}"
          datasourceUid: gdev-tempo

Or in the UI like this

So you can have a query such as {namespace="monitoring"} | logfmt | traceID != '' | line_format '[{{.level}}]: {{.msg}}'

And get derived fields correctly setup with the link.

now users can add line_format function to nicely format their logs while at the same time enjoy the ability to quickly jump into traces.

maybe @mladedav or @ThommyH can further confirm my work fixes these issues #73598 grafana/loki#9209 grafana/loki#6614 grafana/loki#9209 because they are all the same.

gtk-grafana · 2023-10-23T14:13:11Z

@svennergr can we help make progress on this one?

jseiser · 2023-10-23T19:53:02Z

Just wanted to chime in here, we just ran into this issue in our POC of replacing our linkerd Jaeger deployment, with Linkerd/Tempo, since the integration between Loki and Tempo would be a huge plus for us.

All of our logs come in structured, so while I can see the traceID in the logs, and i can copy/paste into tempo, we have no way of actually using the integration that would be its biggest feature.

ivanahuckova · 2023-10-24T10:56:41Z

@jseiser you should be able to do this trough new feature in Grafana - correlations. Here is the documentation for it https://grafana.com/docs/grafana/latest/administration/correlations/

JakeCooper · 2023-11-07T05:27:52Z

@jseiser you should be able to do this trough new feature in Grafana - correlations. Here is the documentation for it https://grafana.com/docs/grafana/latest/administration/correlations/

I've tried to set this up but it doesn't seem to work in the way we'd want? We'd want to ingest JSON and be able to show a link on the JSON key

I've attempted to use the correlations to do that, but it seems to only go off fields, so I need to use a derived field anyways, which only works if the field is itself in the query

JakeCooper · 2023-11-07T05:28:14Z

@ivanahuckova ^^

ivanahuckova · 2023-11-07T09:17:14Z

only go off fields

It does off data frame fields. In query inspector, you can see data frame fields. And you can then use transformation on extracting the value. So if your field is in labels, you can choose that as field and use regex to capture the field you want

svennergr · 2023-11-10T15:06:46Z

Hi @5cat and others,

I know this PR has been open for a long time. I just wanted to let you know that I will be looking into this in the next days.
Thank you for your patience and your contribution to Grafana!

…eld-from-detected-fields

ivanahuckova

Left couple of comments, but overall looks good! 👍

ivanahuckova · 2023-11-14T10:37:58Z

public/app/plugins/datasource/loki/configuration/DerivedField.tsx

+              />
+            }
+          >
+            <Select


Select is too small for the "Regex in log line" option

Also, I think the UI would be better if we had Name Type Regex/Label - to make type and regex/label more connected.

public/app/plugins/datasource/loki/configuration/DerivedField.tsx

ivanahuckova · 2023-11-14T10:42:42Z

public/app/plugins/datasource/loki/configuration/DerivedFields.tsx

-              const newDerivedFields = [...fields, { name: '', matcherRegex: '', urlDisplayLabel: '', url: '' }];
+              const newDerivedFields = [
+                ...fields,
+                { name: '', matcherRegex: '', urlDisplayLabel: '', url: '', enableNameMatcher: false },


Should this be labelMatcher: 'true'?

Hmm, not sure, because that would technically break the default from "regex" to "label" and it might not be what users expect?

public/app/plugins/datasource/loki/configuration/DerivedField.tsx

public/app/plugins/datasource/loki/getDerivedFields.ts

ivanahuckova

Nice job, LGTM! 🚀

Thank you so much for your contribution @5cat . This is going to be EXTREMLY useful for many users! Really appreciate your contribution. 🧡

svennergr · 2023-11-14T14:07:23Z

Also a big thank you from my side, @5cat. As you probably have noticed we decided to release this without a feature toggle, such that every Grafana user can make use of your great feature! Hope to see further contributions from you 😊

5cat · 2023-11-14T15:35:01Z

Thank you @svennergr and @ivanahuckova for your amazing support. I'm very happy and excited to use this feature.

javiplx · 2023-12-29T13:41:45Z

Would this work with loki's structured metadata or just with real labels?

ivanahuckova · 2024-01-02T10:09:34Z

Would this work with loki's structured metadata or just with real labels?

Yes, this will work with loki's structured metadata.

5cat requested review from grafanabot and a team as code owners October 7, 2023 08:01

grafana-pr-automation bot added type/docs area/backend area/frontend datasource/Loki pr/external This PR is from external contributor labels Oct 7, 2023

5cat force-pushed the 73598-loki-derived-field-from-detected-fields branch from 8232c69 to 7c5ee01 Compare October 7, 2023 08:04

gtk-grafana assigned svennergr Oct 23, 2023

svennergr added the no-backport Skip backport of PR label Nov 10, 2023

Merge remote-tracking branch 'origin/main' into 73598-loki-derived-fi…

3f0f22f

…eld-from-detected-fields

svennergr requested review from svennergr and removed request for a team and grafanabot November 10, 2023 15:29

svennergr added 2 commits November 14, 2023 07:32

improve settings page to have a fieldType

3ea5976

improve derived fields getter to use matcherRegex

b679893

svennergr changed the title ~~Plugin: Deriving fields by name from parsed loki logs~~ Loki: Option to add derived fields based on labels Nov 14, 2023

svennergr added the add to changelog label Nov 14, 2023

fix failing test

7b1c8f8

svennergr added this to the 10.4.x milestone Nov 14, 2023

rename feature toggle to lokiDerivedFieldsFromLabels

9e086a3

svennergr removed their request for review November 14, 2023 10:05

ivanahuckova reviewed Nov 14, 2023

View reviewed changes

svennergr added 3 commits November 14, 2023 13:00

added suggestions from review

2406223

add empty config object

568dcf4

remove feature flag

e56f06d

svennergr requested a review from ivanahuckova November 14, 2023 12:12

svennergr added 2 commits November 14, 2023 13:33

fix width of select

e3406d2

default to regex derived field

d3e6c98

ivanahuckova approved these changes Nov 14, 2023

View reviewed changes

fix failing test

de20087

svennergr merged commit 53758ad into grafana:main Nov 14, 2023
14 checks passed

svennergr mentioned this pull request Nov 14, 2023

Loki: Add documentation for derived fields based on labels #78120

Closed

5cat mentioned this pull request Nov 14, 2023

Loki: fix missing entries when there arent any derived fields #78123

Merged

3 tasks

svennergr modified the milestones: 10.4.x, 10.3.x Nov 14, 2023

Samreay mentioned this pull request Dec 12, 2023

Derived fields in combination with line_format grafana/loki#9209

Closed

aangelisc modified the milestones: 10.3.x, 10.2.3 Dec 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Loki: Option to add derived fields based on labels #76162

Loki: Option to add derived fields based on labels #76162

5cat commented Oct 7, 2023 •

edited by svennergr

svennergr commented Oct 9, 2023

5cat commented Oct 9, 2023

gtk-grafana commented Oct 23, 2023

jseiser commented Oct 23, 2023

ivanahuckova commented Oct 24, 2023

JakeCooper commented Nov 7, 2023

JakeCooper commented Nov 7, 2023

ivanahuckova commented Nov 7, 2023 •

edited

svennergr commented Nov 10, 2023

ivanahuckova left a comment

ivanahuckova Nov 14, 2023

ivanahuckova Nov 14, 2023

ivanahuckova Nov 14, 2023

svennergr Nov 14, 2023

ivanahuckova left a comment

svennergr commented Nov 14, 2023

5cat commented Nov 14, 2023

javiplx commented Dec 29, 2023

ivanahuckova commented Jan 2, 2024

Loki: Option to add derived fields based on labels #76162

Loki: Option to add derived fields based on labels #76162

Conversation

5cat commented Oct 7, 2023 • edited by svennergr

svennergr commented Oct 9, 2023

5cat commented Oct 9, 2023

gtk-grafana commented Oct 23, 2023

jseiser commented Oct 23, 2023

ivanahuckova commented Oct 24, 2023

JakeCooper commented Nov 7, 2023

JakeCooper commented Nov 7, 2023

ivanahuckova commented Nov 7, 2023 • edited

svennergr commented Nov 10, 2023

ivanahuckova left a comment

Choose a reason for hiding this comment

ivanahuckova Nov 14, 2023

Choose a reason for hiding this comment

ivanahuckova Nov 14, 2023

Choose a reason for hiding this comment

ivanahuckova Nov 14, 2023

Choose a reason for hiding this comment

svennergr Nov 14, 2023

Choose a reason for hiding this comment

ivanahuckova left a comment

Choose a reason for hiding this comment

svennergr commented Nov 14, 2023

5cat commented Nov 14, 2023

javiplx commented Dec 29, 2023

ivanahuckova commented Jan 2, 2024

5cat commented Oct 7, 2023 •

edited by svennergr

ivanahuckova commented Nov 7, 2023 •

edited