Loki: Add backend functionality to parse structured metadata from Loki #77361
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Backend code coverage report for PR #77361
|
|
Frontend code coverage report for PR #77361 |
matyax
approved these changes
Nov 2, 2023
pkg/tsdb/loki/api.go
Outdated
| @@ -30,6 +30,8 @@ type LokiAPI struct { | |||
| url string | |||
| log log.Logger | |||
| tracer tracing.Tracer | |||
|
|
|||
There was a problem hiding this comment.
Does this new line has any particular purpose?
pkg/tsdb/loki/api.go
Outdated
| } | ||
|
|
||
| func makeDataRequest(ctx context.Context, lokiDsUrl string, query lokiQuery) (*http.Request, error) { | ||
| func makeDataRequest(ctx context.Context, lokiDsUrl string, query lokiQuery, requestStructuredMetadata bool) (*http.Request, error) { |
There was a problem hiding this comment.
No need to change anything, but given the flag we set calls to "categorize labels", maybe this could have been called categorizeLabels too?
pkg/util/converter/prom.go
Outdated
| return rspErr(err) | ||
| } | ||
|
|
||
| plabelsMap, clabelsMap, err := readCategorizedStreamField(iter) |
There was a problem hiding this comment.
plabel and clabel are just too bad to read. Could these names be improved so they are easier to understand?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is this feature?
Copy of #76350.
Loki has recently added functionality for structured metadata. This PR adds the correct parsing of structured metadata to Grafana's data source.
When the
lokiStructuredMetadatafeature flag is set, the datasource will add aX-Loki-Response-Encoding-FlagsHTTP header with the value ofcategorize-labels. That will tell Loki to respond with structured metadata attached to each logline. Grafana's parser will parse the response into a new DataFrame field calledlabelTypes. That field stores a JSON object for each log line containing the label name and a single character to distinguish between label types:Iwill be indexed labelsPwill be parsed labelsSwill be structured metadataWhy do we need this feature?
To support structured metadata.
Special notes for your reviewer:
go build ./cmd/loki./loki -config.file=../grafana/devenv/docker/blocks/loki/loki-config.yamlgrafanadirectory:node devenv/docker/blocks/loki/data/data.js http://localhost:3100lokiStructuredMetadatafeature flag enabled.[ { "schema": { "refId": "A", "meta": { "typeVersion": [ 0, 0 ], "custom": { "frameType": "LabeledTimeValues", "lokiQueryStatKey": "Summary: total bytes processed" }, "stats": [ { "displayName": "Summary: bytes processed per second", "unit": "Bps", "value": 493885 }, { "displayName": "Summary: lines processed per second", "value": 2602 }, { "displayName": "Summary: total bytes processed", "unit": "decbytes", "value": 1708 }, { "displayName": "Summary: total lines processed", "value": 9 }, { "displayName": "Summary: exec time", "unit": "s", "value": 0.003458 }, { "displayName": "Ingester: total reached", "value": 1 }, { "displayName": "Ingester: total chunks matched", "value": 5 }, { "displayName": "Ingester: total batches", "value": 1 }, { "displayName": "Ingester: total lines sent", "value": 1 }, { "displayName": "Ingester: head chunk bytes", "unit": "decbytes", "value": 0 }, { "displayName": "Ingester: head chunk lines", "value": 0 }, { "displayName": "Ingester: decompressed bytes", "unit": "decbytes", "value": 0 }, { "displayName": "Ingester: decompressed lines", "value": 0 }, { "displayName": "Ingester: compressed bytes", "unit": "decbytes", "value": 0 }, { "displayName": "Ingester: total duplicates", "value": 0 } ], "executedQueryString": "Expr: {age=\"new\", place=\"moon\"}", "preferredVisualisationType": "logs", "limit": 1, "searchWords": [] }, "fields": [ { "name": "labels", "type": "other", "typeInfo": { "frame": "json.RawMessage" }, "config": {} }, { "name": "Time", "type": "time", "typeInfo": { "frame": "time.Time" }, "config": {} }, { "name": "Line", "type": "string", "typeInfo": { "frame": "string" }, "config": {} }, { "name": "tsNs", "type": "string", "typeInfo": { "frame": "string" }, "config": {} }, { "name": "labelTypes", "type": "other", "typeInfo": { "frame": "json.RawMessage" }, "config": { "custom": { "hidden": true } } }, { "name": "id", "type": "string", "typeInfo": { "frame": "string" }, "config": {} }, { "name": "test", "type": "string", "config": { "links": [ { "title": "test", "url": "http://test.com" } ] } } ] }, "data": { "values": [ [ { "source": "data", "nonIndexed": "value", "age": "new", "instance": "server\\1", "job": "\"grafana/data\"", "place": "moon", "re": "one.two$three^four" } ], [ 1697027135016 ], [ "{\"_entry\":\"log text with ANSI \\u001b[31mpart of the text\\u001b[0m [272230661]\",\"counter\":\"10034\",\"float\":\"73.589\",\"wave\":-0.9510565162952045,\"label\":\"val2\",\"level\":\"error\"}" ], [ "1697027135016798346" ], [ { "nonIndexed": "S", "place": "I", "re": "I", "source": "I", "age": "I", "instance": "I", "job": "I" } ], [ "1697027135016798346_daa2f713" ], [ "73.589" ] ], "nanos": [ null, [ 798346 ], null, null, null, null, null ] } } ]