fix: Fix log level detection #12651

shantanualsi · 2024-04-17T12:55:37Z

What this PR does / why we need it:
Currently detection of log levels is not truly case insensitive. For eg. "warn" or "WARN" is detected, however "Warn" is not. The implementation uses string.Contains over the entire log line to check for these levels.

However, checking ignoring case would need us to toLower the entire log line which is inefficient. This PR detects the level first -> "level": in case of JSON and level= in case of logfmt and then compares the next 12 characters after that ignoring case.

The benchmarks show similar result on my computer with the fixed version at times few 100 ns faster.

implementation from main branch:

implementation with Fix (This PR):

The PR also supports log level TRACE and FATAL and even 3 letter abbreviations that might be common.

Edit:
Added more improvements after some discussions-

Using logfmt and jsonparsers to parse each log line. After benchmarking, this performs slightly better than the previous implementation
Changed the label from level to detected_level so that it is clear for users
Support "Level", "Severity" and "Lvl" in lowercase, uppercase and Pascal case.

Few caveats:
~~- "level" should still be lowercase or block case ("LEVEL"). Documentation for configuration is updated. This does not support labels like Severity yet.~~

Which issue(s) this PR fixes:
Ref: #12645

Special notes for your reviewer:

Checklist

Reviewed the CONTRIBUTING.md guide (required)
Documentation added
Tests updated
Title matches the required conventional commits format, see here
Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
For Helm chart changes bump the Helm chart version in production/helm/loki/Chart.yaml and update production/helm/loki/CHANGELOG.md and production/helm/loki/README.md. Example PR
If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

cyriltovena · 2024-04-17T13:35:23Z

pkg/distributor/distributor.go

@@ -904,42 +904,46 @@ func extractLogLevelFromLogLine(log string) string {
 	}

 	if firstNonSpaceChar == '{' && lastNonSpaceChar == '}' {
-		if strings.Contains(log, `:"err"`) || strings.Contains(log, `:"ERR"`) ||
-			strings.Contains(log, `:"error"`) || strings.Contains(log, `:"ERROR"`) {
+		levelIndex := strings.Index(log, `"level":`)


problem is that it could be Level, LEVEL, serverity, Serverity.

yeah, i've mentioned it in the caveats in notes.. can we impose this in our documentation? or make this string configurable (case sensetive)?

what if we pass a hint to this function? for example, I think we could change this code to something like the following

firstEntry := stream.Entries[0] firtsStructuredMetadata := logproto.FromLabelAdaptersToLabels(firstEntry.StructuredMetadata) prevTs := firstEntry.Timestamp addLogLevel := validationContext.allowStructuredMetadata && validationContext.discoverLogLevels && !lbs.Has(labelLevel) logLevelField := detectLogLevelFieldFromLogEntry(firstEntry, firtsStructuredMetadata)

this would allow detectLogLEvelFieldFromLogEntry to do a case insensitive search for a bunch of different options (ie. level, severity, etc) since it was only doing it once for the whole stream per push. We could then pass that as a hint to detectLogLevelFromLogEntry, allowing it to short-circuit when passed.

pkg/distributor/distributor.go

cyriltovena

I like the idea, I think we should may be support different field name ?

almost wonder if we should use some fast json and logfmt parser really to grab the key !

shantanualsi · 2024-04-19T10:13:52Z

I found an old issue, a comment on which claims that 3 letter abbreviations for levels (dbg, wrn) are common as well (i haven't seen them yet). I've added support for them too in this PR.

Also, supports "level" or "LEVEL" and updated the documentation.

@cyriltovena If everything else looks okay, we can merge this fix first?

To support different labels such as severity including any other cases of ``level`, I think it should be a part of configuration. We can add support for that in a separate PR.

shantanualsi · 2024-04-19T10:36:07Z

almost wonder if we should use some fast json and logfmt parser really to grab the key !

this is a good idea, i'll try and benchmark this. My hunch is that it's going to be much slower than the current bruteforce way...

cyriltovena · 2024-04-19T13:07:04Z

almost wonder if we should use some fast json and logfmt parser really to grab the key !

this is a good idea, i'll try and benchmark this. My hunch is that it's going to be much slower than the current bruteforce way...

don't know about that I think it can be quite good. I'm specially worried for structured log.

trevorwhitney

What do you think about trying to detect the level field once per stream per push rather than on every line?

trevorwhitney · 2024-04-19T19:59:40Z

pkg/distributor/distributor.go

@@ -904,42 +904,46 @@ func extractLogLevelFromLogLine(log string) string {
 	}

 	if firstNonSpaceChar == '{' && lastNonSpaceChar == '}' {
-		if strings.Contains(log, `:"err"`) || strings.Contains(log, `:"ERR"`) ||
-			strings.Contains(log, `:"error"`) || strings.Contains(log, `:"ERROR"`) {
+		levelIndex := strings.Index(log, `"level":`)


what if we pass a hint to this function? for example, I think we could change this code to something like the following

firstEntry := stream.Entries[0] firtsStructuredMetadata := logproto.FromLabelAdaptersToLabels(firstEntry.StructuredMetadata) prevTs := firstEntry.Timestamp addLogLevel := validationContext.allowStructuredMetadata && validationContext.discoverLogLevels && !lbs.Has(labelLevel) logLevelField := detectLogLevelFieldFromLogEntry(firstEntry, firtsStructuredMetadata)

this would allow detectLogLEvelFieldFromLogEntry to do a case insensitive search for a bunch of different options (ie. level, severity, etc) since it was only doing it once for the whole stream per push. We could then pass that as a hint to detectLogLevelFromLogEntry, allowing it to short-circuit when passed.

cyriltovena · 2024-04-22T07:37:24Z

Actually think it's fine for doing this on every line, I suggest though we make a custom image that runs the detection without applying the change, to test in prod and see the impact.

shantanualsi · 2024-04-22T07:43:55Z

I'm spending some time today using json-parser and logfmt parser and benchmark.

shantanualsi · 2024-04-22T07:47:08Z

is it entirely safe to assume that the label for "LEVEL" will be exactly the same in every line within a stream?

shantanualsi · 2024-04-22T11:05:24Z

Turns out parsing is ~3x slower than the brute force way. (using our inbuilt parser implementations from logql_log)

1st run is using no parsers (original implementation with bugfix). 2nd run makes an attempt to parse using json and logfmt
Check commit

pkg/distributor/distributor.go

cyriltovena · 2024-05-02T15:51:54Z

pkg/distributor/distributor.go

@@ -385,10 +396,10 @@ func (d *Distributor) Push(ctx context.Context, req *logproto.PushRequest) (*log
 				}

 				structuredMetadata := logproto.FromLabelAdaptersToLabels(entry.StructuredMetadata)
-				if addLogLevel && !structuredMetadata.Has(labelLevel) {
+				if addLogLevel && !hasAnyLevelLabels(structuredMetadata) {
 					logLevel := detectLogLevelFromLogEntry(entry, structuredMetadata)
 					entry.StructuredMetadata = append(entry.StructuredMetadata, logproto.LabelAdapter{


should we consider not adding any data if we don't find the level ?

i think we should still keep the detected_level and set it to unknown even if we cannot find it to keep the queries uniform for the user? Might break queries or rules if the label is absent for some log lines at random

Might break queries or rules if the label is absent for some log lines at random.

I can't think of an example, this label shouldn't be in used.

pushed the change to not set detected_level if unknown

shantanualsi · 2024-05-06T07:57:57Z

sorry, ruined the PR by doing pull --rebase. Will try to reflog and revert or raise another PR with changes

Fix log level detection

a9ad857

pull-request-size bot added the size/M label Apr 17, 2024

shantanualsi changed the title ~~Fix log level detection~~ fix: Fix log level detection Apr 17, 2024

Small bugfix

a1d643d

shantanualsi marked this pull request as ready for review April 17, 2024 13:21

shantanualsi requested a review from a team as a code owner April 17, 2024 13:21

cyriltovena reviewed Apr 17, 2024

View reviewed changes

pkg/distributor/distributor.go Outdated Show resolved Hide resolved

cyriltovena reviewed Apr 17, 2024

View reviewed changes

shantanualsi added 4 commits April 18, 2024 12:11

Add support for trace and fatal detection

7d70360

Merge branch 'main' into fix-log-level-detection

cbdc4c7

Merge branch 'main' into fix-log-level-detection

8d2ce13

Support unknown log level and add some tests

00397ee

pull-request-size bot added size/L and removed size/M labels Apr 19, 2024

Add support for LEVEL for discover levels

147233a

github-actions bot added the type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories label Apr 19, 2024

shantanualsi added 2 commits April 19, 2024 15:26

Fix tests

5d33553

Merge branch 'main' into fix-log-level-detection

bc62249

Merge branch 'main' into fix-log-level-detection

9cd72f9

trevorwhitney reviewed Apr 19, 2024

View reviewed changes

shantanualsi added 2 commits April 22, 2024 13:42

Merge branch 'main' into fix-log-level-detection

441586b

Add method to parse both logfmt and json to detect level

979244b

cyriltovena reviewed May 2, 2024

View reviewed changes

pkg/distributor/distributor.go Show resolved Hide resolved

cyriltovena reviewed May 2, 2024

View reviewed changes

pkg/distributor/distributor.go Show resolved Hide resolved

cyriltovena reviewed May 2, 2024

View reviewed changes

shantanualsi requested review from periklis and xperimental as code owners May 6, 2024 07:56

github-actions bot added sig/operator area/helm labels May 6, 2024

pull-request-size bot added size/XXL and removed size/L labels May 6, 2024

shantanualsi marked this pull request as draft May 6, 2024 07:56

Add detected_level from label or existing structured metadata

9b1b91c

shantanualsi force-pushed the fix-log-level-detection branch from 9c7ab15 to 9b1b91c Compare May 6, 2024 08:13

pull-request-size bot added size/L and removed size/XXL labels May 6, 2024

github-actions bot removed sig/operator area/helm labels May 6, 2024

Merge branch 'main' into fix-log-level-detection

7795d81

shantanualsi marked this pull request as ready for review May 6, 2024 08:14

Do not populate detected_level when level is unknown

bae53e1

This was referenced May 6, 2024

Replace the new structured metadata level by detected_level grafana/explore-logs#293

Closed

Default to no level labels when not found grafana/explore-logs#294

Closed

cyriltovena merged commit 6904a65 into main May 6, 2024
59 checks passed

cyriltovena deleted the fix-log-level-detection branch May 6, 2024 11:44

cyriltovena mentioned this pull request May 6, 2024

Support new detected_level label grafana/explore-logs#297

Closed

svennergr mentioned this pull request May 6, 2024

[queries] change level aggregation to detected_level grafana/explore-logs#299

Closed

This was referenced May 13, 2024

chore(k202): release 3.1.0 #12945

Open

chore(k203): release 3.1.0 #12988

Open

This was referenced May 27, 2024

chore(k204): release 3.1.0 #13037

Open

chore(k205): release 3.1.0 #13102

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: Fix log level detection #12651

fix: Fix log level detection #12651

shantanualsi commented Apr 17, 2024 •

edited

cyriltovena Apr 17, 2024

shantanualsi Apr 17, 2024

trevorwhitney Apr 19, 2024

cyriltovena left a comment

shantanualsi commented Apr 19, 2024

shantanualsi commented Apr 19, 2024

cyriltovena commented Apr 19, 2024

trevorwhitney left a comment

trevorwhitney Apr 19, 2024

cyriltovena commented Apr 22, 2024

shantanualsi commented Apr 22, 2024 •

edited

shantanualsi commented Apr 22, 2024

shantanualsi commented Apr 22, 2024 •

edited

cyriltovena May 2, 2024

shantanualsi May 6, 2024 •

edited

cyriltovena May 6, 2024

shantanualsi May 6, 2024

shantanualsi commented May 6, 2024

fix: Fix log level detection #12651

fix: Fix log level detection #12651

Conversation

shantanualsi commented Apr 17, 2024 • edited

implementation from main branch:

implementation with Fix (This PR):

cyriltovena Apr 17, 2024

Choose a reason for hiding this comment

shantanualsi Apr 17, 2024

Choose a reason for hiding this comment

trevorwhitney Apr 19, 2024

Choose a reason for hiding this comment

cyriltovena left a comment

Choose a reason for hiding this comment

shantanualsi commented Apr 19, 2024

shantanualsi commented Apr 19, 2024

cyriltovena commented Apr 19, 2024

trevorwhitney left a comment

Choose a reason for hiding this comment

trevorwhitney Apr 19, 2024

Choose a reason for hiding this comment

cyriltovena commented Apr 22, 2024

shantanualsi commented Apr 22, 2024 • edited

shantanualsi commented Apr 22, 2024

shantanualsi commented Apr 22, 2024 • edited

cyriltovena May 2, 2024

Choose a reason for hiding this comment

shantanualsi May 6, 2024 • edited

Choose a reason for hiding this comment

cyriltovena May 6, 2024

Choose a reason for hiding this comment

shantanualsi May 6, 2024

Choose a reason for hiding this comment

shantanualsi commented May 6, 2024

shantanualsi commented Apr 17, 2024 •

edited

shantanualsi commented Apr 22, 2024 •

edited

shantanualsi commented Apr 22, 2024 •

edited

shantanualsi May 6, 2024 •

edited