Improve Level detection

[cyriltovena]

We've seen case where the log would contains Warn as the level and an error message but it would still eventually end up as error because we don't properly search for Warn with a first capital level. This should be the very least.

We might want to investigate to support logfmt and json level, severity keys to make this more robust.

cc @shantanualsi @sandeepsukhani

[cyriltovena]

We should also consider using another field like detected_level

[bentonam]

I am working on Alloy modules for log processing porting what was done for module v1 in agent to alloy modules v2 this is what i'm currently doing to detect / default a log-level if it helps.

https://github.com/grafana/alloy-modules/blob/log-modules/modules/utils/logs/log-levels.alloy

[jtackaberry]

Looking forward to the upcoming improvements. Currently we have some JSON logs containing "level":"INFO" that are being misdetected as debug. I understand it'll never be as sophisticated as what @bentonam is doing on the Alloy side, but getting level detection of common formats for free is a nice value-add.

We should also consider using another field like detected_level

If so, could be configurable?

[YevhenLodovyi]

@cyriltovena Hi, sorry for troubling. I am using fluent-bit to send logs to loki 3.0.0. I decided no try non-released version of loki(due to fixes in loki-bloom feature), but noticed that I do not have level label anymore and logs are gray in logs volume.

just in case i have enabled log level discovery:

discover_log_levels: true
allow_structured_metadata: true

version: grafana/loki:main-fa2c789'

Is it expected?

[jtackaberry]

I see Loki 3.1 just landed and indeed level has been renamed to detected_level.

Is there going to be a way to reconfigure this to level? Users have gotten accustomed to it, and also Grafana uses level for the log volume histogram.

Should I open a separate issue?

[YevhenLodovyi]

Is there going to be a way to reconfigure this to level? Users have gotten accustomed to it, and also Grafana uses level for the log volume histogram.

As far as i can see there is a task in grafana: grafana/grafana#87564

To be honest it is a bit disappoint that it was changed in loki, but grafana code was not changed accordingly. I have to wait for grafana/grafana#87564 and only after that i can update to 3.1 version.

[cyriltovena]

@svennergr can we do something for Grafana to fallback on detected level ?

[diranged]

Since this is sort of a breaking change - is there a way to push a feature to revert this optionally, until Grafana is updated?

[jtackaberry]

I'd love the label name to be configurable myself. Even though I completely understand the it was changed, I'm partial to level instead of detected_level from both an aesthetic perspective as well as, more importantly, a user familiarity perspective.

AFAICT it's just a single line change so I'm almost tempted to do our own internal build, but it's hard to claim that's sane from a long-term support perspective.

I'd be happy to submit a PR if someone from Grafana Labs could assure me that wouldn't be wasted time.

[runenilsenoe]

1up on this