Loki does not recognize log level ERRO

[alextes]

Is your feature request related to a problem? Please describe.
It seems some applications like to make their log levels all four letters, meaning "warn", "info" are fine but "debug" and "error" break.
Describe the solution you'd like
Support these levels or custom level mapping somehow so log levels are appropriately recognized by Loki.

Describe alternatives you've considered
Match on the literal string level: ERRO then add a static label.

Additional context

Thanks for a great piece of monitoring!

[alextes]

Using a replace stage might also work here I just realized! In fact, it may feel adequate, I'll try.

[alextes]

Yup, adding a stage like this works quite well:

- replace:
    expression: "(ERRO)"
    source: level
    replace: "ERROR"

It's good enough for me. Still, leaving this open. I'm noticing quite a few logging libraries, I think even all of golang likes to do things this way. It may still be worth supporting. If maintainers don't feel the same, feel free to close (:

[MichelHollands]

Thanks for reporting this and the solution you found. I'm going to close it as we can't accommodate every logging library's idiosyncrasies.

[niluan304]

@alextes
Excuse me, I hope to receive your help.

My Except

I hope to see level=debug in grafana instead of unknown

What I did

I try to add a stage , but it didn't work:

- replace:
    expression: "(?P<level>DEBU)"
    source: level
    replace: "debug"

# or 
- replace:
    expression: "(DEBU)"
    source: level
    replace: "debug"    

Other Stage

I successfully modify label msg:

    - replace:
        expression: '(default)'
        source: msg
        replace: "~MySQL~"

    - replace:
        expression: '(http://127.0.0.1)'
        source: msg
        replace: "http://192.168.32.132"

Logs Info

Here are some logs:

2023-03-21 09:34:53.998 [INFO] swagger ui is serving at address: http://127.0.0.1:8180/swagger/
2023-03-21 09:34:54.001 [DEBU] {f42c68674a4b4e178559bb682711750b} [ 11 ms] [default] [exam_dev_pi] [rows:1  ] SELECT * FROM 

promtail config.yml more info:

scrape_configs:

  - job_name: job
    static_configs:
    - labels:
        __path__: /var/log

    pipeline_stages:

    - multiline:
        firstline: '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d{3}'
        max_wait_time: 3s
    - regex:
        expression: '(?P<time>^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d{3}) \[(?P<level>[A-Z]+)\]( {(?P<trace>\w*)}){0,1} (?P<msg>(.|\n)*$)'

    - labels:
        time:
        level:
        trace:

    # 1st try
    - replace:
        expression: "(?P<level>DEBU)"
        source: level
        replace: "debug"

    # 2nd try
    - replace:
        expression: "(DEBU)"
        source: level
        replace: "debug"

    - timestamp:
        source: time
        format: '2006-01-02 15:04:05.000'
        
    - output:
        source: msg

I'm confused, I'd appreciate it if I could get some advice.

[alextes]

@niluan304

Here's a log line and the config I used to parse it. I hope that helps. I'm not spotting straight from your config posted what is wrong with it but I hope this helps.

{"msg":"Error updating deposit contract cache","level":"ERRO","ts":"2023-03-21T21:22:09.551785Z","service":"deposit_contract_rpc","error":"Invalid endpoint state: RequestFailed(\"eth_chainId call failed Reqwest(reqwest::Error { kind: Request, url: Url { scheme: \\\"http\\\", cannot_be_a_base: false, username: \\\"\\\", password: None, host: Some(Domain(\\\"localhost\\\")), port: Some(8551), path: \\\"/\\\", query: None, fragment: None }, source: hyper::Error(Connect, ConnectError(\\\"tcp connect error\\\", Os { code: 61, kind: ConnectionRefused, message: \\\"Connection refused\\\" })) })\")","retry_millis":60000}

        pipeline_stages:
          - cri: {}
          - match:
              selector: '{app="geth-lighthouse-pair", container="lighthouse"}'
              stages:
              - json: 
                  expressions:
                    level: level
                    message: msg
                    timestamp: ts
              - replace:
                  expression: "(ERRO)"
                  source: level
                  replace: "ERROR"
              - replace:
                  expression: "(DEBU)"
                  source: level
                  replace: "DEBUG"
              - labels:
                  level:
                  message:
              - timestamp:
                  source: timestamp
                  format: RFC3339Nano

[niluan304]

@alextes
Thank you, I'm trying. Maybe some bug happen.

[JohannesEH]

I know it's closed and you cant accommodate everything but it would've been nice with 3-letter abbreviations as they're fairly common: INF, ERR, DBG, TRC, WRN