Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Promtail - config file spilling over to new line #11273

Open
majandres opened this issue Nov 20, 2023 · 0 comments
Open

Promtail - config file spilling over to new line #11273

majandres opened this issue Nov 20, 2023 · 0 comments
Labels
area/helm help wanted We would love help on these issues. Please come help us!

Comments

@majandres
Copy link

majandres commented Nov 20, 2023

Describe the bug
When adding a stage to config.snippets.pipelineStages in helm values, it looks like it's not templated correctly. Some values seems like they are too long and are spilling over to the next line, invalidating the config? I'm trying to capture multiline logs. I know the regex is correct, but the multiline logs are not getting grouped into one when viewing in Grafana.

To Reproduce
Steps to reproduce the behavior:

  1. Started Loki (2.9.2)
  2. Started Promtail (2.9.2) to tail '...'
  3. Query: {} term

Expected behavior
I expect that the pipeline_staqges config will remain when it's evaluated into a config for promtail. In my case, it's affecting the multiline stage, but it seems that this can affect anything that is using a long string value.

Environment:

  • Infrastructure: EKS
  • Deployment tool: helm

Screenshots, Promtail config, or terminal output
This is the helm config for promtail

pipelineStages:
  - cri: {}
  - match:
      selector: '{app="my-app-name"}'
      stages:
      - multiline:
          # Identify timestamps as first line of a multiline block. Enclose the string in single quotes.
          firstline: '^\[\d{4}-\d{2}-\d{2}T\d{1,2}:\d{1,2}:\d{1,2}\.\d{3}\] \[\w+\] \w+ -'
          max_wait_time: 3s
      - regex:
          # Flag (?s:.*) needs to be set for regex stage to capture full traceback log in the extracted map.
          expression: '^(?P<time>\[\d{4}-\d{2}-\d{2}T\d{1,2}:\d{1,2}:\d{1,2}\.\d{3}\]) \[\w+\] \w+ - (?P<message>(?s:.*))$'

Here is an example log
[2023-11-15T22:29:30.537] [INFO] default - Successfully connected to backend.

This is the relevant snippet from promtail's config that is generated

  - job_name: kubernetes-pods
    pipeline_stages:
      - cri: {}
      - match:
          selector: '{app="my-app-name"}'
          stages:
          - multiline:
              firstline: ^\[\d{4}-\d{2}-\d{2}T\d{1,2}:\d{1,2}:\d{1,2}\.\d{3}\] \[\w+\] \w+
                -
              max_wait_time: 3s
          - regex:
              expression: ^(?P<time>\[\d{4}-\d{2}-\d{2}T\d{1,2}:\d{1,2}:\d{1,2}\.\d{3}\])
                \[\w+\] \w+ - (?P<message>(?s:.*))$

Note how the firstline and expression values flow into the next line.

Something is happening when the helm chart is evaluates the values and stuffs the promtail.yml config into a secret... which is loaded into promtail to use as it's config.

@cstyan cstyan added help wanted We would love help on these issues. Please come help us! area/helm labels Dec 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/helm help wanted We would love help on these issues. Please come help us!
Projects
None yet
Development

No branches or pull requests

2 participants