auto_multi_line_extra_patterns is not supported #570

sullerandras · 2022-03-16T09:42:22Z

The logs_config.auto_multi_line_extra_patterns option is described in this doc but it looks like it's not supported by the helm chart.
Would it be possible to add support for this?

The text was updated successfully, but these errors were encountered:

clamoriniere · 2022-03-21T08:50:04Z

Hi @sullerandras

Is it difficult to add an option in the chart for each possible option in the agent.
When an option is not available the recommended solution is to add it thank to the environment variable.

it the values.yaml file

datadog:
  env:
  - name: DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION
    value: "true"

please let me know if it solve the log agent configuration issue.

vrej-abramian · 2022-03-21T09:15:09Z

Hi @clamoriniere

Is it difficult to add an option in the chart for each possible option in the agent. When an option is not available the recommended solution is to add it thank to the environment variable.

it the values.yaml file
datadog:
  env:
  - name: DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION
    value: "true"
please let me know if it solve the log agent configuration issue.

I have used this parameter and value just as you mentioned, this helped to enable the auto detection of multiline logs with the built-in patterns.

For my case I need logs_config.auto_multi_line_extra_patterns to work as expected, to define specific patterns to be detected (outof the box) for some multiline logs. I have tested logs_config.auto_multi_line_extra_patterns for my multiline logs but it did not work (seems has no impact at all).

P.S. Using datadog agent deployed by helm (chart version: datadog-2.30.16).

olenirwing · 2022-03-21T13:51:34Z

@sullerandras did you get it to work by setting DD_LOGS_CONFIG_AUTO_MULTI_LINE_EXTRA_PATTERNS env variable? Not working for me.

clamoriniere · 2022-03-22T20:19:28Z

Hi @sullerandras
Indeed you can set the logs_config.auto_multi_line_extra_patterns thanks to env variable. If you want to add several patterns it is possible to do using the " " separator between the patterns.

vrej-abramian · 2022-03-23T08:21:56Z

Hi @clamoriniere ,
Would you please help?
When I have set DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION=true by setting the following in the helm values.yaml file:

# Under datadog:
  env:
    - name: "DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION"
      value: "true"
... omitted lines ...
  logs_config:
    auto_multi_line_detection: true

Unfortunately I have java multiline logs like starts with 23-03-2022 blah-blah.. but these logs next lines are not submitted as a single log :(

I've tried to add the EXTRA_PATTERNS as ENV variables as you mentioned (see bellow code) but from inside the container the I got MULTI_LINE_DETECTION value changed to false: DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION=false

# Under datadog:
  env:
    - name: "DD_LOGS_CONFIG_AUTO_MULTI_LINE_EXTRA_PATTERNS"
      value: ^23-03.*  ### also tried with quoted "^23-03.*"
    - name: "DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION"
      value: "true"  ### also tried without quotes
... omitted lines ...
  logs_config:
    auto_multi_line_detection: true

clamoriniere · 2022-03-23T08:48:31Z

Hi @vrej-abramian

I was able add the extra patterns as env var thanks to the single quote: ``

agents:
  containers:
    agent:
      env: # to only set the env var to the container responsible of logs.
      - name: DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION
        value: "true"
      - name: DD_LOGS_CONFIG_AUTO_MULTI_LINE_EXTRA_PATTERNS
        value: '^23-03.*'

the agent status output:

$ k exec -it dd-datadog-2x65r -c agent -- agent config
...
logs_config:
  ...
  auto_multi_line_default_match_threshold: 0.48
  auto_multi_line_default_match_timeout: 30
  auto_multi_line_default_sample_size: 500
  auto_multi_line_detection: true
  auto_multi_line_extra_patterns:
  - ^23-03.*

vrej-abramian · 2022-03-23T11:23:40Z

Hi @vrej-abramian

I was able add the extra patterns as env var thanks to the single quote: ``

agents:
  containers:
    agent:
      env: # to only set the env var to the container responsible of logs.
      - name: DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION
        value: "true"
      - name: DD_LOGS_CONFIG_AUTO_MULTI_LINE_EXTRA_PATTERNS
        value: '^23-03.*'

the agent status output:

$ k exec -it dd-datadog-2x65r -c agent -- agent config
...
logs_config:
  ...
  auto_multi_line_default_match_threshold: 0.48
  auto_multi_line_default_match_timeout: 30
  auto_multi_line_default_sample_size: 500
  auto_multi_line_detection: true
  auto_multi_line_extra_patterns:
  - ^23-03.*

Thanks @clamoriniere for fast reply,
The DD_LOGS_CONFIG_AUTO_MULTI_LINE_EXTRA_PATTERNS parameter is taken properly but even with or without EXRA_PATTERNS the problem is that from within the container I see auto_multi_line_detection: false so the multiline logs detection not working, any idea how can that possible?

# Under datadog:
  env:
    - name: "DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION"
      value: "true"
... omitted lines ...
  logs_config:
    auto_multi_line_detection: true

vrej-abramian · 2022-03-23T12:03:39Z

Thank you @clamoriniere and every one else here,
I've got the fix, while using DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION env variable, logs_config.auto_multi_line_detection should be commented or removed, by this now I can see from within the container that the auto_multi_line_detection: true is set to true and multiline logs are automatically detected (yet at least) :)

# Under datadog:
  env:
    - name: "DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION"
      value: "true"
... omitted lines ...
##  logs_config:            # Removed
##     auto_multi_line_detection: true   # Removed

clamoriniere · 2022-03-23T12:05:37Z

hi again @vrej-abramian
Sorry about that...
I think if found why you have the issue but not me.
let me explain:
You are setting the env var with datadog.env, but on my side i used agents.containers.agent.env. I also discover that the option datadog.logs.autoMultiLineDetection exist and it is set to false by default, and this option is set thank to the envvar.
But in the Container spec, we first add datadog.env then datadog.logs.autoMultiLineDetection and finally agents.containers.agent.env. In the ContainerSpec.env we can set several time the same env var, but the last one win.

that is why adding DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION in datadog.env was override by the value from datadog.logs.autoMultiLineDetection.

For now what you can do is:

datadog:
  logs:
    enabled: true
    #...
    autoMultiLineDetection: true

agents:
  containers:
    agent:
      env: 
      - name: DD_LOGS_CONFIG_AUTO_MULTI_LINE_EXTRA_PATTERNS
        value: '^23-03.*'

And it should work.
I will fill file a card to improve the logic in the chart. to not add DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION if the value is false.

please let me know if it finally works for you.

vrej-abramian · 2022-03-23T12:45:03Z

    - name: "DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION"
      value: "true"

Hey @clamoriniere,
No worries, thank you for the clarification.
Well lets forget about the DD_LOGS_CONFIG_AUTO_MULTI_LINE_EXTRA_PATTERNS for a moment because auto_detection, detects my logs, I'll use EXTRA_PATTERNS maybe latter if I need to.
But the important update that I can share with you is the :

This did not work ( from within container the multi_line_detection is not enabled `auto_multi_line_detection: false` )

By setting the datadog.logs.autoMultiLineDetection to true.
I have also added DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION to agents.containers.agent.env like the following but that doesn't help.

  env:
    - name: "DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION"
      value: "true"

So I've finally get back to my last working configuration which is as follows:

This worked for me to have `auto_multi_line_detection: true` within the container and multi_line logs detection works without extra patterns configured.

### Under datadog.env config block.
  env:
    - name: "DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION"
      value: "true"
... omitted lines ...
### Under datadog.logs_config block
##  logs_config:            # Removed
##     auto_multi_line_detection: true   # Removed
... omitted lines ...
### under agents.containers.agent.env block
  env: []

UPDATE 1

@clamoriniere

Same configuration that worked in one of our clusters, does not work in another cluster, I have no idea about this (((

Update 1.1

I have tried previously discussed configuration combination, but none of them worked on this second cluster.

Update 1.2

@clamoriniere
I wanted to update that I was using wrong parameter datadog.logs_config.auto_multi_line_detection instead of
datadog.logs.autoMultiLineDetection which you have suggested. Probably I was tired and did not notice the difference between them.
By this my issue is resolved.
So thanks again ;)

sullerandras · 2022-03-26T05:45:09Z

Thanks for all the replies. I set the env variable but i'm not sure what should i set it to. Our application is running in a kubernetes cluster, and we log using JSON. Mostly works fine, however, long json logs are cut at 8192 bytes, and the rest is going on a new line. This must be a line length limit somewhere, i'm not sure where. So basically all log lines start with {, except when it's split into two lines. I tried enabling multi line detection and set DD_LOGS_CONFIG_AUTO_MULTI_LINE_EXTRA_PATTERNS: "{". It worked for the first split log line when i tested this, and i was very happy that i was able to solve this problem. However it never worked for any subsequent tests, it only merged the split log lines from the first occurrence. What am i doing wrong?
Any help is appreciated.

keatsfonam · 2022-04-14T19:02:58Z

With latest 2.32.3:
I have found that setting datadog.logs.autoMultiLineDetection: true is being overriden in the config unless also setting the env var DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION=true when I am using the workaround for DD_LOGS_CONFIG_AUTO_MULTI_LINE_EXTRA_PATTERNS

Setting only one of DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION=true or datadog.logs.autoMultiLineDetection: true results in auto_multi_line_detection: false at runtime for some reason.

vrej-abramian · 2022-04-15T09:26:25Z

With latest 2.32.3: I have found that setting datadog.logs.autoMultiLineDetection: true is being overriden in the config unless also setting the env var DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION=true when I am using the workaround for DD_LOGS_CONFIG_AUTO_MULTI_LINE_EXTRA_PATTERNS

Setting only one of DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION=true or datadog.logs.autoMultiLineDetection: true results in auto_multi_line_detection: false at runtime for some reason.

@keatsfonam good finding, I did not used the DD_LOGS_CONFIG_AUTO_MULTI_LINE_EXTRA_PATTERNS yet, but I'll consider your pointing if some day I need to use extra patterns.
Thanks for sharing.

iahmad-khan · 2022-08-11T09:55:16Z

Hi, I am having problem with how to provide the value field for DD_LOGS_CONFIG_AUTO_MULTI_LINE_EXTRA_PATTERNS in case there are multiple patteerns. thanks.

iahmad-khan · 2022-08-12T04:32:30Z

Also, what should be the patterns for log lines like these?


/go/src/recording/vendor/github.com/go-chi/chi/middleware/recoverer.go:106 +0x15d
github.com/go-chi/chi/middleware.prettyStack.decorateLine({}, {0xc000247985?, 0xc000509800?}, 0x9f?, 0x1?)


/go/src/recording/vendor/github.com/go-chi/chi/middleware/recoverer.go:130 +0x545
github.com/go-chi/chi/middleware.prettyStack.decorateFuncCallLine({}, {0xc000247985, 0x1c}

vrej-abramian · 2022-08-15T09:24:10Z

Also, what should be the patterns for log lines like these?


/go/src/recording/vendor/github.com/go-chi/chi/middleware/recoverer.go:106 +0x15d
github.com/go-chi/chi/middleware.prettyStack.decorateLine({}, {0xc000247985?, 0xc000509800?}, 0x9f?, 0x1?)


/go/src/recording/vendor/github.com/go-chi/chi/middleware/recoverer.go:130 +0x545
github.com/go-chi/chi/middleware.prettyStack.decorateFuncCallLine({}, {0xc000247985, 0x1c}

@iahmad-khan, assuming that all your logs start with /go, and using the relevant environmental variable in the datadog-values.yaml file deployed by helm in K8S environment, have you tried such pattern?

- [/]go

Kind reminder: Please consider using a test environment and test configurations for specific issues/cases.

parasappafluke · 2022-09-20T11:23:10Z

Hi,
I have enabled datadog.log.autoMultiLineDetection: true and the agent config command shows below output

logs_config:
  auto_multi_line_default_match_threshold: 0.48
  auto_multi_line_default_match_timeout: 30
  auto_multi_line_default_sample_size: 500
  auto_multi_line_detection: true

But still the exception logs are not coming in single line for java springboot application.
Date format for logs is yyyy-MM-dd hh:mm:ss.SSS and yyyy-MM-dd hh:mm:ss

agent: 7.37.1-jmx
helm chart: 2.36.0

Thanks
Parasappa

dyleph · 2023-02-11T04:24:30Z

Hi @clamoriniere and @vrej-abramian ,
Currently I have the strange issue after setting DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION env variable to allow the datadog agent to aggregate multi lines logs.

Helm chart v3.3.3
Agent 7.40.1

Here is my scenario with step by step:

Adding autoMultiLineDetection: true in values.yaml

datadog:
     logs:
       enabled: true
       autoMultiLineDetection: true

Then upgrade datadog-agent by helm command.
Result: Logs are not aggregated as single line as expected -> KO

This is my agent config:

 logs_config:
   ...
   auto_multi_line_default_match_threshold: 0.48
   auto_multi_line_default_match_timeout: 30
   auto_multi_line_default_sample_size: 500
   auto_multi_line_detection: false
  ...

Then I try adding DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION: true as an environment parameter under datadog as you guys did, and disabling autoMultiLineDetection: true in the previous (1)

 datadog:
     logs:
       enabled: true
       #autoMultiLineDetection: true
     env:
       - name: DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION
         value: "true"

Then upgrade datadog-agent again.
Result: Logs are aggregated as single line as expected -> OK
I re-configured (1) and (2) in many times to make sure (2) is a right config.

This is my agent config:

 logs_config:
   ...
   auto_multi_line_default_match_threshold: 0.48
   auto_multi_line_default_match_timeout: 30
   auto_multi_line_default_sample_size: 500
   auto_multi_line_detection: true
  ...

Now, when merging this change to main. I start upgrading on other cluster agents on different environments and also re-upgrading on this working cluster agent. But logs are not affected after the upgrade. It's really strange, not sure what's happened. :(

clamoriniere · 2023-02-13T10:58:48Z

hi @dyleph

I was not able to reproduce the issue that you describe.

Could you check when it happens that the value is properly set in helm get values <release-name>

and also in the daemonset resource that with only datadog.logs.autoMultiLineDetection:true set. the envvar DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION is added to the agent container with the value "true"?

thanks 🙇

vrej-abramian · 2023-02-13T12:31:15Z

Hi @dyleph, @clamoriniere,
Well to be honest I wish I could try to reproduce and dig deep in your case, but I won't be able soon, so I'll going to share from one of my experiences, hopping it will be useful.

If you want to get rid of the multiline-logs issue forever, the shortest, best, and official resolution from Datadog (which I tested and satisfied) is to logging-out to JSON format from your Application level if possible.

dyleph · 2023-02-15T08:15:15Z

Hi @clamoriniere and @vrej-abramian,
Thank you for your reply.

ATM, the logs are aggregated as well on datadog agent after I set both configs:
datadog:
logs:
enabled: true
autoMultiLineDetection: true
env:
- name: DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION
value: "true"

About configure JSON logging from the application, I did try by writing consoleLog and fileLog with JSON format.

babbottscott · 2023-12-14T23:32:24Z

Try removing the ^ from your pattern. The code that processes these strings prepends one for you (which feels a bit bad form...).

clamoriniere added the chart/datadog This issue or pull request is related to the datadog chart label Mar 21, 2022

clamoriniere added the waiting-user-feedback label Mar 21, 2022

dogeahill linked a pull request Feb 15, 2024 that will close this issue

Update helm chart to allow automultline extra patterns to be configured #1321

Open

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

auto_multi_line_extra_patterns is not supported #570

auto_multi_line_extra_patterns is not supported #570

sullerandras commented Mar 16, 2022

clamoriniere commented Mar 21, 2022

vrej-abramian commented Mar 21, 2022 •

edited

olenirwing commented Mar 21, 2022 •

edited

clamoriniere commented Mar 22, 2022

vrej-abramian commented Mar 23, 2022 •

edited

clamoriniere commented Mar 23, 2022

vrej-abramian commented Mar 23, 2022

vrej-abramian commented Mar 23, 2022

clamoriniere commented Mar 23, 2022

vrej-abramian commented Mar 23, 2022 •

edited

sullerandras commented Mar 26, 2022

keatsfonam commented Apr 14, 2022 •

edited

vrej-abramian commented Apr 15, 2022

iahmad-khan commented Aug 11, 2022

iahmad-khan commented Aug 12, 2022

vrej-abramian commented Aug 15, 2022

parasappafluke commented Sep 20, 2022 •

edited

dyleph commented Feb 11, 2023 •

edited

clamoriniere commented Feb 13, 2023

vrej-abramian commented Feb 13, 2023 •

edited

dyleph commented Feb 15, 2023 •

edited

babbottscott commented Dec 14, 2023

auto_multi_line_extra_patterns is not supported #570

auto_multi_line_extra_patterns is not supported #570

Comments

sullerandras commented Mar 16, 2022

clamoriniere commented Mar 21, 2022

vrej-abramian commented Mar 21, 2022 • edited

olenirwing commented Mar 21, 2022 • edited

clamoriniere commented Mar 22, 2022

vrej-abramian commented Mar 23, 2022 • edited

clamoriniere commented Mar 23, 2022

vrej-abramian commented Mar 23, 2022

vrej-abramian commented Mar 23, 2022

clamoriniere commented Mar 23, 2022

vrej-abramian commented Mar 23, 2022 • edited

This did not work ( from within container the multi_line_detection is not enabled auto_multi_line_detection: false )

This worked for me to have auto_multi_line_detection: true within the container and multi_line logs detection works without extra patterns configured.

UPDATE 1

Update 1.1

Update 1.2

sullerandras commented Mar 26, 2022

keatsfonam commented Apr 14, 2022 • edited

vrej-abramian commented Apr 15, 2022

iahmad-khan commented Aug 11, 2022

iahmad-khan commented Aug 12, 2022

vrej-abramian commented Aug 15, 2022

parasappafluke commented Sep 20, 2022 • edited

dyleph commented Feb 11, 2023 • edited

clamoriniere commented Feb 13, 2023

vrej-abramian commented Feb 13, 2023 • edited

dyleph commented Feb 15, 2023 • edited

babbottscott commented Dec 14, 2023

vrej-abramian commented Mar 21, 2022 •

edited

olenirwing commented Mar 21, 2022 •

edited

vrej-abramian commented Mar 23, 2022 •

edited

vrej-abramian commented Mar 23, 2022 •

edited

This did not work ( from within container the multi_line_detection is not enabled `auto_multi_line_detection: false` )

This worked for me to have `auto_multi_line_detection: true` within the container and multi_line logs detection works without extra patterns configured.

keatsfonam commented Apr 14, 2022 •

edited

parasappafluke commented Sep 20, 2022 •

edited

dyleph commented Feb 11, 2023 •

edited

vrej-abramian commented Feb 13, 2023 •

edited

dyleph commented Feb 15, 2023 •

edited