[nri-bundle:5.0.26] Logs not parsing as CRI but parsing as Docker

[JPrice904]

Logs not parsing as CRI

Upgraded the NRI Bundle from 4.9.3 -> 5.0.26 and observed attributes that were previously available no longer available to filter on and log messages that were once being parsed with the CRI configuration were now appearing to be parsed with the Docker configuration.

Version of Helm and Kubernetes

Helm chart installed via ArgoCD running on OpenShift 4.11

Which chart?

nri-bundle 5.0.26 which uses newrelic-logging 1.18.1

What happened?

After the upgrade the message field for springboot applications was now unfiltered and one long json message.

What you expected to happen?

I was expecting the upgrade to make no large changes to the logging format or for docker to truly be the fallback option and for CRI to remain the desired parsing format.

Anything else we need to know?

I am maintaining the nri-bundle version and deploying it with ArgoCD and Kustomization. I was able to patch the environment variable utilizing patches with kustomization so that cri was the only value in the LOG_PARSER env variable. Once that was done logs appeared to be normal again with additional attributes and json parsing fixed.

Kustomization file

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ./namespace.yaml
  - ./scc.yaml

helmCharts:
  - name: nri-bundle
    repo: https://helm-charts.newrelic.com
    version: "5.0.26"
    releaseName: newrelic-bundle
    namespace: newrelic
    valuesFile: values.yaml

patches:
  - target:
      kind: DaemonSet
      name: newrelic-bundle-newrelic-logging
    patch: |-
      - op: replace
        path: /spec/template/spec/containers/0/env/5
        value:
          name: LOG_PARSER
          value: cri

[jsubirat]

Hi @JPrice904 , thank you for informing us of this. Are you perhaps overriding the fluentBit.config.inputs value in values.yml? The INPUT block that we now generate use the multiline.parser instead of parser, which accepts multiple parsers and attempts each one in order. This ensures that the parsing is done even if the value of fluentBit.criEnabled is wrong. But if you override the fluentBit.config.inputs block and are still using parser instead of multiline.parser, you'll face the problem you described in the ticket.

[jsubirat]

Hi @JPrice904 , we proceed to close this ticket due to inactivity. Please come back to us if you experience any other issue. Thanks a lot for your feedback!

[khauser]

Hi @jsubirat,

I do have the same problem and I already tried using extraInputs, but that doesn't help. Logs are still not parsed after update to nri-bundle 5.0.46.

Here is our newrelic yaml file:

global:
  lowDataMode: true
newrelic-infrastructure:
  privileged: true

ksm:
  enabled: true
  podAnnotations:
    fluentbit.io/exclude: "true"
kubeEvents:
  enabled: true
  podAnnotations:
    fluentbit.io/exclude: "true"
newrelic-prometheus-agent:
  enabled: true
  podAnnotations:
    fluentbit.io/exclude: "true"
  lowDataMode: true
  config:
    kubernetes:
      integrations_filter:
        enabled: false
logging:
  enabled: true
newrelic-logging:
  podAnnotations:
    fluentbit.io/exclude: "true"
  lowDataMode: true
  fluentBit:
    criEnabled: true
    k8sLoggingExclude: "On"
    config:
      extraInputs: |
        [INPUT]
            Name              tail
            Tag               kube.*
            Path              ${PATH}
            Exclude_Path      /var/log/containers/*_kube-system_*.log, /var/log/containers/*_flux-system_*.log, /var/log/containers/*_newrelic_*.log
            Parser            ${LOG_PARSER}
            DB                ${FB_DB}
            Mem_Buf_Limit     7MB
            Skip_Long_Lines   On
            Refresh_Interval  10
            Buffer_Chunk_Size 64k
            Buffer_Max_Size   64k

What could be the problem here?

Best,
Karsten