why does fluentd filling logs with backslashes?

[conquerorAlex]

Need use fluentd to collect logs of kubernetes. But fluentd logs are spammed with backslashes like this:
record={"log"=\u003e"\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\", \"stream\"=\u003e\"stdout\", \"docker\"=\u003e{\"container_id\"=\u003e\"3138957a81d0f0479ed285f5cff1ae66a6d8afa8dc976514bb1d8881139d0220\"}, \"kubernetes\"=\u003e{\"container_name\"=\u003e\"fluentd-es\", \"namespace_name\"=\u003e\"kube-system\", \"pod_name\"=\u003e\"fluentd-es-v2.2.0-xhw49\", \"container_image\"=\u003e\"bossjones/fluentd-elasticsearch:v2.2.0\", \"container_image_id\"=\u003e\"docker-"

but also the logs contain hundreds of lines of just pure backslashes with no messages contained.

the config of fluentd as follow:

<filter kubernetes.**>
     @type record_transformer
      enable_ruby true
      <record>
        dockerid ${record["docker"]["container_id"]}
      </record>
</filter>

I found when getting nested field "container_id", the problem happens. However, if just set dockerid ${record["docker"]}, the problem solved. But now I need get the nested field container id. Then how to solve this?
I use the test case from: https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/fluentd-elasticsearch/fluentd-es-configmap.yaml. And added the above record_transformer to get container id from logs in path of /var/log/containers.

[bhargavamin]

I'm having the same issue but while using a tail plugin to read json logs from a file. #2405

I think this is an encoding issue of json logs just check if the logs are properly escaped.

[hvoigt]

Is is possible that your fluentd is reading its own logs? Have a look at #2545 and the FLUENT_CONTAINER_TAIL_EXCLUDE_PATH variable which you can use to exclude fluentd's own logs from being forwarded.

[mickdewald]

I got this issue as well, because I was using containerd instead of docker. I solved it by putting in the following configuration:

- name: FLUENT_CONTAINER_TAIL_PARSER_TYPE
  value: /^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$/

[github-actions]

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

[github-actions]

This issue was automatically closed because of stale in 30 days