Coming full of backslashes in logs

[yphanikumar1995]

\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"},"message":"dump an error event: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error=\"400 - Rejected by Elasticsearch\" location=nil

[arascheuski]

Also affected with this issue.

Environment details:

• Kubernetes plarform: EKS v.1.21 / AKS v.1.20 / AKS v.1.21
• kokuwa/fluentd-elasticsearch hel chart version : 13.2.0
• elastic/elasticsearch helm chart version : 7.16.3

Issue reproduces spontaneously. It is not possibly to catch the reason of this behavior.

Log example:

\\\\\\\\\\\u001c\xCD\xFCL\\\\\\\

[warn]: dump an error event: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch" location=nil tag="kubernetes.var.log.containers.fluentd-fluentd-elasticsearch-hgv2h_logging_fluentd-fluentd-elasticsearch-cc4a0f9583b5900ae037ab8972aa2859ce5b357bd44c4506c88a42dccd440930.log"

Looks like as a good idea / remediation is to suppress this warning level message.

[jhuebner79]

I had the same problem and resorted to stop shipping fluentd's own logs for now.