[BUG] Failed to perform any bulk index operations: 400 Bad Request:

[lt2865]

Upgraded from Opensearch 2.5.0 to 2.11.0 and since then I've been seeing these errors in Winlogbeat and Filebeat logs for both Linux and Windows and I'm not sure how to tell what pipelines it's affecting or exactly why. I can see it's due to an illegal character but never had this issue before upgrading to 2.11.0. I can see some events are still going into Opensearch via both Filebeat and Winlogbeat on both Linux and Windows but these errors are being spammed in the logs so it appears some events aren't getting it and I'm not able to identify which. I've also not added any new pipelines or index templates since the upgrade so seems to be affecting the ones we currently have.

ERROR [publisher_pipeline_output] pipeline/output.go:180 failed to publish events: 400 Bad Request: {"error":{"root_cause":[{"type":"json_parse_exception","reason":"Illegal character ((CTRL-CHAR, code 31)): only regular white space (\\r, \\n, \\t) is allowed between tokens\n at [Source: (byte[])\"\\u001F�\\u0008\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000��}ko�8����\\u0015��\\u0003t/쬮���\\u0000�Nef\"; line: 1, column: 2]"}],"type":"json_parse_exception","reason":"Illegal character ((CTRL-CHAR, code 31)): only regular white space (\\r, \\n, \\t) is allowed between tokens\n at [Source: (byte[])\"\\u001F�\\u0008\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000��}ko�8����\\u0015��\\u0003t/쬮���\\u0000�Nef\"; line: 1, column: 2]"},"status":400}

ERROR [elasticsearch] elasticsearch/client.go:224 failed to perform any bulk index operations: 400 Bad Request: {"error":{"root_cause":[{"type":"json_parse_exception","reason":"Illegal character ((CTRL-CHAR, code 31)): only regular white space (\\r, \\n, \\t) is allowed between tokens\n at [Source: (byte[])\"\\u001F�\\u0008\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000��X[o�\\u0011~�_A\\u0010]�\\u0005DC7ے��f�\"�n�h�]�"����F"uH�\u0017\u0004��\u0007�$�rl�݃�\u0016�Ùo.��EO8�@5��\u0009?0��\u0006'Xm\u0015�J�5�(�@5q�5��-�]? �O�\u0010;�f5��\u0003N���g������Y\u0005JӪ�\u0009>x�y^�F�?\u0019�}�\u0007v0��k\u0003%\u00139�$�D��k)V,\u0007�\u0013��eR(���;�X+r��jr�\u0019/��i���8�_��@)tmR4KN�m\u0016F4"; line: 1, column: 2]"}],"type":"json_parse_exception","reason":"Illegal character ((CTRL-CHAR, code 31)): only regular white space (\r, \n, \t) is allowed between tokens\n at [Source: (byte[])"\u001F�\u0008\u0000\u0000\u0000\u0000\u0000\u0000��X[o�\u0011~�_A\u0010]�\u0005DC7ے��f�"�n�h�]�\"����F\"uH�\\u0017\\u0004��\\u0007�$�rl�݃�\\u0016�Ùo.��EO8�@5��\\u0009?0��\\u0006'Xm\\u0015�J�5�(�@5Q�5��-�]? �O�\\u0010;�f5��\\u0003N���~g������Y\\u0005JӪ�\\u0009>x�y^�F�?\\u0019�}�\\u0007v0��k\\u0003%\\u00139�$�D��k)V,\\u0007�\\u0013��eR(���;�X+r��jr�\\u0019/��i���8�_��@)tmR4KN�m~\\u0016F4\"; line: 1, column: 2]"},"status":400}

These errors weren't present before the upgrade and I have "compatibility.override_main_response_version" : true applied due to the Filebeat and Winlogbeat versions I'm running (7.10.2). Error looks to be the similar to the one logged at #3484 but the fixes mentioned don't seem to solve the issue. Any insight would be appreciated.

To Reproduce
Upgrade OpenSearch from 2.5 to 2.11

• Filebeat: 7.10.2
• Winlogbeat: 7.10.2
• Opensearch 2.11.0 (Current Version)

Possibly Related
#3484

[dblock]

This is #10802 and was fixed in 2.11.1 that was just released, sorry for the hassle :(

[lt2865]

No problem thanks for confirming. The new minor version seems to have resolved the issue for me!