[tcp]: Line Delimiter input text field does not work for control characters like \r

[andrewkroh]

Integration Name

Custom TCP Logs [tcp]

Dataset Name

tcp.generic

Integration Version

1.19.1

Agent Version

8.15.0

Agent Output Type

elasticsearch

Elasticsearch Version

8.15.0

OS Version and Architecture

Linux

Software/API Version

No response

Error Message

From Filebeat log output:

{"log.level":"debug","@timestamp":"2024-08-07T20:12:49.340Z","message":"Client error","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"tcp-default","type":"tcp"},"log":{"source":"tcp-default"},"log.logger":"tcp","log.origin":{"file.line":202,"file.name":"streaming/listener.go","function":"github.com/elastic/beats/v7/filebeat/inputsource/common/streaming.(*Listener).handleConnection"},"remote_address":"127.0.0.1:57109", "error":{"message":"bufio.Scanner: token too long"},"ecs.version":"1.6.0","service.name":"filebeat","address":"localhost:2107","ecs.version":"1.6.0"}

This is an indicator that the delimiter is not working correctly and it is over running the bufio scanner.

Event Original

No response

What did you do?

Configured the "Line Delimiter" option with \r with the intention of using a carriage return as the delimiter.

POST kbn:/api/fleet/package_policies
{
  "policy_ids": [
    "6a93bbd0-6234-49f3-b0d4-50a0c762458a"
  ],
  "package": {
    "name": "tcp",
    "version": "1.19.1"
  },
  "name": "tcp-1",
  "description": "",
  "namespace": "",
  "inputs": {
    "tcp-tcp": {
      "enabled": true,
      "streams": {
        "tcp.generic": {
          "enabled": true,
          "vars": {
            "listen_address": "localhost",
            "listen_port": 2107,
            "data_stream.dataset": "tcp.generic",
            "line_delimiter": "\\r",
            "tags": [],
            "syslog_options": "field: message\n#format: auto\n#timezone: Local\n",
            "preserve_original_event": false,
            "ssl": "#certificate: |\n#    -----BEGIN CERTIFICATE-----\n#    ...\n#    -----END CERTIFICATE-----\n#key: |\n#    -----BEGIN PRIVATE KEY-----\n#    ...\n#    -----END PRIVATE KEY-----\n",
            "custom": ""
          }
        }
      }
    }
  }
}

What did you see?

The package policy was generated with "line_delimiter": "\\r", which is not a carriage return, but instead a \ followed by r (or 0x5c72).

What did you expect to see?

I expected to see the package policy request generated with "line_delimiter": "\r", which would be 0x0d.

Anything else?

The workaround is to use the custom configuration YAML option to specify the line_delimiter instead of using the "Line Delimiter" input text field.

line_delimiter: "\r"

[elasticmachine]

Pinging @elastic/sec-deployment-and-devices (Team:Security-Deployment and Devices)

[taylor-swanson]

@andrewkroh, do you think this is an issue within Kibana/Fleet itself, or how we are building the agent file in the integration?

[andrewkroh]

I think we should try to make the UI work for this use case through collaboration with @elastic/fleet.

The other problem is properly building the YAML representation (you can use the API to set the line_delimiter to \n today, but it still won't work b/c the agent policy YAML is wrong). My preferred approach is to use {{{ to_json line_delimiter }}} where we depend on JSON serialization logic to properly write the string into the YAML file. This depends on removing these three lines that make the currently unused to_json handlebar function useless.

[qcorporation]

@kpollich, do you think your team could check whether the delimiter serialization from Kibana to the fleet policy works properly for this use case?