Logstash Source - Failed framing bytes.","error":"i/o error: Connection reset by peer (os error 104)

[greycel]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

We've been using Vector in production to collect logs from around 500 Windows endpoints and 100 Linux machines:
Windows Setup: Winlogbeat agent is on the sending side, and the Vector logstash source (winevents) is on the receiving side.
Linux Setup: Auditbeat is on the sending side, and the Vector logstash source (linux_auditbeat) is on the receiving side.
Vector Version: Vector-Dev v 0.46.0

I'm encountering the following error in Vector: Failed framing bytes.","error":"i/o error: Connection reset by peer (os error 104) Shortly after, this is followed by: '"Internal log [Failed framing bytes.] is being suppressed to avoid flooding.'

Despite restarting the Vector service, the issue reappears within a few minutes. I'm unable to determine the root cause. Any insights or suggestions would be greatly appreciated. The following is the vector-dev error log.

-- Logs begin at Thu 2025-04-24 10:16:28 UTC. --
Apr 24 11:01:03 collector_02 vector[1620]: {"timestamp":"2025-04-24T11:01:03.947035Z","level":"ERROR","message":"Failed framing bytes.","error":"i/o error: Connection reset by peer (os error 104)","error_code
":"decoder_frame","error_type":"parser_failed","stage":"processing","internal_log_rate_limit":true,"target":"vector::internal_events::codecs","span":{"peer_addr":"191.168.11.220:32854","name":"connection"},"s
pans":[{"component_id":"input_auditbeat","component_kind":"source","component_type":"logstash","name":"source"},{"peer_addr":"191.168.11.220:32854","name":"connection"}]}
Apr 24 11:01:03 collector_02 vector[1620]: {"timestamp":"2025-04-24T11:01:03.947159Z","level":"ERROR","message":"Internal log [Failed framing bytes.] is being suppressed to avoid flooding.","target":"vector::
internal_events::codecs","span":{"peer_addr":"191.168.11.220:32860","name":"connection"},"spans":[{"component_id":"input_auditbeat","component_kind":"source","component_type":"logstash","name":"source"},{"pee
r_addr":"191.168.11.220:32860","name":"connection"}]}
Apr 24 11:01:06 collector_02 vector[1620]: {"timestamp":"2025-04-24T11:01:06.523442Z","level":"ERROR","message":"Internal log [Error writing acknowledgement, dropping connection.] has been suppressed 2 times.
","target":"vector::internal_events::tcp","span":{"peer_addr":"191.168.11.220:48358","name":"connection"},"spans":[{"component_id":"input_windows","component_kind":"source","component_type":"logstash","name":"
source"},{"peer_addr":"191.168.11.220:48358","name":"connection"}]}
Apr 24 11:01:06 collector_02 vector[1620]: {"timestamp":"2025-04-24T11:01:06.523561Z","level":"ERROR","message":"Error writing acknowledgement, dropping connection.","error":"Broken pipe (os error 32)","error
_code":"ack_failed","error_type":"writer_failed","stage":"sending","internal_log_rate_limit":true,"target":"vector::internal_events::tcp","span":{"peer_addr":"191.168.11.220:48358","name":"connection"},"spans
":[{"component_id":"input_windows","component_kind":"source","component_type":"logstash","name":"source"},{"peer_addr":"191.168.11.220:48358","name":"connection"}]}
Apr 24 11:01:07 collector_02 vector[1620]: {"timestamp":"2025-04-24T11:01:07.742411Z","level":"ERROR","message":"Internal log [Error writing acknowledgement, dropping connection.] has been suppressed 2 times.
","target":"vector::internal_events::tcp","span":{"peer_addr":"191.168.11.220:61894","name":"connection"},"spans":[{"component_id":"input_auditbeat","component_kind":"source","component_type":"logstash","name
":"source"},{"peer_addr":"191.168.11.220:61894","name":"connection"}]}
Apr 24 11:01:07 collector_02 vector[1620]: {"timestamp":"2025-04-24T11:01:07.742558Z","level":"ERROR","message":"Error writing acknowledgement, dropping connection.","error":"Connection reset by peer (os erro
r 104)","error_code":"ack_failed","error_type":"writer_failed","stage":"sending","internal_log_rate_limit":true,"target":"vector::internal_events::tcp","span":{"peer_addr":"191.168.11.220:61894","name":"conne
ction"},"spans":[{"component_id":"input_auditbeat","component_kind":"source","component_type":"logstash","name":"source"},{"peer_addr":"191.168.11.220:61894","name":"connection"}]}
Apr 24 11:01:08 collector_02 vector[1620]: {"timestamp":"2025-04-24T11:01:08.907292Z","level":"ERROR","message":"Internal log [Error writing acknowledgement, dropping connection.] is being suppressed to avoid
 flooding.","target":"vector::internal_events::tcp","span":{"peer_addr":"191.168.11.220:48876","name":"connection"},"spans":[{"component_id":"input_auditbeat","component_kind":"source","component_type":"logst
ash","name":"source"},{"peer_addr":"191.168.11.220:48876","name":"connection"}]}
Apr 24 11:01:14 collector_02 vector[1620]: {"timestamp":"2025-04-24T11:01:14.699174Z","level":"ERROR","message":"Internal log [Failed framing bytes.] has been suppressed 4 times.","target":"vector::internal_e
vents::codecs","span":{"peer_addr":"191.168.11.220:35052","name":"connection"},"spans":[{"component_id":"input_auditbeat","component_kind":"source","component_type":"logstash","name":"source"},{"peer_addr":"1
72.29.108.251:35052","name":"connection"}]}

Configuration

sources:
  input_windows:
    address: "0.0.0.0:12102"
    type: "logstash"
  input_auditbeat:
    address: "0.0.0.0:12514"
    type: "logstash"
transforms:
  process_windows_events:
    inputs:
    - "input_windows"
    source: |-
      .log.schema = "raw"
      .log.format = "json"
    type: "remap"
  process_linux_auditbeat:
    inputs:
    - "input_auditbeat"
    source: |-
      .log.schema = "raw"
      .log.format = "json"
    type: "remap"
sinks:
  output_S3:
    acl: "private"
    auth:
      access_key_id: "*****************"
      region: "ap-south-1"
      secret_access_key: "***************"
    bucket: "Customer03-logs"
    compression: "gzip"
    encoding:
      codec: "json"
    framing:
      method: "newline_delimited"
    key_prefix: "logs/"
    region: "ap-south-1"
    storage_class: "STANDARD"
    type: "aws_s3"

Version

0.46.0

Debug Output

Example Data

No response

Additional Context

No response

References

No response