Drop "connection closed before message completed" messages to debug

[jszwedko]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

A few users have reported that the HTTP warning messages "connection closed before message completed" are noisy since having connections terminate abruptly is not unusual and the requests are retried.

Some users also think these are hard errors that are resulting in dropped events because of the log level.

Attempted Solutions

No response

Proposal

Drop these log messages to the DEBUG level.

References

• The aws_s3 sink is logging connection_closed_before_message_completed errors #9671
• The aws_kinesis_firehose sink is logging "connection closed before message completed" errors #12483

Version

vector 0.25.1

[lordsherman]

These messages were fairly concerning to me in my setup until I stumbled upon this posted issue. While I never saw lost data due to this it was making me think something was wrong.

I send out data to central Loki and Victoriametrics instances. Some of my regional vector instances that push this data out will see these errors way more than others. It depends on how many servers locally push to these regional instances. I am going to have to set log limits for my containers in the locations that have a lot of servers pushing to them or my instances will run out of storage space because of this.

I would love to see this change.