Logging: Log back pressure from Logstash

[ppf2]

We have seen situations where Logstash is not able to keep up downstream (eg. 5.4-5.5 performance regression bug in Logstash). For these situations, filebeat can leave file handles open (unless close_timeout is used). For troubleshooting, it will be nice if there is some indication in the beats log file indicating that Logstash is not able to keep up so users can quickly focus on downstream components. @ph noted that Logstash will send keep alive info to beats so we can potentially leverage that.

[ruflin]

We recently had a discussion about this in the team and definitively want do some improvements here. I suggest we use this issue also to brainstorm on what we could do to improve the logging here.

Some additional logs entries which could be useful:

• Data on how many bytes per file/total filebeat is behind on reading. Note: Tracking how much behind in lines is trickier as it doesn't know line numbers of a file before reading all lines
• How many events in the queue / pipeline
• Current window size used
• Change in window size to LS compared to 30s ago

@elastic/beats Can you please add your own thoughts here?

[andrewkroh]

I think having a histogram of the (max?) age of items in the publisher queue with some different time windows like 1m, 5m, 15m would be helpful.

[ruflin]

@andrewkroh So you would compare the current timestamp on sending to the timestamp in the event itself and get the max value? I like that :-)

[cdenneen]

Trying to diagnose a beat 5.1.1 -> logstash 5.6.8 issue and having some sort of back pressure log message would be useful here...

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[jlind23]

Backlog grooming: closing for now until the needs pops up again.