Metricbeat 7.5.2 silently stops collecting Logstash monitoring data

[clintonjimmie]

Issue: Moving over to Metricbeat Monitoring from the stack internal Monitoring tools and after an roughly 2 hours, the Logstash stats entirely drop out of the Monitoring app in Kibana. If I leaves the Monitoring tab open, the charts are empty of data. The logs are totally silent in regard to the failure. Nothing in Beats logs and nothing in Logstash logs.

In my lab setup I have one Metricbeat instance monitoring Kibana, Elasticsearch and Logstash. When Logstash stops showing up, it's the only monitoring module that stops / fails. Elasticsearch monitoring data is still coming in and so is Kibana monitoring data. To get Logstash information back into Monitoring, restarting the Metricbeat process is necessary. Restarting Logstash had no effect.

For confirmed bugs, please report:

• Version: 7.5.2
• Operating System: Ubuntu 18.04 LTS
• Discuss Forum URL: N/A
• Steps to Reproduce:

set up stack install of Elasticsearch (7.5.2 in my lab's case)
install and set up one instance of Metricbeat local to the node to monitor the components in the stack specifically Logstash
wait for Logstash to drop out of the Monitoring app in Kibana

[elasticmachine]

Pinging @elastic/stack-monitoring (Stack monitoring)

[ycombinator]

I think I'm able to reproduce this locally with Metricbeat 7.5.2. The only reason I'm not sure it's the same issue is because I do start seeing HTTP 400 errors in the Metricbeat log when Logstash data stops appearing in the Stack Monitoring UI whereas @clintonjimmie said in the issue description that they didn't see anything in the Metricbeat logs.

At any rate, the issue I'm seeing is this:

1. When Metricbeat first starts up, the logstash module makes an HTTP call to the Logstash API: GET /_node/stats?vertices=true.
2. The second time the logstash module calls the Logstash API, it adds another ?vertices=true to the URL. The third time, it adds one more, and so on.
3. Evidently, Logstash happily accepts these URLs, but up to a point. At some point, it starts rejecting them with HTTP 400 responses. My guess is there is some length threshold that's being crossed in the Logstash HTTP server.

I will work on a fix to remove the repeated ?vertices=true occurrences in the API call.