Agent cannot connect to ES once it has been upgraded

[honzakral]

For confirmed bugs, please report:

• Version: 8.7.0
• Operating System: k8s
• Steps to Reproduce: Create 8.6.0 deployment on elastic cloud, see agent fail to connect, upgrade the deployment on cloud and the agent still cannot connect with the same error:

Failed to connect to backoff(elasticsearch(https://NOT_TELLING.REGION.gcp.elastic-cloud.com:443)): Connection marked as failed because the onConnect callback failed: Elasticsearch is too old. Please upgrade the instance. If you would like to connect to older instances set output.elasticsearch.allow_older_versions to true. ES=8.6.1, Beat=8.7.0

Once the pod has been restarted it can connect fine

[clwo]

I noticed the same issue on our cluster as well. This bug is still present for version 8.7.1.

{"log.level":"error","@timestamp":"2023-05-16T09:22:49.305Z","message":"Failed to connect to backoff(elasticsearch(<REDACTED>)): Connection marked as failed because the onConnect callback failed: Elasticsearch is too old. Please upgrade the instance. If you would like to connect to older instances set output.elasticsearch.allow_older_versions to true. ES=8.6.2, Beat=8.7.1","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.logger":"publisher_pipeline_output","log.origin":{"file.line":150,"file.name":"pipeline/client_worker.go"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-16T09:22:49.305Z","message":"Attempting to reconnect to backoff(elasticsearch(<REDACTED>)) with 1655 reconnect attempt(s)","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.logger":"publisher_pipeline_output","log.origin":{"file.line":141,"file.name":"pipeline/client_worker.go"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-16T09:22:49.306Z","message":"Attempting to connect to Elasticsearch version 8.7.1","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"esclientleg","log.origin":{"file.line":291,"file.name":"eslegclient/connection.go"},"ecs.version":"1.6.0"}

[cmacknz]

Setting allow_older_versions: true in the output settings should work around this.

This should just work though, likely we are caching the version number from the first connection.

[pierrehilbert]

Should be solved by elastic/beats#36884