Debug level agent logs are generated on upgrading agent to 8.16.0 BC3 when info logging level is selected.

[amolnater-qasource]

Kibana Build details:

VERSION: 8.16.0 BC3
BUILD: 79556
COMMIT: f02d1303b5230c357ac7e4c49c8adadd5f66af38

Preconditions:

1. 8.16.0-BC3 Kibana cloud environment should be available.
2. Lower version say 7.17.25 or 8.15.3 agent should be installed.

Steps to reproduce:

1. Navigate to Fleet>Agents tab.
2. Observe info logging level is selected.
3. Upgrade agent to latest 8.16.0 BC3.
4. Observe debug level log is generated on successful upgrade:
   [elastic_agent][debug] received state: HEALTHY:Running
5. Observe same logs are generated multiple times.

Expected Result:
Debug level agent logs should not be generated on upgrading linux agent to 8.16.0 BC3 when info logging level is selected.

Screenshot:

Agent Logs:
elastic-agent-diagnostics-2024-11-04T05-16-32Z-00.zip

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[amolnater-qasource]

@muskangulati-qasource Please review.

[muskangulati-qasource]

Secondary review is Done for this ticket!

[pierrehilbert]

You mentioned on Linux, does it mean that this is not happening with other OS?
Is it also happening when upgrading to 8.15.4?

[amolnater-qasource]

Hi @pierrehilbert

Thank you for looking into this issue.

We had run the tests on Linux so reported issue on the same.

Further we have revalidated this issue on 8.15.4 while upgrading 8.15.3 agent and found it reproducible at our end.

• Debug logs are populated more frequently on upgrading from 8.15.3>8.15.4
• Issue is reproducible on all OS's.

Agent logs:
elastic-agent-diagnostics-2024-11-05T08-34-51Z-00.zip
elastic-agent-diagnostics-2024-11-05T10-05-59Z-00.zip

Please let us know if anything else is required from our end.

Thanks!

[pchila]

@pierrehilbert @amolnater-qasource From a quick look in the code, those debug logs are generated by the watcher that runs for the grace period after the upgrade, and from what I can see this is defaulted to debug here called from newWatchCommand...

It seems like that there's not log level settings coming from config so watcher keeps logging at debug level...
I checked the actual elastic-agent log files and there's no debug log in there so this seems to be limited only to the elastic-agent watcher