[Fleet] Change default agent log level from info to warning

[kpollich]

Currently, Fleet sets the default log level to info here.

We should set this to warning instead to saved on storage space and reduce the verbosity of agent logs by default.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[strawgate]

@nimarezainia we'd still have the ability to troubleshoot individual agents and we do have a sample script customers can use to change the log level across agents individually https://gist.github.com/octavioranieri/116cdb48eab548eb9fc38ae0f79332ca

[strawgate]

Maybe paired with the "Agent-Policy log-level" here: #158861 I believe the global setting has a bigger estimate than the agent-policy change which I think just requires exposing a UI change

[strawgate]

In our previous testing we expect an 80-90% reduction in Agent monitoring volume with a change to Warn log level.

[juliaElastic]

I changed the default log level to warning locally here, but it seems the agents set info level (tested with docker and local agent too) and the UI takes the agent setting first here.
I wonder if the default setting should be changed on agent side? Checking where the local_metadata log_level default value is set.
I found it set here.
@kpollich @cmacknz Should we raise a corresponding issue in agent repo to change it to warning?

[cmacknz]

In our previous testing we expect an 80-90% reduction in Agent monitoring volume with a change to Warn log level.

Yes because we barely log anything at the warning level. We could reduce storage by 100% by simply not logging anything :) Before we change the log level, we need to make sure that we log enough for basic troubleshooting first. We currently assume this is the Info level in our code, but we can change that and probably get rid of some verbose but low value logging.

I wonder if the default setting should be changed on agent side? Checking where the local_metadata log_level default value is set.

Yes if we change this we also need to change it in the default elastic-agent.yml file the agent uses when it is installed.

It would also be good to support #158861 if we do end up changing the default log level, in case we miss moving a critical log we don't want users having to change the log level for each individual agent.

[strawgate]

Agree on having:

1. Easy ability to change individual agent's logging level from Fleet (Done)
2. Easy ability to change agent logging level per-policy from Fleet ([Fleet] Support changing the default log level per policy (or globally) #158861)

I dont think we need the ability to switch it globally for this.

If we have the above 2 how much work auditing the existing log level do we need to make this happen @cmacknz ?

We could reduce storage by 100% by simply not logging anything

New north star goal 😄

[cmacknz]

If we have the above 2 how much work auditing the existing log level do we need to make this happen @cmacknz ?

Hard to give an exact number. More than a day, less than a week.

[strawgate]

@cmacknz is there a task in the agent backlog yet? Shall I create one?

[cmacknz]

There is no agent issue yet, go ahead and create one.

[jlind23]

@strawgate please let us know once the issue is created then @pierrehilbert will prioritize it.

[strawgate]

elastic/elastic-agent#4252 (comment)

[jen-huang]

#158861 should be done first prior to this one so that there is a rollback option (hence marking this as blocked).

The one-line change to change the default from info to warn should not be merged until the data is validated in elastic/elastic-agent#4252.

[kpollich]

This is now blocked by #180778 which is in turn blocked by elastic/elastic-agent#4747

[nimarezainia]

the original intention here was to reduce the logs collected by having WARN as the default. Changing this default could be a breaking change.
The INFO logs (current default) was audited and logging was considerably reduced. So the original intent of this issue has been addressed. Parallel development in having the log level controlled at a policy level.

Do we need to still change the default from INFO to WARN? this could cause problems for our customers.

I think we can just close this issue. @cmacknz @kpollich ?

[kpollich]

I'm happy to close this if other work has superseded the need to change the default here.