[Breaking change] Logging configuration format change

[joshdover]

Change description

Which release will ship the breaking change?

8.0

Describe the change. How will it manifest to users?

A new logging system and configuration format have replaced the 7.x logging system. This new logger provides functionality similar to the logging capabilities of Elasticsearch, based on the log4j 2 framework. Users may now configure multiple appenders, layouts, and loggers to customize where log messages are sent, their format, and filter based on logging context. This allows for more fine-grained control over the logging output.

The following logging configuration keys have been removed:

• logging.dest
• logging.json
• logging.quiet
• logging.silent
• logging.verbose
• logging.timezone
• logging.rotate.*
• logging.filter.*
• logging.events.*
• elasticsearch.logQueries

Users will need to remove any of these keys and leverage the new logging configuration format to achieve similar results. More details can be found here: https://github.com/elastic/kibana/blob/master/src/core/server/logging/README.md (We will move these docs into the standard asciidoc system before 8.0).

How many users will be affected?

We do not appear to have telemetry on the legacy logging configuration, but we suspect many users use the logging.dest config for logging to a file.

How can we programmatically determine whether the cluster is affected by this breaking change?

Detect any of these deprecated config keys being set. Should be provided by config deprecation API once supported by Core.

What can users do to address the change manually?

Docs TBD

How could we make migration easier with the Upgrade Assistant?

Notifying user of use of deprecated setting + link to docs. One tricky aspect may be muting this warning for Cloud users, since Cloud will be supplying the logging config keys.

Are there any edge cases?

Test Data

Cross links

#60391

[elasticmachine]

Pinging @elastic/es-ui (Team:Elasticsearch UI)

[alisonelizabeth]

I'm going to remove the Elasticsearch UI team label and add the Core team, as this should be handled via the core deprecations service (#97159).

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[lukeelmers]

All of the above items are deprecated yml configs, which means they're already being surfaced by the deprecations service for use in the upgrade assistant. And we already have #50660 to track the breaking changes which will be made to remove the legacy logging infrastructure.

I'll go ahead and close this issue, but @cjcenizal feel free to re-open if it is needed for some other reason.