[Alerting] Add telemetry for config deprecation

[chrisronline]

Relates to #76385
Relates to #100289

We must be able to answer these questions:

• What percentage of clusters are using xpack.actions.whitelistedHosts?
• What percentage of clusters are using xpack.actions.customHostSettings[<index>].ssl.rejectUnauthorized? (Blocked by Track usage for deprecated array configs #112845)
• What percentage of clusters are using xpack.actions.rejectUnauthorized?
• What percentage of clusters are using xpack.actions.proxyRejectUnauthorizedCertificates?
• What percentage of clusters are using xpack.alerts.healthCheck?
• What percentage of clusters are using xpack.alerts.invalidateApiKeysTask.interval?
• What percentage of clusters are using xpack.alerts.invalidateApiKeysTask.removalDelay?
• What percentage of clusters are using xpack.task_manager.max_workers values with a value greater than 100?

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[chrisronline]

@pgayvallet

In regards to our offline conversation about xpack.actions.whitelistedHosts, because we want to also perform a rename on this config, simply adding it back into our config doesn't achieve the necessary results because the rename will prevent the original config from showing up in the usage telemetry (I think because the rename happens before the telemetry code is able to determine which configs are used). Is there a short term fix for this? Or will we have to change our read code to look for either config (and not rely on the rename)?

[chrisronline]

One problem with xpack.actions.customHostSettings[<index>].ssl.rejectUnauthorized is that it's an array and we cannot track usage through the built-in way to do this (see the last line on the README)

[pgayvallet]

because we want to also perform a rename on this config, simply adding it back into our config doesn't achieve the necessary results because the rename will prevent the original config from showing up in the usage telemetry

That's right, rename totally get rid of the legacy key and simply moves it to the new key/path instead.

I created #112633 to talk about this.

In the short term, as discussed offline, the only option I see would be to stop using the rename deprecation and add a (custom) copy deprecation instead that would copy the old key to the new path, while preserving it. That way, the config collector would be aware of the old key (note that any change to the old key would also be collected for the new key though, as it would be copied before the collector's config inspection)

[Bamieh]

We report the value (or [redacted]) of the non-deprecated config regardless which config was used.

lets say for example we have:

rename('whitelistedHosts', 'allowedHosts')

Our config collector will report the config key and value of allowedHosts regardless if whitelistedHosts or allowedHosts was set in the kibana configs.

To check if the deprecated renamed config whitelistedHosts was used you need to check another collector we have in telemetry in the path: core.config.deprecatedKeys.unset

To check the value of the config follow this usage path: kibana.plugins.kibana_config_usage

I believe a combination of both collectors should answer all the metrics you're after in this issue. We already have dashboards in the telemetry cluster for other plugins to answer similar questions. Happy to share offline