Write Slow Logs to a data stream

[pgomulka]

Indexing slow logs to a data stream would help identifying slow queries from Kibana ES UI as well as gathering statistics about overall health of the cluster as presented on prototypes here cc @cjcenizal

Often times users do not have easy access to slow logs, so making them available on data stream would make troubleshooting easier.
similar issue raised recently - #61118

This would be an easy follow up on #58924

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Logging)

[jimczi]

This would be an easy follow up on #58924

While I understand why deprecation logs would be treated differently, I wonder if we should rather aim at integrating filebeat for this kind of task. There's an effort that started to bundle metricbeat in ES to replace monitoring so that could be similar. That would make the indexing and progress completely asynchronous and less prone to memory explosion in case there's a lot of write/search activities on the cluster ?

[pgomulka]

we discussed this on core infra sync 16th sep
the concussion is that we don't want index slow logs with the same mechanism we use for deprecation logs because this would cause a performance problem.
If slow logs are to be indexed, they should be indexed with an external process (like filebeat) ideally to a separate monitoring cluster.