Is it possible to log Scheduler and WebServer using Elasticsearch?

[kimminw00]

As S3 has limited support for appending operations, the logs of DAG runs only appeared in the bucket when using S3 remote logging with Airflow.
Then is it possible to log Scheduler and WebServer using elasticsearch (or opensearch)?

Reference
S3 remote logging not working for airflow server components

[mis98zb]

As my understanding, there is no official support to write log directly to elasticsearch.
You should use something like filebeat to collect logs from files to elasticsearch yourself, and tell airflow how to read it as this

[potiuk]

Just follow https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/logging-tasks.html#advanced-configuration - airflow components use standard logging and when you look at DEFAULT_LOGGING_CONFIG you will see that you can configure any loggers for any handlers easily by modifying the config dictionary. Just use any standard elasticsearch or opensearch python logger you will find works for you.

[potiuk]

I also added a PR to improve the docs slightly about it #32131 and separate out the "advanced logging" from "logging for tasks" because it's not only task logs you can cofigure the way described there.