S3 remote logging not working for airflow server components

[kakarukeys]

Apache Airflow version: 2.0.1

Environment:

• Cloud provider or hardware configuration: on my laptop
• OS (e.g. from /etc/os-release): MacOS Majave 10.14.6
• Kernel (e.g. uname -a): Darwin Wongs-MBP 18.7.0 Darwin Kernel Version 18.7.0: Tue Jan 12 22:04:47 PST 2021; root:xnu-4903.278.56~1/RELEASE_X86_64 x86_64

What happened:

configured remote logging to S3 bucket, only the logs of DAG runs appeared in the bucket.
logs of airflow server components: scheduler, web server, etc did not appear

What you expected to happen:

all logs go to S3 bucket

How to reproduce it:

1. follow the quick start guide in https://airflow.apache.org/docs/apache-airflow/stable/start/local.html

2. before starting web server set the following variables:

export AIRFLOW__LOGGING__REMOTE_LOGGING=True
export AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER=s3://my-bucket/
export AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID=my_remote_logging_conn_id

3. start the web server and set your S3 connection settings in the web server "connections" section.

Conn Id * my_remote_logging_conn_id
Conn Type  S3
Extra {"region_name": "nyc3",
 "host": "https://nyc3.digitaloceanspaces.com",
 "aws_access_key_id": "xxx",
 "aws_secret_access_key": "xxx"}

4. Restart the web server
5. Start the scheduler in another console window (setting the same env variables)
6. Execute a DAG
7. Head to your S3 bucket UI, you will see only logs of DAG runs appear.

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[xinbinhuang]

Hi @kakarukeys , This is the correct behavior: currently, remote logging only sends task logs to remote storage. If you need to configure other logging behaviors, e.g. scheduler, webserver, you can create a custom logging config for use with your deployment. (see Logging for Tasks - Advanced Confgiuration)

Another production deployment option is to use FluentD to capture logs and send them to destinations such as ElasticSearch or Splunk. (ref Logging Architecture)

[kakarukeys]

@xinbinhuang the "correct behavior" you mentioned was not documented anywhere in the doc. Even the Logging Architecture picture shows logging, remote or not, apply to all components. See where the lines above "Logging" connect to?

[xinbinhuang]

See below the part inside the red circle:

Regardless, I agree with you that remote logging applies to only tasks may not obvious enough in the documentation. Would you like to submit a PR to improve the documentation?

[kakarukeys]

I was following this doc: https://airflow.apache.org/docs/apache-airflow/1.10.2/howto/write-logs.html

Since I opine that it is a questionable design decision to have some logs writing to one place, mostly ephemeral storage, and other logs to other place, under one logging config, I think I will decline that.

I can look into changing the code instead, if everyone agrees it is a bug.

[xinbinhuang]

Please refer to the most up-to-date documentation instead. There is not much point to reference old documentation to a new version, especially across major releases (i.e. 1.10.2 -> 2.0.*).

That being said, I believe that remote logging has been always only applying to task instance logs (#5175), and at some point, the old documentation was updated to reflect the fact.

[kakarukeys]

ah my bad habit of treating first link of Google as source of truth.

On the first read of the 2.0.1 doc, I can see Logging and Monitoring architecture page is still misleading.

diagram

By default, Airflow supports logging into the local file system. These include logs from the Web server, the Scheduler, and the Workers running tasks. This is suitable for development environments and for quick debugging.
For cloud deployments, Airflow also has handlers contributed by the Community for logging to cloud storage such as AWS, Google Cloud, and Azure.

"Logging for Tasks" is lacking any examples for different platforms.
"Logging for Others" can't be found.

[mik-laj]

Airflow handles some logs in a special way and unfortunately, it is not easy to unify this as our logs have different characteristics.

Logs for tasks are saved to files because they are small and we can upload them after completing the tasks. When they are sent to object storage it is much easier to read to them, but each time you add a new line, we need to get all the contents and upload a new file with the full log contents. Object storages have limited support for appending operations.
On the other hand, logs for other components are an endless stream of data that cannot be stopped to send the content. For this reason, conventional tools fit better because they are optimized to handle such operations.

I don't think we can fix this problem, but we can update the documentation to better describe these project assumption.

Related issue: #10593

[kakarukeys]

Since this is not classified as a bug, I will close this. Thanks for your help.