FORBIDDEN access to worker's logs from Web UI

[denisvolokh]

Apache Airflow version:
Airflow version: 2.1.2

Airflow Web UI is not able to fetch logs (actually stopped working) from the worker's container. It fails with the following error:

*** Log file does not exist: /app/airflow/logs/PROD-synch_strategies_data/t_synch_strategies_summaries/2021-07-27T06:08:44.532400+00:00/1.log
*** Fetching from: http://a32f9e563f01:8793/log/PROD-synch_strategies_data/t_synch_strategies_summaries/2021-07-27T06:08:44.532400+00:00/1.log
*** Failed to fetch log file from worker. 403 Client Error: FORBIDDEN for url: http://a32f9e563f01:8793/log/PROD-synch_strategies_data/t_synch_strategies_summaries/2021-07-27T06:08:44.532400+00:00/1.log
For more information check: https://httpstatuses.com/403

When I checked the logs at the worker container, I can see all logs at the worker's container.

What could be wrong here?

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[jedcunningham]

@denisvolokh, this is from #16754 and implies you don't have the same [webserver] secret_key on your webserver and worker. How are you running Airflow? Can you make sure you have the same value for both?

If you are running the official helm chart, the latest release (1.1.0) has a fix to ensure the same value is used, so if you haven't updated yet, give that a shot.

[denisvolokh]

Thank you @jedcunningham, after adding AIRFLOW__WEBSERVER__SECRET_KEY config I am able to see logs from web ui.

[zachliu]

in case someone else is in the same situation, i used to do

export AIRFLOW__WEBSERVER__SECRET_KEY=${SECRET_KEY:=$(python -c "from base64 import b64encode; import os; SECRET_KEY = b64encode(os.urandom(16)).decode('utf-8'); print(SECRET_KEY)")}

in my entrypoint.sh when running Airflow locally. this makes webserver and worker have different keys
now i just need to

export AIRFLOW__WEBSERVER__SECRET_KEY=${<READ_FROM_OUR_SECRET_MANAGER>}