Airflow 2.2.2 remote worker logging getting 403 Forbidden

[armourshield]

Apache Airflow version

Other Airflow 2 version

What happened

I have a setup where airflow is running in kubernetes (EKS) and remote worker running in docker-compose in a VM behind a firewall in a different location.

Problem
Airflow Web server in EKS is getting 403 forbidden error when trying to get logs on remote worker.

Build Version

• Airflow - 2.2.2
• OS - Linux - Ubuntu 20.04 LTS

Kubernetes

• 1.22 (EKS)
• Redis (Celery Broker) - Service Port exposed on 6379
• PostgreSQL (Celery Backend) - Service Port exposed on 5432

Airflow ENV config setup

  AIRFLOW__API__AUTH_BACKEND: airflow.api.auth.backend.basic_auth
  AIRFLOW__CELERY__BROKER_URL: redis://<username>:<password>@redis-master.airflow-dev.svc.cluster.local:6379/0
  AIRFLOW__CELERY__RESULT_BACKEND: >-
    db+postgresql://<username>:<password>@db-postgresql.airflow-dev.svc.cluster.local/<db>
  AIRFLOW__CLI__ENDPOINT_URL: http://{hostname}:8080
  AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
  AIRFLOW__CORE__EXECUTOR: CeleryExecutor
  AIRFLOW__CORE__FERNET_KEY: <fernet_key>
  AIRFLOW__CORE__HOSTNAME_CALLABLE: socket.getfqdn
  AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
  AIRFLOW__CORE__SQL_ALCHEMY_CONN: >-
    postgresql+psycopg2://<username>:<password>@db-postgresql.airflow-dev.svc.cluster.local/<db>
  AIRFLOW__LOGGING__BASE_LOG_FOLDER: /opt/airflow/logs
  AIRFLOW__LOGGING__WORKER_LOG_SERVER_PORT: '8793'
  AIRFLOW__WEBSERVER__BASE_URL: http://{hostname}:8080
  AIRFLOW__WEBSERVER__SECRET_KEY: <secret_key>
  _AIRFLOW_DB_UPGRADE: 'true'
  _AIRFLOW_WWW_USER_CREATE: 'true'
  _AIRFLOW_WWW_USER_PASSWORD: <username-webserver>
  _AIRFLOW_WWW_USER_USERNAME: <password-webserver>

Airflow is using CeleryExecutor

Setup Test

1. Network reach ability by ping - OK
2. Celery Broker reach ability for both EKS and remote worker - OK
3. Celery Backend reach ability for both EKS and remote worker - OK
4. Firewall Port expose for remote worker Gunicorn API - OK
5. curl -v telnet://:8793 test - OK (Connected)
6. Airflow flower recognizing both workers from Kubernetes and remote worker - OK
7. All the ENV on both webserver, worker (EKS, remote) and scheduler are identical
8. Queue is setup so the DAG runs exactly in that particular worker
9. Time on both docker, VM and EKS is on UTC. There is a slight 5 to 8 seconds difference in docker and the pod in EKS
10. Ran webserver on the remote VM as well which can pick up and show logs

Description
Airflow is able to execute the DAG in remote worker, the logs can be seen in the remote worker. I have tried all combinations of setting but still keep getting 403.

Another test which was done was just normal curl with webserver auth

This curl was done both from EKS and remote server which hosts docker-compose. Results are the same on all the server.

curl --user <username-webserver> -vvv http:<remote-worker>:8793/logs/?<rest-of-the-log-url>
Getting 403 Forbidden

I might have miss configured it, but I doubt that is the case.
Any tips on what I am missing here? Many thanks in advance.

What you think should happen instead

Airflow web-server in EKS should be able to access the remote logs from remote worker as the port is accessible and all the secret key, fernet key matches and all the env matches.

How to reproduce

Use K3S, minikube or kubernetes of your choice. to replicate remote-worker run docker-compose setup for the installation.

Build Version

• Airflow - 2.2.2
• OS - Linux - Ubuntu 20.04 LTS

Kubernetes

• 1.22 EKS, K3S, minikube or kubernetes
• Redis (Celery Broker) - Service Port exposed on 6379
• PostgreSQL (Celery Backend) - Service Port exposed on 5432

Airflow ENV config setup

  AIRFLOW__API__AUTH_BACKEND: airflow.api.auth.backend.basic_auth
  AIRFLOW__CELERY__BROKER_URL: redis://<username>:<password>@redis-master.airflow-dev.svc.cluster.local:6379/0
  AIRFLOW__CELERY__RESULT_BACKEND: >-
    db+postgresql://<username>:<password>@db-postgresql.airflow-dev.svc.cluster.local/<db>
  AIRFLOW__CLI__ENDPOINT_URL: http://{hostname}:8080
  AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
  AIRFLOW__CORE__EXECUTOR: CeleryExecutor
  AIRFLOW__CORE__FERNET_KEY: <fernet_key>
  AIRFLOW__CORE__HOSTNAME_CALLABLE: socket.getfqdn
  AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
  AIRFLOW__CORE__SQL_ALCHEMY_CONN: >-
    postgresql+psycopg2://<username>:<password>@db-postgresql.airflow-dev.svc.cluster.local/<db>
  AIRFLOW__LOGGING__BASE_LOG_FOLDER: /opt/airflow/logs
  AIRFLOW__LOGGING__WORKER_LOG_SERVER_PORT: '8793'
  AIRFLOW__WEBSERVER__BASE_URL: http://{hostname}:8080
  AIRFLOW__WEBSERVER__SECRET_KEY: <secret_key>
  _AIRFLOW_DB_UPGRADE: 'true'
  _AIRFLOW_WWW_USER_CREATE: 'true'
  _AIRFLOW_WWW_USER_PASSWORD: <username-webserver>
  _AIRFLOW_WWW_USER_USERNAME: <password-webserver>

Operating System

Ubuntu 20.04 LTS

Versions of Apache Airflow Providers

No response

Deployment

Docker-Compose

Deployment details

Airflow ENV config setup

  AIRFLOW__API__AUTH_BACKEND: airflow.api.auth.backend.basic_auth
  AIRFLOW__CELERY__BROKER_URL: redis://<username>:<password>@redis-master.airflow-dev.svc.cluster.local:6379/0
  AIRFLOW__CELERY__RESULT_BACKEND: >-
    db+postgresql://<username>:<password>@db-postgresql.airflow-dev.svc.cluster.local/<db>
  AIRFLOW__CLI__ENDPOINT_URL: http://{hostname}:8080
  AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
  AIRFLOW__CORE__EXECUTOR: CeleryExecutor
  AIRFLOW__CORE__FERNET_KEY: <fernet_key>
  AIRFLOW__CORE__HOSTNAME_CALLABLE: socket.getfqdn
  AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
  AIRFLOW__CORE__SQL_ALCHEMY_CONN: >-
    postgresql+psycopg2://<username>:<password>@db-postgresql.airflow-dev.svc.cluster.local/<db>
  AIRFLOW__LOGGING__BASE_LOG_FOLDER: /opt/airflow/logs
  AIRFLOW__LOGGING__WORKER_LOG_SERVER_PORT: '8793'
  AIRFLOW__WEBSERVER__BASE_URL: http://{hostname}:8080
  AIRFLOW__WEBSERVER__SECRET_KEY: <secret_key>
  _AIRFLOW_DB_UPGRADE: 'true'
  _AIRFLOW_WWW_USER_CREATE: 'true'
  _AIRFLOW_WWW_USER_PASSWORD: <username-webserver>
  _AIRFLOW_WWW_USER_USERNAME: <password-webserver>

Anything else

This has been failing to fetch logs even there is there complete access to the port and remote worker. 403 Forbidden keeps saying secret key when it is the same across the whole environment.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

re: Curl : Logs can only be retrieved by webserver that is authenticated using secret key. That's why you get "forbidden". Airflow webserver passes signed JWT tokens that authenticate requests to read logs and this is internal API of Airflow - not meant to be used by external clients. You might want to take a look at the sources of Airfflow to see the details on how it is done and replicate it, but be warned - it might (and will) change - for example in airlfow 2.4 this authentication mechanism is changed.

If you want to access logs, I recommend you to use persistent storage shared between your workers where the logs are placed and access them via this persistent storage. It's much more stable and "public" approach.

Re: webserver - make sure you have the same version of airflow and the same secret key and that your time is synchronized for all your machines. If it does not work Upgrade to latest version of Airflow - and look at worker logs. In the latest (2.3.4 I think but 2.4.0 for sure) version of airflow we have much more detailed information in the worker logs on why Forbidden was returned.

[potiuk]

Note TIME synchronisation - those tokens have short (few minutes) expiry time (for security) . If the time on your servers is even few minutes off, you will get the problem. Use NTPd or similar to make sure the time is not drifting.

[armourshield]

Thanks @potiuk , I will be testing out with NTPd services setup on the remote worker will let you know how it goes. Thanks

[armourshield]

@potiuk Thanks for the pointers.

I tried NTPd Syncing with EKS, Remote workers
All working on same timezone and syncing to NTP now
Verified timesync with

watch timedatectl status

On all the servers, it is working to the second.
But still facing the issue.

Now the only point I might be missing is secret key variable, is special characters allowed or is that breaking the auth secret-key I have.

As you suggested need to see how the 2.2.2 auth works. Let me know if I am missing here.

[armourshield]

@potiuk I made it work! thanks for the pointers.

I rebuilt the curl auth from this file of branch 2.2 https://github.com/apache/airflow/blob/main/airflow/utils/log/file_task_handler.py

Later realized that the auth doesn't like special characters in secret key, and added to that there was NTP time drift of 135 seconds (2min 15seconds) which would also factor in causing confusion.

I would recommend people who would face this problem to avoid special characters in secret key. Just a airflow user recommendation, I wouldn't want to say it is the only solution but something which helped me. Special character and combined with NTP caused confusion for debugging the issue resolving NTP should be first than to the auth.

And again thanks for the timely support appreciate it.

Will close this discussion.

[potiuk]

I am glad it helped.