[stable/airflow] Errno 13 - Permission denied: '/opt/airflow/logs/scheduler

[axk7812]

Describe the bug
When mounting persistent volume for logs - encountering Permission denied: '/opt/airflow/logs/scheduler due to mismatch in uid:gid permissions in logs directory. Airflow user runs as 50000 and /opt/airflow/logs is airflow:root (uid:gid).

Version of Helm and Kubernetes:
Helm: v3.0.2
Kubernetes: v1.17.0

Which chart:
stable/airflow

What happened:

After installing and configuring Persistent Volume (AWS EFS) storage for logs - airflow is unable to write to /opt/airflow/logs
This is visible during this initial startup script.

*** executing Airflow initdb...
Unable to load the config, contains a configuration error.
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/logging/config.py", line 565, in configure
    handler = self.configure_handler(handlers[name])
  File "/usr/local/lib/python3.6/logging/config.py", line 738, in configure_handler
    result = factory(**kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/log/file_processor_handler.py", line 50, in __init__
    os.makedirs(self._get_log_directory())
  File "/usr/local/lib/python3.6/os.py", line 220, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/opt/airflow/logs/scheduler/2020-08-20'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/bin/airflow", line 25, in <module>
    from airflow.configuration import conf
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/__init__.py", line 47, in <module>
    settings.initialize()
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/settings.py", line 374, in initialize
    LOGGING_CLASS_PATH = configure_logging()
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/logging_config.py", line 68, in configure_logging
    raise e
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/logging_config.py", line 63, in configure_logging
    dictConfig(logging_config)
  File "/usr/local/lib/python3.6/logging/config.py", line 802, in dictConfig
    dictConfigClass(config).configure()
  File "/usr/local/lib/python3.6/logging/config.py", line 573, in configure
    '%r: %s' % (name, e))
ValueError: Unable to configure handler 'processor': [Errno 13] Permission denied: '/opt/airflow/logs/scheduler/2020-08-20'

What you expected to happen:
Logs should be stored in persistent volume

How to reproduce it (as minimally and precisely as possible):
In this post I share my values.yaml as well as the yaml to create the PV and PVC. Again I'm using EFS storage on AWS.

helm pull stable/airflow --untar=true

Replace the values.yaml with the one in above linked post (after creating PV/PVC from provided yaml)

helm install airflow airflow/ --values airflow/values.yaml --namespace airflow

Anything else we need to know:
I've tried debugging by editing the scheduler-deployment.yaml file and adding some commands in the section where airflow initdb is called. When I create a new directory it works. The uid:gid for this directory is airflow:airflow. When I see who owns /opt/airflow/logs it's airflow:root. Also, when I mount the external volume it gets mounted as 1000:1000. I've tried changing the RUN_AS_USER config but apparently it doesn't work if it's not 50000. So what I don't understand is howcome the startup script running as user 'airflow' is not allowed to create files in a directory with owner airflow? This seems like a bug.

I've tried to set the securityContext for scheduler but this didn't work

  securityContext:
    runAsUser: 1000
    runAsGroup: 1000
    fsGroup: 1000

All I got was the error bash: line 2: /home/airflow/airflow_env.sh: Permission denied.

I've also checked the airflow-scheduler deployment and I see that there is a mount for logs:

volumeMounts:
        - mountPath: /opt/airflow/logs
          name: logs-data

And there is a corresponding volume:

volumes:
      - name: logs-data
        persistentVolumeClaim:
          claimName: efs-claim

In the logs section of the helm chart I set enable to true, and set the existing claim as efs-claim. I've also set some configs in the airflow.config section. There are more but these I saw were relevant to maybe the issue.

AIRFLOW__KUBERNETES__LOGS_VOLUME_CLAIM: "efs-claim"
AIRFLOW__CORE__LOAD_EXAMPLES: "False"

when I exec into the airflow-scheduler I do not see any log folders created. /opt is empty.

[mykhailoguk]

we are facing the same issue with Permission denied for /opt/airflow/logs/scheduler

[ivanrezic-maistra]

We are facing same problem too

[gj-9lt]

Facing the same issue.

Thing were running perfect when no EFS. Issue occurred when we started using EFS as PV.

[ivanrezic-maistra]

I have solved the issue by calling "chmod -R 777 logs/" from root folder (outside container). Maybe it isn't best solution, but it works.

[gj-9lt]

I have solved the issue by calling "chmod -R 777 logs/" from root folder (outside container). Maybe it isn't best solution, but it works.

how did you manage to do it. I am using helm chart in which I added volume as "/opt/airflow/logs"

[axk7812]

@gj-9lt you can mount a volume to the pod and create a directory at the root directory of the container. Something like /airflow-logs is what I did

[NoyMiranFyber]

How did you manage to solve it? I occur the same error: "PermissionError: [Errno 13] Permission denied: '/opt/airflow/logs/dag_processor_manager/dag_processor_manager.log'"

[akuzni2]

I specified the following in the values.yaml file

airflow:
  extraVolumeMounts:
    - name: airflow-efs-volume
      mountPath: /airflow-logs

  extraVolumes:
    - name: airflow-efs-volume
      persistentVolumeClaim:
        claimName: airflow-efs-claim

logs:
  path: /airflow-logs/process-logs

With this you need to have a PVC created airflow-efs-claim or name it whatever you want. Also the log path doesn't have to be /airflow-logs you can name it whatever you want. This mounts a volume at the root of the pod. Now that I'm posting this I'm curious if anyone else who's solved this came up with a different solution?

I don't think this issue should be closed yet, I still think it's a bug because the documentation says that the logs can work if written to /opt/airflow/logs.

Quote from the documentation:

NOTE: it is also possible to persist logs by mounting a PersistentVolume to the log directory (/opt/airflow/logs by default) using airflow.extraVolumes and airflow.extraVolumeMounts.

I feel either the documentation needs to get updated and reflect that writing to this path won't work and to use a different one, or the permissions issue should be solved.

[gj-9lt]

When I tried working with the a normal pod(Which will be created by a root user), everything works fine. But with Airflow, things are not good. I even tried giving full permissions to the directory which I was working with, but no luck.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[akuzni2]

Not stale.

[thesuperzapper]

If this is still an issue, please raise it on the new repo: https://github.com/airflow-helm/charts/tree/main/charts/airflow

The helm/charts repo is being archived soon.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[jinnabaalu]

I am running containers in docker, Workaround for this issue is to add the volume for logs

volumes:
  aiflow_logs:
services:
  webserver:
     image:
     volumes:
             ....
        - aiflow_logs:/opt/airflow/logs

[randidwiputra]

To solved this issue, you can change volumes to directory /home/user.
I have changed directory permission chmod -R 777 /opt, but nothing impact still have error.

[leftiness]

I had the same error message. I fixed it by setting the selinux label on the volume in docker-compose.yaml. I don't know if this will fix the issue of the original poster.

error

[brandon@localhost airflow-prototype]$ sudo docker-compose up airflow-init
...
airflow-init_1       | ValueError: Unable to configure handler 'processor': [Errno 13] Permission denied: '/opt/airflow/logs/scheduler'
airflow-init_1       | 
airflow-prototype_airflow-init_1 exited with code 1

troubleshoot

[brandon@localhost airflow-prototype]$ cat .env
AIRFLOW_UID=1000
AIRFLOW_GID=0

[brandon@localhost airflow-prototype]$ sudo docker-compose run --entrypoint="" airflow-init bash
Creating airflow-prototype_airflow-init_run ... done
I have no name!@3c1f666ddbd6:/opt/airflow$ ls -l
total 0
drwxrwxr-x. 1 1000 1000 26 May 21 01:30 dags
drwxrwxr-x. 1 1000 1000  0 May 21 00:33 logs
drwxrwxr-x. 1 1000 1000  0 May 21 00:33 plugins
I have no name!@3c1f666ddbd6:/opt/airflow$ id -u
1000
I have no name!@3c1f666ddbd6:/opt/airflow$ ls dags
ls: cannot open directory 'dags': Permission denied

fix

  volumes:
    - ./dags:/opt/airflow/dags:z
    - ./logs:/opt/airflow/logs:z
    - ./plugins:/opt/airflow/plugins:z

[brandon@localhost airflow-prototype]$ sudo docker-compose up airflow-init
...
airflow-init_1       | Admin user airflow created
airflow-init_1       | 2.0.2
airflow-prototype_airflow-init_1 exited with code 0

environment

[brandon@localhost airflow-prototype]$ cat /etc/fedora-release 
Fedora release 33 (Thirty Three)
[brandon@localhost airflow-prototype]$ sudo docker version
[sudo] password for brandon: 
Client:
 Version:           19.03.13
 API version:       1.40
 Go version:        go1.15.1
 Git commit:        4484c46
 Built:             Fri Oct  2 19:31:30 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.13
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.15.1
  Git commit:       4484c46
  Built:            Fri Oct  2 00:00:00 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.4
  GitCommit:        
 runc:
  Version:          1.0.0-rc93
  GitCommit:        8cadede98333f0007f4218b9123e6fe652662680
 docker-init:
  Version:          0.18.0
  GitCommit:        
[brandon@localhost airflow-prototype]$ sudo docker-compose --version
docker-compose version 1.27.4, build unknown

links

• https://docs.docker.com/storage/bind-mounts/
• https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html

[ralsouza]

have changed directory permission chmod -R 777
This resolved my issue.

[thesuperzapper]

@ralsouza is this still an issue on the new chart (as I have set uid/gid correctly I believe):
https://github.com/airflow-helm/charts/tree/main/charts/airflow

[ntoxlut]

Hi I had the same issue and I got it resolved buy changing permissions inside the containers:

docker exec -u root -ti my_airflow_container_id bash In this case it's the scheduler container - after this you cd out to the outer base dir

Then do "chmod -R 777 /opt"

[nicnguyen3103]

I also have the same problem. I am create a non-root user to run the airflow image and here is the error:

Unable to load the config, contains a configuration error.
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/pathlib.py", line 1284, in mkdir
    self._accessor.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/home/reporter/airflow/logs/scheduler/2021-06-24'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/logging/config.py", line 563, in configure
    handler = self.configure_handler(handlers[name])
  File "/usr/local/lib/python3.8/logging/config.py", line 744, in configure_handler
    result = factory(**kwargs)
  File "/home/reporter/airflow/airflowvirtual/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 46, in __init__
    Path(self._get_log_directory()).mkdir(parents=True, exist_ok=True)
  File "/usr/local/lib/python3.8/pathlib.py", line 1288, in mkdir
    self.parent.mkdir(parents=True, exist_ok=True)
  File "/usr/local/lib/python3.8/pathlib.py", line 1284, in mkdir
    self._accessor.mkdir(self, mode)
PermissionError: [Errno 13] Permission denied: '/home/reporter/airflow/logs/scheduler'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/reporter/airflow/airflowvirtual/bin/airflow", line 6, in <module>
    from airflow.__main__ import main
  File "/home/reporter/airflow/airflowvirtual/lib/python3.8/site-packages/airflow/__init__.py", line 46, in <module>
    settings.initialize()
  File "/home/reporter/airflow/airflowvirtual/lib/python3.8/site-packages/airflow/settings.py", line 446, in initialize
    LOGGING_CLASS_PATH = configure_logging()
  File "/home/reporter/airflow/airflowvirtual/lib/python3.8/site-packages/airflow/logging_config.py", line 73, in configure_logging
    raise e
  File "/home/reporter/airflow/airflowvirtual/lib/python3.8/site-packages/airflow/logging_config.py", line 68, in configure_logging
    dictConfig(logging_config)
  File "/usr/local/lib/python3.8/logging/config.py", line 808, in dictConfig
    dictConfigClass(config).configure()
  File "/usr/local/lib/python3.8/logging/config.py", line 570, in configure
    raise ValueError('Unable to configure handler '
ValueError: Unable to configure handler 'processor'

chmod -R 777 the folder does not work, and the gid, uid and airflowhomepath is set as follow

#values.yaml
# User and group of airflow user
uid: 1014
gid: 1014

# Airflow home directory
# Used for mount paths
airflowHome: /home/reporter/airflow

I wonder why Airflow does not have the permission to the logs folder? Does anyone have similar issue?

[thesuperzapper]

@ntoxlut @nicnguyen3103 if you are having this issue on the newer versions of this chart, which are developed/available here, please raise a new issue on that repo.

[gchamon]

running on my own PC for testing:

mkdir -p dags logs plugins
chown 50000:50000 dags logs plugins

[thesuperzapper]

For those watching, please migrate to the newer versions of the chart on the new airflow-helm/charts repo. If you follow the instructions for "Option 1" under "How to persist logs?", you won't end up with permission errors.

PS: I realise there are a lot of securityContext settings, and will be adding a global config in the next chart version (on the new repo), see issue airflow-helm/charts#296

[marrocksd]

running on my own PC for testing:

mkdir -p dags logs plugins
chown 50000:50000 dags logs plugins

confirm this works for me(docker-compose)

[thesuperzapper]

Just wanted to ping everyone watching this and say the version 8.5.0 of the User-Community Helm Chart just released and these permissions issue should never happen again. There is now a airflow.defaultSecurityContext value with {fsGroup: 0} as default.

NOTE: the user-community chart is maintained descendent of stable/airflow

[romellfudi]

This works for me:

chmod 777 /var/run/docker.sock

[bgsthiago]

I have solved the issue by calling "chmod -R 777 logs/" from root folder (outside container). Maybe it isn't best solution, but it works.

Worked for me, tks!

[NguyenKyThinh94]

I have solved the issue by calling "chmod -R 777 logs/" from root folder (outside container). Maybe it isn't best solution, but it works.

why it can work, cause we have problems permission with /opt/airflow/logs

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[bridgetkromhout]

Hi, folks - as @thesuperzapper explained, this issue is solved in the community-maintained chart available at https://github.com/airflow-helm/charts/.

Just wanted to ping everyone watching this and say the version 8.5.0 of the User-Community Helm Chart just released and these permissions issue should never happen again. There is now a airflow.defaultSecurityContext value with {fsGroup: 0} as default.

NOTE: the user-community chart is maintained descendent of stable/airflow