OSError: [Errno 30] Read-only file system: '/opt/airflow/webserver_config.py'

[ajeet2808]

I am trying to run airflow with containerSecurityContext: readOnlyRootFilesystem as true. And airflow is trying to auto generate webserver_config.py file on runtime. So it's failing with error : OSError: [Errno 30] Read-only file system: '/opt/airflow/webserver_config.py'

Error Log:
Traceback (most recent call last): File "/usr/local/bin/airflow", line 5, in <module> from airflow.__main__ import main File "/usr/local/lib/python3.8/site-packages/airflow/__init__.py", line 34, in <module> from airflow import settings File "/usr/local/lib/python3.8/site-packages/airflow/settings.py", line 37, in <module> from airflow.configuration import AIRFLOW_HOME, WEBSERVER_CONFIG, conf # NOQA F401 File "/usr/local/lib/python3.8/site-packages/airflow/configuration.py", line 1098, in <module> conf = initialize_config() File "/usr/local/lib/python3.8/site-packages/airflow/configuration.py", line 858, in initialize_config shutil.copy(_default_config_file_path('default_webserver_config.py'), WEBSERVER_CONFIG) File "/usr/local/lib/python3.8/shutil.py", line 415, in copy copyfile(src, dst, follow_symlinks=follow_symlinks) File "/usr/local/lib/python3.8/shutil.py", line 261, in copyfile with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst: OSError: [Errno 30] Read-only file system: '/opt/airflow/webserver_config.py'

Please help with this.

[potiuk]

AIRFLOW_HOME is generally dynamically writeable, you will likely have more problems if you keep it read-only. I suggest to mount writeable volume in /opt/airflow.

[ajeet2808]

Thank you!

[malthe]

I think it might be a good idea to take a second look at this because some organizations very much want readOnlyRootFilesystem to be set unless there is a really good reason.

And it seems like Airflow doesn't need to copy those default configs in many/most cases.

[potiuk]

Yeah @malthe. Agree.. Why not. I think we could benefit a lot from clarifying some of the assumptions that Airflow makes. I mostly know the asumptions from running it locally but they are not written down anywhere. Maybe a good idea to describe it in our docs and while describing it, clarify it and describe how Airflow should approach "read"/"writes" part of it's installation.

And maybe change it's behaviour along the way.

Would you maybe like to take a lead on it :) and create PR? I'd love to contribute.

[malthe]

Yes I will take a lead on it because we have it as a hard requirement (well, if it is possible ...)

[tyler-uipath]

Hi @malthe, curious if you ended up making any progress here. as you mention, many enterprise would really like to stay read-only.

[Raghuvarun594]

Hi Team,

Myself facing same problem
I am trying to run airflow with containerSecurityContext: readOnlyRootFilesystem as true. So it's failing with error : OSError: [Errno 30] Read-only file system: '/opt/airflow/airflow.cfg'

reason placing as readonly file system as true in my config file, please see the below error as to bypass the error updated in my config file as readonly true
Error creating : admission webhook "validation.gatekeeper.sh" denied the request: [allow-privilege-escalation] privilege escalation container is not allowed: airflow-webserver-container...

Please help me on this as i got stuck in middle of my deployments

[potiuk]

Generate the file outside of image and extend the image to contain the file. Then airflow will not try to generate it.

[Raghuvarun594]

@potiuk Thanks for your reply

I have breakdown the docker file and mounted as read write for the /opt/airflow filesystem it works and built a new image but after deploying through helm still facing issue read only filesystem, Please find below my contexts in values.yaml file of airflow mentioned as like to overcome the gatekeeper issue, Is this was culprit as mentioned manually from my end so it was not working ? If so i have tried to remove the lines readonly fs and allowpriviligeescalation after deploying again facing old error as Error creating : admission webhook "validation.gatekeeper.sh"

securityContexts:
pod: {}
containers: {}
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true

[potiuk]

No idea. really. can't help more than that at this stage - no time. But airflow chart was never designed to handle root only filestem and validation.gatekeeper.sh is unknwing to me, you can keep on posting more details here if you find them in the hopes someone had similar issue - but I personally have no other answers for you.

[potiuk]

Generally if you need some special support for special case by your organisation that Airflow does not handle, either find a way to switch to standard case that is supported out-of-the-box, or maybe hire/find someone in the organisation admins who will help you to debug it?

[Raghuvarun594]

@potiuk Thank you for ur time, will try to check it out....