"Operation not permitted" error when chmod on log folder

[denskh]

Official Helm Chart version

1.7.0 (latest released)

Apache Airflow version

2.5.1

Kubernetes Version

1.24.6

Helm Chart configuration

executor: "KubernetesExecutor" # however same issue happens with LocalExecutor
logs:
persistence:
enabled: true
size: 50Gi
storageClassName: azurefile-csi

Docker Image customizations

Using airflow-2.5.1-python3.10 as a base image.
Copy custom shared libraries into folder under /opt/airflow/company
Copy DAGs /opt/airflow/dags

What happened

After migrating from airflow 2.4.3 to 2.5.1 start getting error below. No other changes to custom image. No task is running because of this error:
Traceback (most recent call last):
  File "/home/airflow/.local/bin/airflow", line 8, in <module>
    sys.exit(main())
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/__main__.py", line 39, in main
    args.func(args)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/cli/cli_parser.py", line 52, in command
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/utils/cli.py", line 108, in wrapper
    return f(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/cli/commands/task_command.py", line 384, in task_run
    ti.init_run_context(raw=args.raw)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/models/taskinstance.py", line 2414, in init_run_context
    self._set_context(self)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/utils/log/logging_mixin.py", line 77, in _set_context
    set_context(self.log, context)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/utils/log/logging_mixin.py", line 213, in set_context
    flag = cast(FileTaskHandler, handler).set_context(value)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/utils/log/file_task_handler.py", line 71, in set_context
    local_loc = self._init_file(ti)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/utils/log/file_task_handler.py", line 382, in _init_file
    self._prepare_log_folder(Path(full_path).parent)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/utils/log/file_task_handler.py", line 358, in _prepare_log_folder
    directory.chmod(mode)
  File "/usr/local/lib/python3.10/pathlib.py", line 1191, in chmod
    self._accessor.chmod(self, mode, follow_symlinks=follow_symlinks)
PermissionError: [Errno 1] Operation not permitted: '/opt/airflow/logs/dag_id=***/run_id=manual__2023-01-22T02:59:43.752407+00:00/task_id=***'

What you think should happen instead

Seem like airflow attempts to set change log folder permissions and not permissioned to do it.
Getting same error when executing command manually (confirmed folder path exists): chmod 511 '/opt/airflow/logs/dag_id=/run_id=manual__2023-01-22T02:59:43.752407+00:00/task_id='
chmod: changing permissions of '/opt/airflow/logs/dag_id=/run_id=scheduled__2023-01-23T15:30:00+00:00/task_id=': Operation not permitted

How to reproduce

My understanding is that this error happens before any custom code is executed.

Anything else

Error happens every time, unable to start any DAG while using airflow 2.5.1. Exactly same configuration works with 2.5.0 and 2.4.3.
Same image and configuration works fine while running locally using docker-composer.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[Taragolis]

I guess it is regression of #28477

[potiuk]

Should be easy fix but it is likely another candidate to speed up 2.5.2 cc: @pierrejeambrun @ephraimbuddy

[ephraimbuddy]

@pingzh do you want to take a look at it?

[nsphung]

I have the same issue with image based on apache/airflow:2.5.1-python3.9. I have updated due to CVE-2023-22884

[potiuk]

You can upgrade MySQL provider and build your image @nsphung if you do not want to wait for the 2.5.2 to release.

[nsphung]

I don't need the provider so for now. I've just uninstalled it in my airflow instances. It's just the Dependabot alerts that is annoying me for this when it sees Airflow < 2.5.1 at the moment.

[mmichaels01]

@denskh Did wind up with a workaround for this?

I'm having the same issue, would like to use the shared logs, but I can disable them until we can upgrade to the 2.5.2 version when that's released

[denskh]

Nope, still on 2.5.0 which doesn't have this issue, hope it gets fixed soon.

[SamWheating]

I have been trying to replicate this issue this morning, here's what I did:

• set up a Kind cluster with a ReadWriteMany PVC (running on NFS)
• Deployed Airflow 2.5.1 with the latest helm chart, writing task logs to said PVC with similar logging config.
• Ran some tasks, which all succeeded.

So I think that it must be an issue of the access / fsGroup on your volume, and might be related to the CSI?

If anyone has any other suggestions, I am happy to work on a fix, but would like to replicate the issue first.

Update: I wasn't using impersonation 🤦 once I set AIRFLOW__CORE__DEFAULT_IMPERSONATION=nobody I was able to replicate the issue.

[potiuk]

Apparently there are cases where changing the permission is impossible. We should ignore the error rather than fail the log file creation in such cases (as it is anyhow only needed to handle a very specific subdag issue).

[thesuperzapper]

@potiuk I am still seeing many users report errors in airflow 2.5.3 as the fix in PR #30123 was not back-ported.

The error is always associated with this line airflow/utils/log/file_task_handler.py#L358, perhaps we can put a temporary fix for 2.5.4 which puts a try/catch around that (and any other chmod attempts)?

Alternatively, can we revert #28477 which is the cause of this problem?