Operation not permitted: '/opt/airflow/logs/...'

[obeyda]

Checks

• I have checked for existing issues.
• This report is about the User-Community Airflow Helm Chart.

Chart Version

8.6.1

Kubernetes Version

Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.1", GitCommit:"8f94681cd294aa8cfd3407b8191f6c70214973a4", GitTreeState:"clean", BuildDate:"2023-01-18T15:58:16Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}

Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.6", GitCommit:"08d3594304660f86cfbd17bbb862041b4b75fe6c", GitTreeState:"clean", BuildDate:"2023-02-08T17:22:59Z", GoVersion:"go1.18.6", Compiler:"gc", Platform:"linux/amd64"}

Helm Version

version.BuildInfo{Version:"v3.7.1", GitCommit:"1d11fcb5d3f3bf00dbe6fe31b8412839a96b3dc4", GitTreeState:"clean", GoVersion:"go1.16.9"}

Description

• persistence enabled
• used airflow.defaultSecurityContext.fsGroup: 0 without setting any other securityContext
• followed the "how to persist logs" doc

result: No problem in airflow pods but KubernetesExecutorPods can't write to the logs volume

Relevant Logs

2023-04-05T02:46:39.148471134Z stderr F PermissionError: [Errno 1] Operation not permitted: '/opt/airflow/logs/dag_id=xxx/run_id=manual__2023-04-05T02:46:26.041418+00:00/task_id=xxx_01'

Custom Helm Values

airflow:
  image:
    tag: 2.5.3-python3.10
  executor: "KubernetesExecutor"
  usersUpdate: false
  config:
    AIRFLOW__KUBERNETES__DELETE_WORKER_PODS: "False"
    AIRFLOW__WEBSERVER__COOKIE_SAMESITE: "Lax"
    AIRFLOW__WEBSERVER__ENABLE_PROXY_FIX: "True"
    AIRFLOW__WEBSERVER__EXPOSE_CONFIG: "True"
  defaultSecurityContext:
    fsGroup: 0
  kubernetesPodTemplate:
    podLabels:
      my-label: "true"
logs:
  path: /opt/airflow/logs
  persistence:
    enabled: true
    storageClass: azurefile
    size: 128Gi
scheduler:
  logCleanup:
    enabled: false
workers:
  enabled: false
redis:
  enabled: false
flower:
  enabled: false
dags:
  gitSync:
    enabled: true
    repo: xxxx
    branch: main
    revision: HEAD
    depth: 1
    repoSubPath: ""
    httpSecret: airflow-git-credentials

[thesuperzapper]

@obeyda this failure is caused by a bug in airflow itself that was introduced in version 2.5.1 (and is still present in 2.5.2, and 2.5.3), see here apache/airflow#29112. The problem is that Azure File does not allow chmod, and airflow mistakenly tries to run chmod on log files if they are not already 777.

For now, you might want to do one of the following:

• use another method of log persistence, for example, the Azure Blob Storage remote provider
• use airflow 2.5.0 (but this is an older version, which might have separate problems).
• use airflow 2.4.3 (but downgrading your DB might be problematic).

[stale]

This issue has been automatically marked as stale because it has not had activity in 60 days.
It will be closed in 7 days if no further activity occurs.

Thank you for your contributions.

---

Issues never become stale if any of the following is true:

1. they are added to a Project
2. they are added to a Milestone
3. they have the lifecycle/frozen label

[thesuperzapper]

Closing because this is an upstream issue, your best bet is to NOT use Airflow 2.5.1, 2.5.2, or 2.5.3 (which all have this bug).