Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Operation not permitted: '/opt/airflow/logs/...' #712

Closed
2 tasks done
obeyda opened this issue Apr 5, 2023 · 3 comments
Closed
2 tasks done

Operation not permitted: '/opt/airflow/logs/...' #712

obeyda opened this issue Apr 5, 2023 · 3 comments
Labels
kind/bug kind - things not working properly lifecycle/stale lifecycle - this is stale

Comments

@obeyda
Copy link
Contributor

obeyda commented Apr 5, 2023

Checks

Chart Version

8.6.1

Kubernetes Version

Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.1", GitCommit:"8f94681cd294aa8cfd3407b8191f6c70214973a4", GitTreeState:"clean", BuildDate:"2023-01-18T15:58:16Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}

Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.6", GitCommit:"08d3594304660f86cfbd17bbb862041b4b75fe6c", GitTreeState:"clean", BuildDate:"2023-02-08T17:22:59Z", GoVersion:"go1.18.6", Compiler:"gc", Platform:"linux/amd64"}

Helm Version

version.BuildInfo{Version:"v3.7.1", GitCommit:"1d11fcb5d3f3bf00dbe6fe31b8412839a96b3dc4", GitTreeState:"clean", GoVersion:"go1.16.9"}

Description

  • persistence enabled
  • used airflow.defaultSecurityContext.fsGroup: 0 without setting any other securityContext
  • followed the "how to persist logs" doc

result: No problem in airflow pods but KubernetesExecutorPods can't write to the logs volume

Relevant Logs

2023-04-05T02:46:39.148471134Z stderr F PermissionError: [Errno 1] Operation not permitted: '/opt/airflow/logs/dag_id=xxx/run_id=manual__2023-04-05T02:46:26.041418+00:00/task_id=xxx_01'

Custom Helm Values

airflow:
  image:
    tag: 2.5.3-python3.10
  executor: "KubernetesExecutor"
  usersUpdate: false
  config:
    AIRFLOW__KUBERNETES__DELETE_WORKER_PODS: "False"
    AIRFLOW__WEBSERVER__COOKIE_SAMESITE: "Lax"
    AIRFLOW__WEBSERVER__ENABLE_PROXY_FIX: "True"
    AIRFLOW__WEBSERVER__EXPOSE_CONFIG: "True"
  defaultSecurityContext:
    fsGroup: 0
  kubernetesPodTemplate:
    podLabels:
      my-label: "true"
logs:
  path: /opt/airflow/logs
  persistence:
    enabled: true
    storageClass: azurefile
    size: 128Gi
scheduler:
  logCleanup:
    enabled: false
workers:
  enabled: false
redis:
  enabled: false
flower:
  enabled: false
dags:
  gitSync:
    enabled: true
    repo: xxxx
    branch: main
    revision: HEAD
    depth: 1
    repoSubPath: ""
    httpSecret: airflow-git-credentials
@obeyda obeyda added the kind/bug kind - things not working properly label Apr 5, 2023
@thesuperzapper
Copy link
Member

@obeyda this failure is caused by a bug in airflow itself that was introduced in version 2.5.1 (and is still present in 2.5.2, and 2.5.3), see here apache/airflow#29112. The problem is that Azure File does not allow chmod, and airflow mistakenly tries to run chmod on log files if they are not already 777.

For now, you might want to do one of the following:

  • use another method of log persistence, for example, the Azure Blob Storage remote provider
  • use airflow 2.5.0 (but this is an older version, which might have separate problems).
  • use airflow 2.4.3 (but downgrading your DB might be problematic).

@stale
Copy link

stale bot commented Jun 9, 2023

This issue has been automatically marked as stale because it has not had activity in 60 days.
It will be closed in 7 days if no further activity occurs.

Thank you for your contributions.


Issues never become stale if any of the following is true:

  1. they are added to a Project
  2. they are added to a Milestone
  3. they have the lifecycle/frozen label

@stale stale bot added the lifecycle/stale lifecycle - this is stale label Jun 9, 2023
@thesuperzapper
Copy link
Member

Closing because this is an upstream issue, your best bet is to NOT use Airflow 2.5.1, 2.5.2, or 2.5.3 (which all have this bug).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug kind - things not working properly lifecycle/stale lifecycle - this is stale
Projects
None yet
Development

No branches or pull requests

2 participants