airflow tasks are getting randomly terminated with no errors in UI and logs on worker shows module not found #27332
Description
Official Helm Chart version
1.6.0
Apache Airflow version
airflow 2
Kubernetes Version
1.22
Helm Chart configuration
i have set PYTHONPATH as below in extraENV and verified after login to worker pod
extraEnv: |
- name: PYTHONPATH
value: "/opt/airflow/dags:/opt/airflow"
also have setup PYTHONPATH in docker image
Setting python path for importing dag modules
ENV PYTHONPATH="/opt/airflow/dags:/opt/airflow"
Docker Image customisations
base docker image == FROM apache/airflow:2.3.2-python3.8
also have setup PYTHONPATH in docker image
Setting python path for importing dag modules
ENV PYTHONPATH="/opt/airflow/dags:/opt/airflow"
What happened
dags are showing up on UI however after executing it get terminated and below are logs from worker showing its not able to find python modules -- if we re-run same tasks it succeeds however randomly it fails with below error module not found
_execute_in_fork(command_to_exec, celery_task_id)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/executors/celery_executor.py", line 108, in _execute_in_fork
raise AirflowException(msg)
airflow.exceptions.AirflowException: Celery command failed on host: airflow-worker-1.airflow-worker.airflow.svc.cluster.local with celery_task_id f672c24c-d54c-4401-a1c1-8892d54f90f4
[2022-10-27 18:48:32,001: INFO/MainProcess] Task airflow.executors.celery_executor.execute_command[1fdb829f-755a-4ac0-98c5-362e6b6c8a44] received
[2022-10-27 18:48:32,011: INFO/ForkPoolWorker-15] [1fdb829f-755a-4ac0-98c5-362e6b6c8a44] Executing command in Celery: ['airflow', 'tasks', 'run', 'yipit_fsv_data_export_dag', 'verify_all_exports_were_successful', 'manual__2022-10-27T13:42:55-05:00', '--local', '--subdir', 'DAGS_FOLDER/partner_exports/data_export_dag_builder.py']
[2022-10-27 18:48:32,087: INFO/ForkPoolWorker-15] Filling up the DagBag from /opt/airflow/dags/partner_exports/data_export_dag_builder.py
[2022-10-27 18:48:32,144: ERROR/ForkPoolWorker-15] Failed to import: /opt/airflow/dags/partner_exports/data_export_dag_builder.py
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/dagbag.py", line 317, in parse
loader.exec_module(new_module)
File "", line 843, in exec_module
File "", line 219, in _call_with_frames_removed
File "/opt/airflow/dags/partner_exports/data_export_dag_builder.py", line 20, in
from utils.generic_callbacks import on_failure_callback, on_success_callback
ModuleNotFoundError: No module named 'utils'
[2022-10-27 18:48:32,145: ERROR/ForkPoolWorker-15] [1fdb829f-755a-4ac0-98c5-362e6b6c8a44] Failed to execute task Dag 'yipit_fsv_data_export_dag' could not be found; either it does not exist or it failed to parse..
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/executors/celery_executor.py", line 128, in _execute_in_fork
args.func(args)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 51, in command
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/cli.py", line 99, in wrapper
return f(*args, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 360, in task_run
dag = get_dag(args.subdir, args.dag_id)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/cli.py", line 203, in get_dag
raise AirflowException(
airflow.exceptions.AirflowException: Dag 'yipit_fsv_data_export_dag' could not be found; either it does not exist or it failed to parse.
What you think should happen instead
since PYTHONPATH is set in both docker and helm chart environment variables - dag should be able to find modules
How to reproduce
No response
Anything else
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct