Celery worker having issues with a seemingly code error

[xadaemon]

Apache Airflow version

2.2.3 (latest released)

What happened

When we try to run dags we get the error described bellow, I'm at a loss for further diagnosis of this issue and thus decided to open this issue, I'm happy to provide any extra details or even help with a fix.

Regards

What you expected to happen

When we try to run any dag with the celery executor we get the following error (includes the built-in example dags):

[2022-01-20 18:22:09,694: ERROR/ForkPoolWorker-16] Failed to execute task return_code() got an unexpected keyword argument 'timeout'.
Traceback (most recent call last):
  File "/opt/airflow/.env/lib/python3.8/site-packages/airflow/executors/celery_executor.py", line 121, in _execute_in_fork
    args.func(args)
  File "/opt/airflow/.env/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 48, in command
    return func(*args, **kwargs)
  File "/opt/airflow/.env/lib/python3.8/site-packages/airflow/utils/cli.py", line 92, in wrapper
    return f(*args, **kwargs)
  File "/opt/airflow/.env/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 298, in task_run
    _run_task_by_selected_method(args, dag, ti)
  File "/opt/airflow/.env/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 105, in _run_task_by_selected_method
    _run_task_by_local_task_job(args, ti)
  File "/opt/airflow/.env/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 163, in _run_task_by_local_task_job
    run_job.run()
  File "/opt/airflow/.env/lib/python3.8/site-packages/airflow/jobs/base_job.py", line 245, in run
    self._execute()
  File "/opt/airflow/.env/lib/python3.8/site-packages/airflow/jobs/local_task_job.py", line 129, in _execute
    return_code = self.task_runner.return_code(timeout=max_wait_time)
TypeError: return_code() got an unexpected keyword argument 'timeout'
[2022-01-20 18:22:09,694: ERROR/ForkPoolWorker-15] Failed to execute task return_code() got an unexpected keyword argument 'timeout'.
Traceback (most recent call last):
  File "/opt/airflow/.env/lib/python3.8/site-packages/airflow/executors/celery_executor.py", line 121, in _execute_in_fork
    args.func(args)
  File "/opt/airflow/.env/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 48, in command
    return func(*args, **kwargs)
  File "/opt/airflow/.env/lib/python3.8/site-packages/airflow/utils/cli.py", line 92, in wrapper
    return f(*args, **kwargs)
  File "/opt/airflow/.env/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 298, in task_run
    _run_task_by_selected_method(args, dag, ti)
  File "/opt/airflow/.env/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 105, in _run_task_by_selected_method
    _run_task_by_local_task_job(args, ti)
  File "/opt/airflow/.env/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 163, in _run_task_by_local_task_job
    run_job.run()
  File "/opt/airflow/.env/lib/python3.8/site-packages/airflow/jobs/base_job.py", line 245, in run
    self._execute()
  File "/opt/airflow/.env/lib/python3.8/site-packages/airflow/jobs/local_task_job.py", line 129, in _execute
    return_code = self.task_runner.return_code(timeout=max_wait_time)
TypeError: return_code() got an unexpected keyword argument 'timeout'
[2022-01-20 18:22:09,784: ERROR/ForkPoolWorker-15] Task airflow.executors.celery_executor.execute_command[5f812025-28ff-4805-b876-680175d0e51c] raised unexpected: AirflowException('Celery command failed on host: airflow-act.internal.cloudapp.net')
Traceback (most recent call last):
  File "/opt/airflow/.env/lib/python3.8/site-packages/celery/app/trace.py", line 451, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/opt/airflow/.env/lib/python3.8/site-packages/celery/app/trace.py", line 734, in __protected_call__
    return self.run(*args, **kwargs)
  File "/opt/airflow/.env/lib/python3.8/site-packages/airflow/executors/celery_executor.py", line 90, in execute_command
    _execute_in_fork(command_to_exec, celery_task_id)
  File "/opt/airflow/.env/lib/python3.8/site-packages/airflow/executors/celery_executor.py", line 101, in _execute_in_fork
    raise AirflowException('Celery command failed on host: ' + get_hostname())
airflow.exceptions.AirflowException: Celery command failed on host: airflow-act.internal.cloudapp.net
[2022-01-20 18:22:09,784: ERROR/ForkPoolWorker-16] Task airflow.executors.celery_executor.execute_command[2911ab23-945b-4526-9971-aadf20a04536] raised unexpected: AirflowException('Celery command failed on host: airflow-act.internal.cloudapp.net')
Traceback (most recent call last):
  File "/opt/airflow/.env/lib/python3.8/site-packages/celery/app/trace.py", line 451, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/opt/airflow/.env/lib/python3.8/site-packages/celery/app/trace.py", line 734, in __protected_call__
    return self.run(*args, **kwargs)
  File "/opt/airflow/.env/lib/python3.8/site-packages/airflow/executors/celery_executor.py", line 90, in execute_command
    _execute_in_fork(command_to_exec, celery_task_id)
  File "/opt/airflow/.env/lib/python3.8/site-packages/airflow/executors/celery_executor.py", line 101, in _execute_in_fork
    raise AirflowException('Celery command failed on host: ' + get_hostname())
airflow.exceptions.AirflowException: Celery command failed on host: airflow-act.internal.cloudapp.net

How to reproduce

We have installed airflow as per the documentation with the following pip command.

Operating System

Ubuntu 20.04.3 LTS (Focal Fossa)

Versions of Apache Airflow Providers

apache-airflow-providers-celery==2.1.0
apache-airflow-providers-ftp==2.0.1
apache-airflow-providers-http==2.0.1
apache-airflow-providers-imap==2.0.1
apache-airflow-providers-jdbc==2.0.1
apache-airflow-providers-postgres==2.4.0
apache-airflow-providers-redis==2.0.1
apache-airflow-providers-sqlite==2.0.1

Deployment

Virtualenv installation

Deployment details

No response

Anything else

Every time we try to run a DAG.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

I believe your installation has gone somewhat wrong - almost looks like you have mulltple versions of airflow installed at the same time. I suggest you reinstall you Airlfow installation from scratch (alsol you have not really posted how you installed airflow). Converting it into discussion.

[xadaemon]

Took us a while, and we even submitted a patch, check for a hostname missmatch between what airflow is configured to get a hostname from and the hostname command on the host.

[Vishal2696]

@neonimp Appreciate the reply, I checked the hostname on the host, but what do you mean by "what airflow is configured to get a hostname from". Is there any config where we need to pass the hostname to webserver? I'm not aware of such config, is it possible for you to point out the exact property you are referring to here?

[potiuk]

https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#hostname-callable

[xadaemon]

Sorry, ended up missing your reply, it was in the end a mismatch between what the function set in the property @potiuk pointed out and what hostname -f reported, in our case it was caused by azure specific problems.

[xadaemon]

Actually, now I'm unsure if this is the same problem, the timeline of the issues we had is a bit fuzzy in my mind, I will check my notes here and reply. We had another issue where a re-install fixed our problems.

[pingzh]

@neonimp what task runner do you use?

the screenshot is based on the 2.2.3 version and i also confirmed that in the main branch, it is same.

you can find it in your airflow.cfg, task_runner =

@potiuk it looks like it is a bug, the def return_code interface is different in those classes.

[xadaemon]

We have since moved on to a Kubernetes based approach, since that was a validation stage of the implementation project, I have asked a fellow engineer that was closer to it and they said we were using celery and the default runner if memory didn't fail us.