Skip to content

Tasks taking long to get executed #36795

@SalvaChiLlo

Description

@SalvaChiLlo

Apache Airflow version

2.8.0

If "Other Airflow 2 version" selected, which one?

No response

What happened?

Tasks are taking to long to get executed, workers receive the order to execute a task but once the workers execute the command, it takes several seconds for the actual task to get executed. For example, example_complex dag is taking between a minute and a half and two minutes to complete, but the actual task execution time for each of the tasks is less than a second.

I have custom workers that receive tasks from kafka and execute them.

I'll append a few logs:

What you think should happen instead?

Tasks take less than a second to get actually executed, but the total amount of time executing the dag is much bigger than expected.

How to reproduce

I just try to execute the example_complex dag although the described behaviour is the same for every other dag.

Operating System

Debian GNU/Linux 12 (bookworm)

Versions of Apache Airflow Providers

pip freeze | grep providers
apache-airflow-providers-amazon==8.16.0
apache-airflow-providers-celery==3.5.0
apache-airflow-providers-cncf-kubernetes==7.11.0
apache-airflow-providers-common-io==1.1.0
apache-airflow-providers-common-sql==1.9.0
apache-airflow-providers-docker==3.8.2
apache-airflow-providers-elasticsearch==5.3.0
apache-airflow-providers-ftp==3.7.0
apache-airflow-providers-google==10.12.0
apache-airflow-providers-grpc==3.4.0
apache-airflow-providers-hashicorp==3.6.0
apache-airflow-providers-http==4.8.0
apache-airflow-providers-imap==3.5.0
apache-airflow-providers-microsoft-azure==8.4.0
apache-airflow-providers-mysql==5.5.0
apache-airflow-providers-odbc==4.2.0
apache-airflow-providers-openlineage==1.3.0
apache-airflow-providers-postgres==5.9.0
apache-airflow-providers-redis==3.5.0
apache-airflow-providers-sendgrid==3.4.0
apache-airflow-providers-sftp==4.8.0
apache-airflow-providers-slack==8.5.0
apache-airflow-providers-snowflake==5.2.0
apache-airflow-providers-sqlite==3.6.0
apache-airflow-providers-ssh==3.9.0

Deployment

Official Apache Airflow Helm Chart

Deployment details

My deployment is pretty similar to the one using docker compose but a big difference is that I have a Kafka executor and workers, for discarding I made several tests to ensure that they are not introducing unexpected delays.

Anything else?

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:corekind:bugThis is a clearly a bugneeds-triagelabel for new issues that we didn't triage yet

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions