Celery worker_concurrency affecting to number of pods running with kubernetes_executor

[imjoseangel]

Apache Airflow version

Other Airflow 2 version (please specify below)

If "Other Airflow 2 version" selected, which one?

2.9.3

What happened?

When setting up the following parameters:

    max_active_tasks_per_dag: 24
    max_active_runs_per_dag: 24

The kubernetes executor only throws 16 pods

Changing this

  celery:
    worker_concurrency: 24

It throws 24 pods.

What you think should happen instead?

    max_active_tasks_per_dag: 24
    max_active_runs_per_dag: 24

Should work directly with Kubernetes Executor but not Celery configuration, if so, I see worker_concurrency in the core config

How to reproduce

Set

    max_active_tasks_per_dag: 24
    max_active_runs_per_dag: 24

And keep the

  celery:
    worker_concurrency: 16

Operating System

Kubernetes

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==8.25.0
apache-airflow-providers-apache-kafka==1.6.0
apache-airflow-providers-celery==3.7.2
apache-airflow-providers-cncf-kubernetes==8.3.3
apache-airflow-providers-common-io==1.3.2
apache-airflow-providers-common-sql==1.14.2
apache-airflow-providers-databricks==6.9.0
apache-airflow-providers-dbt-cloud==3.10.0
apache-airflow-providers-docker==3.12.2
apache-airflow-providers-elasticsearch==5.4.1
apache-airflow-providers-fab==1.2.2
apache-airflow-providers-ftp==3.10.0
apache-airflow-providers-google==10.21.0
apache-airflow-providers-grpc==3.5.2
apache-airflow-providers-hashicorp==3.7.1
apache-airflow-providers-http==4.12.0
apache-airflow-providers-imap==3.6.1
apache-airflow-providers-microsoft-azure==10.2.0
apache-airflow-providers-mysql==5.6.2
apache-airflow-providers-odbc==4.6.2
apache-airflow-providers-openlineage==1.9.1
apache-airflow-providers-postgres==5.11.2
apache-airflow-providers-redis==3.7.1
apache-airflow-providers-sendgrid==3.5.1
apache-airflow-providers-sftp==4.10.2
apache-airflow-providers-slack==8.7.1
apache-airflow-providers-smtp==1.7.1
apache-airflow-providers-snowflake==5.6.0
apache-airflow-providers-sqlite==3.8.1
apache-airflow-providers-ssh==3.11.2

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.