Abnormal Airflow Scheduler Behavior v2.1.3

[raefezzeldin]

Apache Airflow version

2.1.3 (latest released)

Operating System

PRETTY_NAME="Debian GNU/Linux 10 (buster)" NAME="Debian GNU/Linux" VERSION_ID="10" VERSION="10 (buster)" VERSION_CODENAME=buster ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/"

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==1.4.0
apache-airflow-providers-apache-beam==2.0.0
apache-airflow-providers-apache-druid==1.1.0
apache-airflow-providers-apache-spark==2.0.0
apache-airflow-providers-celery==2.0.0
apache-airflow-providers-cncf-kubernetes==2.0.2
apache-airflow-providers-docker==2.1.0
apache-airflow-providers-elasticsearch==2.0.2
apache-airflow-providers-ftp==2.0.0
apache-airflow-providers-google==5.0.0
apache-airflow-providers-grpc==2.0.0
apache-airflow-providers-hashicorp==2.0.0
apache-airflow-providers-http==2.0.0
apache-airflow-providers-imap==2.0.0
apache-airflow-providers-jdbc==1.0.1
apache-airflow-providers-microsoft-azure==2.0.0
apache-airflow-providers-microsoft-mssql==2.0.0
apache-airflow-providers-mysql==1.1.0
apache-airflow-providers-odbc==2.0.0
apache-airflow-providers-postgres==1.0.2
apache-airflow-providers-redis==2.0.0
apache-airflow-providers-sendgrid==2.0.0
apache-airflow-providers-sftp==2.1.0
apache-airflow-providers-slack==4.0.0
apache-airflow-providers-snowflake==2.0.0
apache-airflow-providers-sqlite==2.0.0
apache-airflow-providers-ssh==2.1.0

Deployment

Official Apache Airflow Helm Chart

Deployment details

Azure kubernetes Service
k8s v 1.18.14

What happened

I am runing tow task one MSSQL Operator and the the other is python operator sleep task

default _args I am using

default_args = {
    'start_date': datetime(2021, 8, 14),
    'depends_on_past': False,
    'retry_delay': timedelta(seconds=3),
    'retries': 3
}

Dag Object

with DAG(dag_id='update_subs_inc'
         , default_args=default_args
         , schedule_interval='0/15 * * * *'
         , catchup=True
         , tags=['microsoftsql', 'incrementals', 'ro', 'hourly']
         , max_active_runs=1) as dag:

What you expected to happen

to run one dag run and wait for the sleep task ( 5mins) then run the next dag run. but what happens many dag runs run in parallel and alot of dag runs are queued. as shown in the below figure. that doesn't happen before upgreade to v 2.1.3 , I was using v 2.1.1

How to reproduce

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[ephraimbuddy]

I think it's resolved in #17786 can you test in main?

[github-actions]

This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.

[github-actions]

This issue has been closed because it has not received response from the issue author.