Setting max_active_runs = 1 - still have multiple instances of the same dag running

[denysivanov]

Apache Airflow version

2.1.3

Operating System

Linux

Versions of Apache Airflow Providers

apache-airflow-providers-microsoft-azure==3.1.1
apache-airflow-providers-microsoft-mssql==2.0.1
apache-airflow-providers-odbc==2.0.1
apache-airflow-providers-slack==4.0.1

Deployment

Other 3rd-party Helm chart

Deployment details

We are using this helm chart
https://github.com/airflow-helm/charts/tree/main/charts/airflow

What happened

I set max_active_runs=1 and still see multiple instances of the same dag running

with DAG(
    dag_id=dag_helper.get_dag_id_from_path(__file__),
    default_args=dag_helper.get_dag_default_args(),
    schedule_interval=dag_helper.set_schedule_interval(prod_interval="30 13,16,19 * * *", qa_interval="*/5 * * * *"),
    max_active_runs=1,
    start_date=days_ago(2),
    tags=dag_helper.get_tags(__file__)) as dag:
    inbound_dag_flat_file_processor(inbound_dag_config
)

At the end of my current dag I am trigger instance of current dag one more time

def trigger_current_dag(trigger: bool):
    if trigger:
        session = settings.Session
        dag_bag = DagBag(settings.DAGS_FOLDER)
        context = get_current_context()
        trigger_dag = dag_bag.get_dag(context['dag_run'].dag_id)
        execution_date = datetime.now(pytz.timezone('utc'))
        trigger_dag.create_dagrun(
            run_id='trig__' + execution_date.isoformat(),
            execution_date=execution_date,
            state=State.RUNNING,
            external_trigger=True,
        )
        session.commit()
        session.close()
    else:
        logging.info("No DagRun created")

What you expected to happen

Should have only one active running instance.
New instance of dag should start in the queued state.

According this this one #16401 this problem should be fixed in v2.1.3

How to reproduce

We start instance of dag inside of the current dag

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[Jorricks]

Hello @denysivanov,

In the example code you submitted, you create a DagRun with status Running. However, this is done directly on the database and thereby you could consider this to be a manual overwrite.

The limits, max_active_runs, are enforced by the scheduler component of Airflow. This component will create DagRuns based on the schedule and put them in the queued state first. It will only mark them as running if that will not surpass any thresholds like the max_active_runs.

By completely skipping those scheduler steps, there is no way for Airflow to enforce the limit on max_active_runs. Hence, I propose you change the code you showed to set the DagRun to Queued state instead of Running. That should fix your issue :)

[denysivanov]

@Jorricks doing it now!
Will update you on status tomorrow September 23, 2021.

[ephraimbuddy]

Also @denysivanov , upgrade to 2.1.4, there’s a bug on max_active_runs that have been fixed. Not related to your issue though. @Jorricks answered you correctly

[denysivanov]

@Jorricks it is working in QA will try in PROD now
Appreciate your help here