DAGs remain in the UI after renaming the dag_id in the same python file

[wolfier]

Apache Airflow version: 1.10.12

What happened:

When I rename the dag_id, the new dag_id shows up in the UI but the old dag_id does not disappear and remains in the UI.

What you expected to happen:

When Airflow lists out the python files and tries to deactivate the deleted dags during dag processing, the old DAG's file location is still in the list of the alive DAG location because the new DAG is now defined in the old DAG's python file. Even when you manually set is_active to False in the metastore, the dag processing process will set it back to True.

This happens whether or not DAG serialization is enabled.

How to reproduce it:

I was able to reproduce it by following these steps:

1. Add a python file in the dags folder and initialize the DAG
2. Verify the dag shows up in the UI
3. Change the dag_id in the same file to something else
4. Verify the new dag shows up in the UI
5. Both the old DAG and the new DAG are visible on the UI as well as the metadata db.

➜  dags pwd
/Users/alan/projects/astro/dags
➜  dags ls
wow.py

from datetime import datetime

from airflow.models import DAG


dag = DAG(
    dag_id='yay',
    schedule_interval='@once',
    start_date=datetime(2020, 1, 1),
    catchup=False
)

In the screenshot, both DAGs are marked as active but there only one DAG defined in wow.py.

Anything else we need to know:

[kaxil]

This needs fixing in Master and is not that critical, so will fix it in 2.0.0

[potiuk]

Hey @kaxil @ashb - are we going to fix it ? Or maybe it already is fixed with the serialization ?

[mattforbes-gr]

I believe this is still an issue with serialized dags. We are dynamically generating dags and had to resort to randomizing our dag py file name to force stale serialized dags to be deleted.

[ashb]

Hey @kaxil @ashb - are we going to fix it ? Or maybe it already is fixed with the serialization ?

No, still remains to be fixed.

[leonsmith]

Started to debug this a little.

I’m thinking the problem is the wrong assumption that you can set is_active=True on any dags with the same file path.
So the fix should be patching out the else statement here:
https://github.com/apache/airflow/blob/1.10.12/airflow/models/dag.py#L1917

I think is_active is set to True in the DagBag.sync_to_db func when parsing the dags.

Going to run a bit of a test & will update to see if that resolves the issue

[HaloKo4]

if the expected behavior is to remove/rename the old dag_id how are we expected to read its past logs?
This will require to also modify past log files in the disk otherwise they will be unaccessable.

[kaxil]

For now, you will have to just use a different file for the new (or renamed) DAG.