Dataset list remembers old datasets

[MatrixManAtYrService]

Apache Airflow version

main (development)

What happened

Here's a simple pair of dags linked by a dataset:

from datetime import datetime
from airflow import Dataset
from airflow.operators.empty import EmptyOperator
from airflow.decorators import dag

dataset = Dataset("dataset")

@dag(start_date=datetime(1970, 1, 1))
def upstream():
    EmptyOperator(task_id="empty", outlets=[dataset])

upstream()


@dag(start_date=datetime(1970, 1, 1), schedule=[dataset])
def downstream():
    EmptyOperator(task_id="empty")

downstream()

I let airflow parse this, then I made the following edit:

d̶a̶t̶a̶s̶e̶t̶ ̶=̶ ̶D̶a̶t̶a̶s̶e̶t̶(̶"̶d̶a̶t̶a̶s̶e̶t̶"̶)̶
dataset = Dataset("dataset1")

I let airflow parse it again, then I made another edit:

d̶a̶t̶a̶s̶e̶t̶ ̶=̶ ̶D̶a̶t̶a̶s̶e̶t̶(̶"̶d̶a̶t̶a̶s̶e̶t̶1̶"̶)̶
dataset = Dataset("dataset2")

Then I viewed the datasets page:

Notice that instead of updating the name of the dataset, as I intended, I ended up creating three datasets.

What you think should happen instead

If there are no dags which reference a dataset, it should not be shown in the dataset list

How to reproduce

update a dataset URI

Operating System

docker/debian

Versions of Apache Airflow Providers

n/a

Deployment

Astronomer

Deployment details

astro dev start

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[pierrejeambrun]

Hello,

@jedcunningham, @uranusjr, I would be glad to help on this one :)

Should we add query strings to the endpoint to be able to filter out datasets that do not have any consuming_dags and producing_tasks?

[jedcunningham]

It's probably worth waiting until #26358 is done first.

That said, I'm a little torn on just not showing they at all. Having a way to toggle the behavior might make sense, maybe something like on the homepage with pause/unpaused?

All | Active | Never Updated | Orphaned

[blag]

I like the idea of separating out dataset update statuses into separate tabs/categories.