Skip to content

fix query to prevent long query when asset count is too much#61519

Open
wjddn279 wants to merge 1 commit intoapache:mainfrom
wjddn279:fix-asset-orphaned-check-query
Open

fix query to prevent long query when asset count is too much#61519
wjddn279 wants to merge 1 commit intoapache:mainfrom
wjddn279:fix-asset-orphaned-check-query

Conversation

@wjddn279
Copy link
Contributor

@wjddn279 wjddn279 commented Feb 6, 2026

related: #61453

Why?

There was an issue where IN operations on a large number of rows in the assets table caused long-running queries.
This commit fixes that issue.

How?

While we could improve this using CTEs, joins, etc., it would make the code messy and introduce side effects such as having to execute the orphan status query multiple times. Therefore, I decided to minimize (eliminate) the IN operations.

The rationale for this logic being equivalent to the original logic is as follows:

  • asset_reference_query groups by the primary key of assets, so it is unique.
  • The sets where orphaned is True and False are complements of each other.
    • The relationship is: not orphaned = active, not active = orphaned.
  • In self._orphan_unreferenced_assets, orphaned rows from active_assets are deleted.
  • Consequently, the IN operation performed in self._activate_referenced_assets is unnecessary.
    • This is because all rows that satisfy the negation of that condition have already been deleted.
    • In most cases, the number of orphaned assets is much smaller, so this is a significant performance improvement.

More?

Nevertheless, if there are too many orphaned assets, performance degradation may still occur at this point.

session.execute(
    delete(AssetActive).where(
        tuple_(AssetActive.name, AssetActive.uri).in_((a.name, a.uri) for a in assets)
    )
)

If the number of orphaned assets is large (above a certain threshold), it would be good to display a DAG warning.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@wjddn279 wjddn279 requested review from XD-DENG and ashb as code owners February 6, 2026 09:26
@boring-cyborg boring-cyborg bot added the area:Scheduler including HA (high availability) scheduler label Feb 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:Scheduler including HA (high availability) scheduler

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant