Skip to content

Add cluster-wide config to override per-pool include_deferred filed - #71220

Open
msumit wants to merge 2 commits into
mainfrom
include_deferred_conf
Open

Add cluster-wide config to override per-pool include_deferred filed#71220
msumit wants to merge 2 commits into
mainfrom
include_deferred_conf

Conversation

@msumit

@msumit msumit commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Value prop: As a centralised platform team, we don't want end users to give control of this field, especially when they are totally unaware of what deferred really means here. For them, pool slots are used to control the load on the downstream systems like Spark and Presto, and it doesn't matter whether that task is deferred or not. So with a cluster level config we'll take control of the behaviour.

Changes: Introduced a new configuration option core.pool_include_deferred. When set to True or False, it fixes the effective value for every pool (including pre-existing pools) and prevents users from setting a conflicting per-pool value via the UI, API, or CLI.

The default remains unset (empty string ""), preserving the existing per-pool behavior.

Updated the pool model, local/FastAPI API serializers, UI config endpoint and pool form, CLI pool commands, default pool creation, and added corresponding tests.

  • When config is set to True
Screenshot 2026-08-06 at 10 04 57 AM
  • When config is set to False
Screenshot 2026-08-06 at 10 06 03 AM
  • When someone tries to set an opposite value via API
Screenshot 2026-08-06 at 12 48 05 PM
  • When API doesn't contain any info regarding to this flag
Screenshot 2026-08-06 at 12 48 37 PM
Was generative AI tooling used to co-author this PR?
  • Yes - Generated by Claude

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

Introduces a new  configuration option. When set
to  or , it fixes the effective  value for every
pool (including pre-existing pools) and prevents users from setting a
conflicting per-pool value via the UI, API, or CLI. The default remains unset,
preserving the existing per-pool behavior.

Updates the pool model, local/FastAPI API serializers, UI config endpoint and
pool form, CLI pool commands, default pool creation, and adds corresponding
tests.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant