Add cluster-wide config to override per-pool include_deferred filed - #71220
Open
msumit wants to merge 2 commits into
Open
Add cluster-wide config to override per-pool include_deferred filed#71220msumit wants to merge 2 commits into
msumit wants to merge 2 commits into
Conversation
Introduces a new configuration option. When set to or , it fixes the effective value for every pool (including pre-existing pools) and prevents users from setting a conflicting per-pool value via the UI, API, or CLI. The default remains unset, preserving the existing per-pool behavior. Updates the pool model, local/FastAPI API serializers, UI config endpoint and pool form, CLI pool commands, default pool creation, and adds corresponding tests.
msumit
requested review from
XD-DENG,
ashb,
bbovenzi,
bugraoz93,
choo121600,
dheerajturaga,
ephraimbuddy,
guan404ming,
henry3260,
jason810496,
pierrejeambrun,
potiuk,
rawwar,
ryanahamilton,
shubhamraj-git and
vatsrahul1001
as code owners
August 6, 2026 08:04
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Value prop: As a centralised platform team, we don't want end users to give control of this field, especially when they are totally unaware of what
deferredreally means here. For them, pool slots are used to control the load on the downstream systems like Spark and Presto, and it doesn't matter whether that task is deferred or not. So with a cluster level config we'll take control of the behaviour.Changes: Introduced a new configuration option
core.pool_include_deferred. When set toTrueorFalse, it fixes the effective value for every pool (including pre-existing pools) and prevents users from setting a conflicting per-pool value via the UI, API, or CLI.The default remains unset (empty string
""), preserving the existing per-pool behavior.Updated the pool model, local/FastAPI API serializers, UI config endpoint and pool form, CLI pool commands, default pool creation, and added corresponding tests.
TrueFalseWas generative AI tooling used to co-author this PR?
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.