Implement comm handlers for callback supervisor#65269
Open
ferruzzi wants to merge 3 commits intoapache:mainfrom
Open
Implement comm handlers for callback supervisor#65269ferruzzi wants to merge 3 commits intoapache:mainfrom
ferruzzi wants to merge 3 commits intoapache:mainfrom
Conversation
kaxil
reviewed
Apr 15, 2026
| # This is a minimal subset of ToSupervisor: read-only access to Connections, | ||
| # Variables, XCom, and Assets, plus MaskSecret for the secrets masker. | ||
| CallbackToSupervisor = Annotated[ | ||
| GetAssetByName | GetAssetByUri | GetConnection | GetVariable | GetXCom | MaskSecret, |
Member
There was a problem hiding this comment.
GetXCom requires dag_id, run_id, and task_id, but callbacks (deadline callbacks especially) do not have an implicit task context. A user calling XCom.get() inside a callback would need to know and pass these explicitly. Might be worth a note in the CallbackToSupervisor comment explaining that XCom access requires explicit identifiers since there is no task context -- or documenting which message types are practical in callbacks vs included for completeness.
Contributor
Author
There was a problem hiding this comment.
I actually debated including this at all. Do you think it's useful to have in a callback?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This enables limited API access from within the callback supervisor. With this change, callbacks can use GetVariable, GetConnection, GetXCom, GetAsset (ByName and ByUri), and MaskSecret. We can add more as they are requested, but this felt like a good foundation. These are common with the existing TaskInstance supervisor so those are extracted out into helpers in a new shared request_handlers.py location.
One thing to note is that Triggerer previously did not mask secrets
conn.passwordorconn.extraand will now, bringing it inline with the other two. This is a change but felt like a bugfix since it's reasonable to assume that passwords, etc are masked by default.Testing:
One way to test this manually, create a new dag with a custom callback:
In your dag bag:
place the get_variable callback in a file in your plugins directory:
launch Airflow or Breeze and add the Variable named
test_keyvia the UI or CLI (note that setting it via EnvVar is a different code path and won't hit this change).Run the Dag. Since the fixed_datetime deadline is in the past, it will fire the callback immediately and you should see the success message bubble up into the Scheduler logs within a few seconds.
@kaxil - This is the final step to enable the Connections, Variables, etc that I believe you (and others) were asking about now that the previous PRs are merged.
Followup to #62645
Was generative AI tooling used to co-author this PR?
Some assistance from Cline/Claude