-
Notifications
You must be signed in to change notification settings - Fork 14.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Prevent start trigger initialization in scheduler #39585
Prevent start trigger initialization in scheduler #39585
Conversation
d621388
to
657a1cf
Compare
657a1cf
to
926b3d3
Compare
maybe we should mark this experimental. trigger_kwargs being None vs {} does not seem like an explicit / obvious enough way to decide whether to start from trigger. why wouldn't we just look at the |
926b3d3
to
ec0a768
Compare
I'm ok with it. maybe wait for others' comment? if we're to mark it as experimental, where should I do so?
In the previous PR, @uranusjr and I discussed whether we could infer |
529ecc1
to
12fb69d
Compare
445274d
to
dac10f5
Compare
4b0ac4a
to
5f99ce7
Compare
…art_trigger_kwargs
…ss StartTriggerArgs
…her to start task execution from triggerer
8dbe9ff
to
e1805e5
Compare
…n scheduler (apache#39585) * fix(baseoperator): change `start_trigger` into `start_trigger_args` * feat(baseoperator): add `start_from_trigger` as the flag to decide whether to start task execution from triggerer * fix(dagrun): set start_date before deferring task from scheduler
Why
During #38674, I introduced a logic that might run user code in the scheduler here. This PR intent to propose a new logic that could avoid this.
What
StartTriggerArgs
class that should containtrigger_cls
(or maybe it could be renamedtrigger_cls_path
?),tirgger_kwargs
,timeout
, andnext_method
, which are what we need when we run an operator in deferrable mode.suggestion needed:move it toairflow.models.abstractoperator
might not be the best module to put this data class but might need some suggestionairflow.triggers.base
(Thanks @jedcunningham !)start_from_trigger
flag to allow an operator to be executed as normal or from the triggererOperator authors will now need to set the
start_trigger_args
,start_from_trigger
(or maybestart_from_triggerer
?) this one to start execution directly from the triggerer.Note that the mapped operator is still not supported, but a draft PR #39912 has been created.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rst
or{issue_number}.significant.rst
, in newsfragments.