-
Notifications
You must be signed in to change notification settings - Fork 14.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix accessing rendered task.x
attributes from within templates
#18516
Conversation
@turbaszek -> I am not 100% sure if I have thought about all side effects, but I see no problem with having the locked-down copy of the task in the context rather than the original one (unless there were some reasons why we wanted to keep a different copy in the context - but I believe it was not intentional) |
d44b4df
to
cec7015
Compare
Also - with the last change you do not need to pass task to the _execute method. |
cec7015
to
2101fec
Compare
And the last version removes the need for task variable at all (it was assigned from self.task few lines above 😕 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease. |
I also tested the task from #13559 with the most recent changes: |
2101fec
to
c406b32
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will look in detail on Monday
When we are referring ``{{task}}` in jinja templates we can also refer some of the fields, which are templated. We are not able to solve all the problems with such rendering (specifically recursive rendering of the fields used in JINJA templating might be problematic. Currently whether you see original, or rendered field depends solely on the sequence in templated_fields. However that would not even explain the rendering problem described in apache#13559 where kwargs were defined after opargs and the rendering of opargs **should** work. It turned out that the problem was with a change introduced in apache#8805 which made the context effectively holds a DIFFERENT task than the current one. Context held an original task, and the curren task was actually a locked copy of it (to allow resolving upstream args before locking). As a result, any changes done by rendering templates were not visible in the task accessed via {{ task }} jinja variable. This change replaces the the task stored in context with the same copy that is then used later during execution so that at least the "sequential" rendering works and templated fields which are 'earlier' in the list of templated fields can be used (and render correctly) in the following fields. Fixes: apache#13559
c406b32
to
edb8d9d
Compare
Rebased to make use of the more stable (hopefully) main with merged flakiness fixes :) |
kind reminder @ashb :) |
Sorry house full of back to school colds here. Might get to it tomorrow but if @turbaszek is okay don't wait for me |
Can wait - I think would be good to get it to 2.2 but I think this is still "default" so I am not in a hurry. |
task.x
attributes from within templates
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yup all good!
When we are referring
{{task}}
in jinja templates we can alsorefer some of the fields, which are templated. We are not
able to solve all the problems with such rendering (specifically
recursive rendering of the fields used in JINJA templating might
be problematic. Currently whether you see original, or rendered
field depends solely on the sequence in templated_fields.
However that would not even explain the rendering problem
described in #13559 where kwargs were defined after opargs and
the rendering of opargs should work. It turned out that
the problem was with a change introduced in #8805 which made
the context effectively holds a DIFFERENT task than the current
one. Context held an original task, and the curren task was
actually a locked copy of it (to allow resolving upstream
args before locking). As a result, any changes done by
rendering templates were not visible in the task accessed
via {{ task }} jinja variable.
This change replaces the the task stored in context with the
same copy that is then used later during execution so that
at least the "sequential" rendering works and templated
fields which are 'earlier' in the list of templated fields
can be used (and render correctly) in the following fields.
Fixes: #13559
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.