Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MappedOperator setter methods #37828

Merged
merged 1 commit into from
Mar 3, 2024

Conversation

nathadfield
Copy link
Collaborator

@nathadfield nathadfield commented Mar 1, 2024

Closes #37013

Addresses this issue referenced that prevents cluster policies from modifying attributes on dynamically mapped tasks by adding lots of additional setter methods to the MappedOperator.

@nathadfield nathadfield marked this pull request as ready for review March 1, 2024 15:42
@nathadfield nathadfield changed the title fix: adding lots of setter methods fix: adding MappedOperator setter methods Mar 1, 2024
@nathadfield nathadfield changed the title fix: adding MappedOperator setter methods fix: MappedOperator setter methods Mar 1, 2024
@nathadfield nathadfield changed the title fix: MappedOperator setter methods MappedOperator setter methods Mar 1, 2024
@uranusjr
Copy link
Member

uranusjr commented Mar 1, 2024

It might be easier to just make these normal instance attributes like task_id, task_group instead of living one level below.

Copy link
Contributor

@jscheffl jscheffl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not understand the proposal of @uranusjr in the comment but from my review this looks like a good fix for the reported problem. Does not fix my other problem as discussed in #32471 but still good to go.

@jscheffl jscheffl added area:Scheduler including HA (high availability) scheduler type:bug-fix Changelog: Bug Fixes area:core labels Mar 2, 2024
@eladkal eladkal added this to the Airflow 2.8.3 milestone Mar 3, 2024
@uranusjr
Copy link
Member

uranusjr commented Mar 3, 2024

Many set-able members on MappedOperator are extracted from partial_kwargs into their own attributes on the class instead. For example, dag is popped out in OperatorPartial and set separately as its own attribute.

I can clean these up in a separate PR instead though.

@uranusjr uranusjr merged commit 1b4b73e into apache:main Mar 3, 2024
59 checks passed
ephraimbuddy pushed a commit that referenced this pull request Mar 6, 2024
@nathadfield nathadfield deleted the fix/mapped_operator_setter_methods branch March 12, 2024 08:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:core area:Scheduler including HA (high availability) scheduler type:bug-fix Changelog: Bug Fixes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Cluster policies on mapped tasks cause broken DAGs
4 participants