Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merge all ECS executor configs following recursive python dict update #37137

Merged
merged 5 commits into from
Feb 9, 2024
Merged

Merge all ECS executor configs following recursive python dict update #37137

merged 5 commits into from
Feb 9, 2024

Conversation

o-nikolas
Copy link
Contributor

This PR changes how exec_config works for the ECS executor. Previously the exec_config was only applied to item zero of the containerOverrides list (the container that receives the Airflow task command). This made it impossible for folks to override any configuration at the base or overrides level of the runTask kwargs (see docs for possible configuration here).

The new strategy is to treat the exec_config as an entire run task kwargs config and do a recursive dict update. This allows folks to set any value from a task or DAG.

This behaviour change is allowed due to the feature still being experimental.

fixes #35490

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@boring-cyborg boring-cyborg bot added area:providers provider:amazon-aws AWS/Amazon - related issues labels Feb 1, 2024
@o-nikolas
Copy link
Contributor Author

CC @syedahsn (since I can't seem to add you via the reviewers list)

@o-nikolas o-nikolas mentioned this pull request Feb 1, 2024
Closed
2 tasks
@o-nikolas
Copy link
Contributor Author

@mshober Can you have a look at this PR and see if it satisfies your use case?

syedahsn
syedahsn approved these changes Feb 2, 2024
View reviewed changes
Copy link
Contributor

@syedahsn syedahsn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Funny that that helper function already existed 😅
LGTM. One thing that occurred to me is whether we want to document this design choice. I think the unit test does a very good job of explaining the behaviour. Might be worth putting that into the documentation.

@o-nikolas
Copy link
Contributor Author

One thing that occurred to me is whether we want to document this design choice. I think the unit test does a very good job of explaining the behaviour. Might be worth putting that into the documentation.

I put a new commit in to address this one! @syedahsn

@o-nikolas o-nikolas force-pushed the onikolas/ecs_executor/task_override_all_properties branch from 2a7cb6b to 93f7b6f Compare February 5, 2024 21:43
ferruzzi
ferruzzi approved these changes Feb 6, 2024
View reviewed changes
Copy link
Contributor

@ferruzzi ferruzzi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like all concerns have been addressed 👍

@o-nikolas o-nikolas merged commit 48bfb1a into apache:main Feb 9, 2024
56 checks passed
@o-nikolas o-nikolas deleted the onikolas/ecs_executor/task_override_all_properties branch February 9, 2024 16:43
satish-chinthanippu added a commit to Teradata/airflow that referenced this pull request Feb 9, 2024
@satish-chinthanippu
Squashed commit of the following:
db35783 
commit 58279b8
Author: Satish Ch <satishchinthanippu@gmail.com>
Date:   Sat Feb 10 00:12:42 2024 +0530

    Update teradata.py

commit ed75e3e
Author: Satish Ch <satishchinthanippu@gmail.com>
Date:   Sat Feb 10 00:09:24 2024 +0530

    D401 support in Teradata Provider

commit f56bede
Merge: e859a1d 00ed467
Author: Satish Ch <satishchinthanippu@gmail.com>
Date:   Fri Feb 9 23:58:15 2024 +0530

    Merge remote-tracking branch 'upstream/main' into pr_teradata_release_1.0.0

commit 00ed467
Author: Vincent <97131062+vincbeck@users.noreply.github.com>
Date:   Fri Feb 9 13:13:36 2024 -0500

    D401 support in fab provider (apache#37283)

commit 48bfb1a
Author: Niko Oliveira <onikolas@amazon.com>
Date:   Fri Feb 9 08:43:32 2024 -0800

    Merge all ECS executor configs following recursive python dict update (apache#37137)

    Also document the behaviour and interaction between exec_config and
    run_task_kwargs config

commit 8317ed9
Author: Amogh Desai <amoghrajesh1999@gmail.com>
Date:   Fri Feb 9 20:17:38 2024 +0530

    Updating the README and visuals for breeze build-docs (apache#37276)

commit 17945fc
Author: Kalyan <kalyan.ben10@live.com>
Date:   Fri Feb 9 20:16:33 2024 +0530

    D401 fixes in Pinecone provider (apache#37270)

commit ab9e2e1
Author: Kalyan <kalyan.ben10@live.com>
Date:   Fri Feb 9 20:15:31 2024 +0530

    fix: D401 lint issues in airflow core (apache#37274)

commit 7835fd2
Author: Jarek Potiuk <jarek@potiuk.com>
Date:   Fri Feb 9 14:59:33 2024 +0100

    The fix-ownership command missed --rm flag and left dangling containers (apache#37277)

    Fixes: apache#37269

commit e859a1d
Author: Satish Ch <satishchinthanippu@gmail.com>
Date:   Fri Feb 9 19:21:28 2024 +0530

    Update teradata.py

commit d1c08e1
Author: Satish Ch <satishchinthanippu@gmail.com>
Date:   Fri Feb 9 19:18:37 2024 +0530

    Update teradata.py

commit e5ac0ec
Author: Satish Ch <satishchinthanippu@gmail.com>
Date:   Fri Feb 9 05:41:42 2024 -0800

    static check issue is fixed

commit ce490f7
Author: Satish Ch <satishchinthanippu@gmail.com>
Date:   Fri Feb 9 19:10:36 2024 +0530

    static format issue fixed

commit f9498c5
Author: Satish Ch <satishchinthanippu@gmail.com>
Date:   Fri Feb 9 05:36:36 2024 -0800

    static check issue is fixed

commit 77bddae
Author: Satish Ch <satishchinthanippu@gmail.com>
Date:   Fri Feb 9 18:23:18 2024 +0530

    common sql unit test failure fixed

commit 9f4f208
Author: Aleksey Kirilishin <54231417+avkirilishin@users.noreply.github.com>
Date:   Fri Feb 9 15:53:04 2024 +0300

    Fix the bug that affected the DAG end date. (apache#36144)

commit 0f8dfeb
Author: Sudipto Baral <sudiptobaral.me@gmail.com>
Date:   Fri Feb 9 06:54:23 2024 -0500

    fix: update  hyperlink to the new documentation section for local virtualenv setup. (apache#37272)

    Signed-off-by: sudipto baral <sudiptobaral.me@gmail.com>
This was referenced Feb 12, 2024
Closed
Closed
sunank200 pushed a commit to astronomer/airflow that referenced this pull request Feb 21, 2024
@o-nikolas @sunank200
Merge all ECS executor configs following recursive python dict update (…
b3c3054 
…apache#37137)

Also document the behaviour and interaction between exec_config and
run_task_kwargs config
abhishekbhakat pushed a commit to abhishekbhakat/my_airflow that referenced this pull request Mar 5, 2024
@o-nikolas @abhishekbhakat
Merge all ECS executor configs following recursive python dict update (…
9e9b5c9 
…apache#37137)

Also document the behaviour and interaction between exec_config and
run_task_kwargs config
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Taragolis Taragolis Taragolis left review comments

@ferruzzi ferruzzi ferruzzi approved these changes

@syedahsn syedahsn syedahsn approved these changes

@eladkal eladkal Awaiting requested review from eladkal eladkal is a code owner

Assignees
No one assigned
Labels
area:providers provider:amazon-aws AWS/Amazon - related issues
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

ECS Executor - Overriding Additional ECS Task Properties
4 participants
@o-nikolas @ferruzzi @Taragolis @syedahsn