ECS Operator does not support launch type "EXTERNAL"

[094459]

Description

You can run ECS tasks either on EC2 instances or via AWS Fargate, and these will run in AWS. With ECS Anywhere, you are now able to run the same ECS tasks on any host that has the ECS agent - on prem, in another cloud provider, etc. The control plane resides in ECS, but the execution of the task is managed by the ECS agent.

To launch tasks on hosts that are managed by ECS Anywhere, you need to specify a launch type of EXTERNAL. This is currently not supported by the ECS Operator. When you attempt do this, you get an error of unsupported launch type.

The current work around is to use boto3 and create a task and then run it using the correct parameters.

Use case/motivation

The ability to run your task code to support hybrid and multi-cloud orchestration scenarios.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

Maybe you would like to create a PR about it @094459 ? Sounds like an easy thing to implement.

[094459]

Yup, thanks @potiuk this is exactly what I plan to do. I am currently in the "planning" phase, and will be working with another developer on this. First step is going through all the detailed docs so we understand how to build Airflow from source and any other important things like style guidelines, expectations.

[potiuk]

Cool. Both of you comment here when you are ready and I will assign it to you signalling to others the task is "in the making" :)

[mrichman]

Hey @potiuk ! @094459 and I are on the case :)

[potiuk]

Assigned both of you :)

[094459]

Apologies for the long time since the last update. During the testing phase, I realised that the ECSOperator does actually support the launch type of "EXTERNAL" so this is not a bug or feature that needs to be added to the ECSOperator code (ecs.py). It would probably be beneficial to update the documentation, add a new test case and provide an updated example DAG. I have these ready to go if this is ok.

[potiuk]

Go ahead!

[094459]

Thanks, just going through the docs carefully so I can understand the mechanisms and what is expected.