ECS Operator: Capture container execution logs in Airflow task execution log

[virendhar]

Description

Objective:

Add functionality to Airflow ECSOperator to capture the task execution logs as they happen from AWS CloudWatch to the ECSOperator's task execution logs.

Use case/motivation

Present Challenge:

When executing Fargate tasks via ECSOperator the logs are not readily available in Airflow task during and after execution. The final status of the container execution is reflected in Airflow ECS Operator task logs. As a result one should navigate the task logs in Airflow to get a handle of the container log path in AWS Cloudwatch and then access the log from AWS Cloudwatch.

The current behavior introduces the following challenges,

• The ongoing/final execution logs are not captured in Airflow task ( ECS Operator ) log. This typically involves couple of additional steps - which can be avoided.
• The visibility is limited to the final status of the task execution

Solution

Mechanism to relay the logs from AWS CloudWatch during execution to Airflow logs as and when they are happening.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[virendhar-aws]

This ticket can be closed.