Replies: 15 comments
-
May I handle this issue? |
Beta Was this translation helpful? Give feedback.
-
Feel free. |
Beta Was this translation helpful? Give feedback.
-
Hi @slycyberguy , are you still handling this issue? I am also interested in it, in case you need any help. |
Beta Was this translation helpful? Give feedback.
-
Hi. Yeah, I'm working on it. I should be done in a few more days. I'll let you know if anything changes. Thanks for offering. |
Beta Was this translation helpful? Give feedback.
-
Hi @mateuslatrova. It looks like I'm getting overwhelmed at work with a lot of things that require high priority at the moment. I think I might be free to work on it next week, but I don't want this sitting around for too long. So if you still want, you can take over the issue. I'm unassigning myself. |
Beta Was this translation helpful? Give feedback.
-
Hi @slycyberguy ! Right now I am focused on another issue. If I finish it before anyone takes this one over, I will let everyone know that I will work on it. Thanks for letting me know! |
Beta Was this translation helpful? Give feedback.
-
Hi @anneadb, in your logs example you are reading logs from the log group |
Beta Was this translation helpful? Give feedback.
-
Hi @vincbeck, For example the successful ECS log looks like this:
And then the successful task log looks like this:
|
Beta Was this translation helpful? Give feedback.
-
I see. I can see that even with a successful task, logs are pushed to Cloudwatch when the task is done. In both examples (successful task and timed-out task), after the two lines below, there is always a delay. I guess this is the issue, I'll take a look.
|
Beta Was this translation helpful? Give feedback.
-
Hey @anneadb, I cannot reproduce the issue. When I run the same DAG as the one in the description, I get the logs in real time in my cloudwatch log group. The task does time out but I get the logs:
|
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
Hi @vincbeck , Could you share your logging configuration? Maybe I can find a difference there. Ours looks like this:
|
Beta Was this translation helpful? Give feedback.
-
I am using Breeze so I guess I have the default configuration. But it should not even matter because I am looking at logs inside the log group I specified through the |
Beta Was this translation helpful? Give feedback.
-
Since we're currently on MWAA 2.7.2, according to https://docs.aws.amazon.com/mwaa/latest/userguide/connections-packages.html we are using I found some hints that the issue may have to do with the "remote_logging" configuration that's why I was curious to know what your config looks like. |
Beta Was this translation helpful? Give feedback.
-
I'm converting this to discussion because for the moment this is not reproducible and more troubleshooting |
Beta Was this translation helpful? Give feedback.
-
Apache Airflow Provider(s)
amazon
Versions of Apache Airflow Providers
apache-airflow-providers-amazon==8.7.1
Apache Airflow version
2.7.2
Operating System
Amazon Linux AMI
Deployment
Amazon (AWS) MWAA
Deployment details
We are deploying MWAA using CDK.
Dags are located in S3.
What happened
We are running several Airflow tasks that start ECS containers and wait for their completion before starting the following tasks. I noticed that logs only appear upon the completion of the task.
However, if a task runs into a TimeoutError in Airflow, no task logs are created at all in Cloudwatch.
I have been in contact with AWS Enterprise Support but they are saying that the issue is located in the Operator and not MWAA.
What you think should happen instead
ECS task logs are written to CloudWatch even if the Airflow task fails.
Even better: task logs are written continuously to CloudWatch.
How to reproduce
Anything else
Logs example:
Are you willing to submit PR?
Code of Conduct
Beta Was this translation helpful? Give feedback.
All reactions