Amazon Glue job providers not printing log when job get completed or failed. #26196
Closed
2 tasks done
Labels
area:providers
good first issue
kind:bug
This is a clearly a bug
provider:amazon-aws
AWS/Amazon - related issues
Apache Airflow Provider(s)
amazon
Versions of Apache Airflow Providers
5.0.0
Apache Airflow version
>= 2.3.2
Operating System
Linux
Deployment
Docker-Compose
Deployment details
docker-compose
What happened
The method job_completion of GlueJobHook call print_job_logs in finally block that will be never call when job get completed either with successfully state or failed state since when job get completed successfully it return value by using return statement (that will not execute finally block) and similarly in case of failure it will raise exception that will also not execute the finally block.
Due to that airflow does not show Glue job logs from
CloudWatch
.What you think should happen instead
It should print the log in all cases, (failure or success) when
verbose=True
.How to reproduce
Use latest version of
amazon providers==5.0.0
and create a Airflow task for any Glue job.Make sure to pass
verbose = True
inGlueJobOperator
as below:Anything else
This is a code bug that will result no
**cloudwatch**
log print in every case. I will provide the solution with enhancement for continuous logging.Are you willing to submit PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: