Airflow ECS Operator: InvalidSignatureException #24946
Description
Apache Airflow version
2.3.2 (latest released)
What happened
Using the Amazon ECSOperator to trigger tasks yields a:
botocore.exceptions.ClientError: An error occurred (InvalidSignatureException) when calling the RunTask operation: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details. [2022-07-10, 09:15:53 UTC] {taskinstance.py:1400} INFO - Marking task as FAILED. dag_id=ecs_dag, task_id=test_task, execution_date=20220710T091047, start_date=20220710T091552, end_date=20220710T091553 [2022-07-10, 09:15:53 UTC] {standard_task_runner.py:97} ERROR - Failed to execute job 124 for task ELeiloes.docker_pt_eleiloes_re (An error occurred (InvalidSignatureException) when calling the RunTask operation: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.; 749131)
What you think should happen instead
The task should be able to successfully trigger the task and monitor its success.
How to reproduce
Create an Amazon Web Services connection, fill in the Access Key and Secret Key in the login and password areas, respectively adding the region's name to the extras.
Create an ECS operator referencing said connection with some custom overrides:
ope = EcsOperator(
task_id=f'docker_{task_name}',
cluster=cluster_name,
task_definition=task_definition_name,
launch_type="EXTERNAL|FARGATE",
aws_conn_id=ecs_connection_name,
overrides={
"containerOverrides": [
{
"name": f"docker_{task_name}'",
"command": ["--flag1", "value1", '--aws_region', 'us-west-2'],
},
],
},
tags={
"Project": "Test",
},
awslogs_group="/ecs/hello-world",
awslogs_region=s3_logs_region,
awslogs_stream_prefix="ecs/hello-world-container"
)
Operating System
UNIX
Versions of Apache Airflow Providers
apache-airflow-providers-amazon 4.0.0
Deployment
Docker-Compose
Deployment details
No response
Anything else
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct