Airflow 2.8.1: awscli installation hindered by version conflicts in requirements.txt #39663
Replies: 2 comments
-
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval. |
Beta Was this translation helpful? Give feedback.
-
That is upstream dependency issue and not related to the Apache Airflow itself aws cli v1 known as "I would turn your life into the hell until you might resolve dependencies". Personal recommendation: swhich to the aws cli v2: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html You also might use https://github.com/apache/airflow/blob/main/scripts/in_container/bin/install_aws.sh as a reference |
Beta Was this translation helpful? Give feedback.
-
Apache Airflow version
Other Airflow 2 version (please specify below)
If "Other Airflow 2 version" selected, which one?
2.8.1
What happened?
After upgrading to Airflow 2.8.1, DAGs are failing with a 'No such file or directory' error related to
awscli
. It seemsawscli
needs to be installed as part of the environment setup. However, when addingawscli
to 'requirements.txt', the installation fails due to version conflicts with dependent packages likebotocore
,docutils
,colorama
, andrsa
. These conflicts arise from the constraint list pinned for Airflow 2.8.1.To match the
botocore
version from the constraint list i.e1.33.13
, which is a dependent package of awscli, I installed specific version of awscli i.e1.31.13
installation logs:
Although after matching botocore version, docutils and other dependent versions were having version conflict.
What you think should happen instead?
awscli
should be installed via requirements.txt without any version conflictHow to reproduce
Steps to be followed for reproducing the error:
Operating System
docker
Versions of Apache Airflow Providers
Deployment
Amazon (AWS) MWAA
Deployment details
No response
Anything else?
awscli related packages:
reducing the package version of docutils, colorama and rsa did the trick for me.
Are you willing to submit PR?
Code of Conduct
Beta Was this translation helpful? Give feedback.
All reactions