awscli --no-sign-request option not working

[joverlee521]

Context

I was trying to implement the conditional use of the --no-sign-request flag for the avian flu but was still running into "Unable to locate credentials" error.

I think this is dependency issue within docker-base as I am able to use the flag successfully in conda-base.

[joverlee521]

I thought it might be a bug with the pinned awscli v1.18.195, but the awscli changelog shows this bug was fixed in v1.7.1.

The flag works within the Nextstrain shell if I re-install awscli with:

pip install awscli==1.18.195 --upgrade
...
Installing collected packages: PyYAML, botocore, s3transfer
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
yte 1.5.4 requires pyyaml<7.0,>=6.0, but you have pyyaml 5.3.1 which is incompatible.
boto3 1.34.106 requires botocore<1.35.0,>=1.34.106, but you have botocore 1.19.35 which is incompatible.
boto3 1.34.106 requires s3transfer<0.11.0,>=0.10.0, but you have s3transfer 0.3.7 which is incompatible.
aiobotocore 2.13.0 requires botocore<1.34.107,>=1.34.70, but you have botocore 1.19.35 which is incompatible.
Successfully installed PyYAML-5.3.1 botocore-1.19.35 s3transfer-0.3.7

There are dependencies updates shown, so it's due to some dependency incompatibility issue?

[joverlee521]

Ah, because the Nextstrain CLI is pip installed after the awscli, it updates the botocore dependencies.

I don't really want to touch the Nextstrain CLI botocore dependency hell, so I think the easiest is to upgrade the awscli.

[jameshadfield]

(Or we could just change avian-flu to use curl, right?)

[joverlee521]

Flag works as expected after matching botocore v1.34.106 with awscli v1.32.106:

pip install awscli==1.32.106 --upgrade 
Collecting awscli==1.32.106
  Downloading awscli-1.32.106-py3-none-any.whl (4.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.5/4.5 MB 39.4 MB/s eta 0:00:00
Requirement already satisfied: rsa<4.8,>=3.1.2 in /usr/local/lib/python3.10/site-packages (from awscli==1.32.106) (4.5)
Requirement already satisfied: s3transfer<0.11.0,>=0.10.0 in /usr/local/lib/python3.10/site-packages (from awscli==1.32.106) (0.10.1)
Requirement already satisfied: colorama<0.4.7,>=0.2.5 in /usr/local/lib/python3.10/site-packages (from awscli==1.32.106) (0.4.3)
Requirement already satisfied: docutils<0.17,>=0.10 in /usr/local/lib/python3.10/site-packages (from awscli==1.32.106) (0.15.2)
Requirement already satisfied: botocore==1.34.106 in /usr/local/lib/python3.10/site-packages (from awscli==1.32.106) (1.34.106)
Requirement already satisfied: PyYAML<6.1,>=3.10 in /usr/local/lib/python3.10/site-packages (from awscli==1.32.106) (6.0.1)
Requirement already satisfied: urllib3!=2.2.0,<3,>=1.25.4 in /usr/local/lib/python3.10/site-packages (from botocore==1.34.106->awscli==1.32.106) (1.26.18)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.10/site-packages (from botocore==1.34.106->awscli==1.32.106) (2.9.0.post0)
Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/local/lib/python3.10/site-packages (from botocore==1.34.106->awscli==1.32.106) (0.10.0)
Requirement already satisfied: pyasn1>=0.1.3 in /usr/local/lib/python3.10/site-packages (from rsa<4.8,>=3.1.2->awscli==1.32.106) (0.6.0)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/site-packages (from python-dateutil<3.0.0,>=2.1->botocore==1.34.106->awscli==1.32.106) (1.16.0)
Installing collected packages: awscli
Successfully installed awscli-1.32.106

---

I don't think it's sustainable to have to maintain the compatibility of awscli with the dependencies of the Nextstrain CLI.
Any reason we can't use pipx to install packages in docker-base?

[tsibley]

We could also install the standalone Nextstrain CLI instead of the PyPI package.

[tsibley]

Or could also install AWS CLI v2, which is not installed via Pip. Or yes, could use Pipx, but that'll require some additional configuration.

[joverlee521]

Or could also install AWS CLI v2, which is not installed via Pip.

Ah, yeah. Conda-base includes AWS CLI v2 so we could match it here.

[joverlee521]

Hrm, seeing an error for my attempt to switch to AWS CLI v2 in ffdac83.

#138 ERROR: failed to calculate checksum of ref u75h8s8905a7n3r90cq32i1t3::8cv66vaxj2qiznjawq63ih86d: "/usr/local/bin/aws": not found

I must be doing something wrong with

docker-base/Dockerfile

Lines 224 to 227 in ffdac83

	RUN XX_MARCH=$(xx-info march) \
	&& curl "https://awscli.amazonaws.com/awscli-exe-linux-${XX_MARCH}.zip" -o "awscliv2.zip" \
	&& unzip awscliv2.zip \
	&& ./aws/install --install-dir /final/bin/aws-cli --bin-dir /final/bin

Will investigate tomorrow.

[tsibley]

Or yes, could use Pipx, but that'll require some additional configuration.

The least change option for now might be installing awscli v1 from PyPI with pip (no change) but doing so into an isolated venv. This requires ~no additional configuration like pipx and doesn't have the extra baggage/changes of pipx.