fix: verify AWS SSL connection

[declanyewlim]

Verify the AWS boto3 connection with botocore default certificates.

A recent botocore release (v1.43.54) now raises an error when the AWS CA bundle value is an empty/whitespace-only string. Error is raised on CI when conda-build is run as it sets the REQUESTS_CA_BUNDLE to an empty string if unset.

Previous tests resolved to an older boto3 (and botocore) version where no verification was done, but no issue was raised.

[ovesh]

1. why?
2. do we need to update meta.yaml in the feedstock? (same for the explicit new dependency)

[declanyewlim]

We don't technically need to pin the version here if we are setting the verify argument in the code. My idea was to pin the version to correctly surface the error of no SSL verification for the boto3 client. If we are setting the verify argument though, then the erroneous path will no longer be valid so we can remove it here.

The issue came from conda_build, where the REQUESTS_CA_BUNDLE env variable becomes an empty string if unset. This ended up with silent non verification for the connection as verify=="".

If we used the certifi certs, we would need to update the meta.yaml. botocore does come with a default set of certificates, so it would make more sense to use those instead rather than choose the certifi certificates.

[ovesh]

botocore does come with a default set of certificates, so it would make more sense to use those instead rather than choose the certifi certificates.

That makes sense, shall we try that?

[declanyewlim]

yep setting verify=True uses the default botocore certs. So certifi is still in the environment (from google-cloud-storage) and sphinx, but the SSL verification for the boto3 session will use the certs that come with the botocore package.