Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Access denied when using gdalinfo with Docker Image on EC2 Instance using AWS federation token #9444

Closed
remtav opened this issue Mar 11, 2024 · 14 comments
Closed

Access denied when using gdalinfo with Docker Image on EC2 Instance using AWS federation token #9444

remtav opened this issue Mar 11, 2024 · 14 comments

Comments

@remtav
Copy link

remtav commented Mar 11, 2024 •
edited
Loading

What is the bug?

I am encountering an "ERROR 14: access denied" using gdalinfo to access an image within a Docker container running on an EC2 instance. The setup involves GDAL and a federation token for authentication.

Some observations:

  • master key works in the same context, the issue is only with federation tokens
  • the federation token functions properly when gdalinfo is run elsewhere, such as on Windows 10 and HPC system with the same Docker container (converted to Apptainer since our HPC doesn't support Docker directly).
  • Configuration Options: configuration options modifying various env vars have been attempted without success.
  • IAM Role Modification: The IAM role was temporarily modified to grant access from the EC2 instance to the bucket, but the issue persists.
  • AWS CLI Functionality: The AWS CLI works correctly with the federation token, suggesting that the problem likely lies within GDAL.

Steps to reproduce the issue

  1. Launch docker container on EC2 instance

  2. set env vars from federation token values:
    AWS_ACCESS_KEY_ID=...
    AWS_SECRET_ACCESS_KEY=...
    AWS_SESSION_TOKEN=...

  3. run gdalinfo on an image stored in aws bucket

Versions and provenance

OS: Debian 11 (edited)
GDAL Versions Tested: 3.8 and 3.0
GDAL built from docker image miniconda (https://hub.docker.com/r/continuumio/miniconda3) in which rio-cogeo was installed (via conda) (https://anaconda.org/conda-forge/rio-cogeo)

Additional context

No response
@rouault
Copy link
Member

rouault commented Mar 11, 2024

@remtav Try setting CPL_DEBUG=ON CPL_CURL_VERBOSE=ON in situations where this work and in situations where it doesn't, and compare the output

@remtav
Copy link
Author

remtav commented Mar 12, 2024 •
edited
Loading

@remtav Try setting CPL_DEBUG=ON CPL_CURL_VERBOSE=ON in situations where this work and in situations where it doesn't, and compare the output

(works) From apptainer container in our HPC

HTTP: libcurl/8.5.0 OpenSSL/3.2.1 zlib/1.2.13 zstd/1.5.5 libssh2/1.11.0 nghttp2/1.58.0
CURL_INFO_TEXT: Couldn't find host geostore-[...].s3.amazonaws.com in the .netrc file; using defaults
CURL_INFO_TEXT: Host geostore-[...].s3.amazonaws.com:443 was resolved.
CURL_INFO_TEXT: IPv6: (none)
CURL_INFO_TEXT: IPv4: 16.12.4.20, 52.95.146.228, 52.95.147.123, 16.12.4.36, 52.95.145.179, 16.12.4.68, 52.95.146.163, 52.95.190.52
CURL_INFO_TEXT:   Trying 16.12.4.20:443...
CURL_INFO_TEXT: Connected to geostore-[...].s3.amazonaws.com (16.12.4.20) port 443
CURL_INFO_TEXT: ALPN: curl offers h2,http/1.1
CURL_INFO_TEXT: TLSv1.3 (OUT), TLS handshake, Client hello (1):
CURL_INFO_TEXT:  CAfile: /etc/ssl/certs/ca-certificates.crt
CURL_INFO_TEXT:  CApath: none
CURL_INFO_TEXT: TLSv1.3 (IN), TLS handshake, Server hello (2):
CURL_INFO_TEXT: TLSv1.2 (IN), TLS handshake, Certificate (11):
CURL_INFO_TEXT: TLSv1.2 (IN), TLS handshake, Server key exchange (12):
CURL_INFO_TEXT: TLSv1.2 (IN), TLS handshake, Server finished (14):
CURL_INFO_TEXT: TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
CURL_INFO_TEXT: TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
CURL_INFO_TEXT: TLSv1.2 (OUT), TLS handshake, Finished (20):
CURL_INFO_TEXT: TLSv1.2 (IN), TLS handshake, Finished (20):
CURL_INFO_TEXT: SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256 / secp256r1 / rsaEncryption
CURL_INFO_TEXT: ALPN: server accepted http/1.1
CURL_INFO_TEXT: Server certificate:
CURL_INFO_TEXT:  subject: CN=*.s3.amazonaws.com
CURL_INFO_TEXT:  start date: Oct 10 00:00:00 2023 GMT
CURL_INFO_TEXT:  expire date: Jul  3 23:59:59 2024 GMT
CURL_INFO_TEXT:  subjectAltName: host "geostore-[...].s3.amazonaws.com" matched cert's "*.s3.amazonaws.com"
CURL_INFO_TEXT:  issuer: C=US; O=Amazon; CN=Amazon RSA 2048 M01
CURL_INFO_TEXT:  SSL certificate verify ok.
CURL_INFO_TEXT:   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
CURL_INFO_TEXT:   Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
CURL_INFO_TEXT:   Certificate level 2: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
CURL_INFO_TEXT: using HTTP/1.x
CURL_INFO_HEADER_OUT: GET /?delimiter=%2F&prefix=data%2Fpath%2Fto%2Ffolder%2FQC_VaudreuilDorion_Planet_20230507_psscene_analytic_sr_udm2%2F HTTP/1.1
Host: geostore-[...].s3.amazonaws.com
User-Agent: GDAL/3.8.4
Accept: */*
x-amz-date: 20240311T205025Z
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Security-Token: IQoJb3JpZ2luX2VjEC0aDGNhLWNlbnRyYWwtMSJHMEUCIExCY/OCd2/ZXuksmdYIx89LxlciJq6ZtWtrL7NmFi96AiEA8Bezr4um5CmUKm0y737u1doyADmlSUVdaGAMDJtt4PkqmwMINhABGgw3OTU3NjM1MzgzNjgiDJT8ohbJa3nebyh2Uir4AlLpejvZwaLMa60l/KEMA9S25Jct7l/575t7YD4xCsI80iAcjoow6FBy7b7dc8DKtBDt51WNifryLsfP4itx9t7NSB5C3pUa+lD08UpbQjCJa9P3ezTcUSa8ivnDhqRObBEi4snSK6NFayT9MlFJKgIswM+qnPdYaWmMsdBhaMubMbDts+EXb0XJU5AWZvmMDVp/zgYSR63sGz38fsWrmA1nBzGNzMX+HC7FI8FYXAIb5lq7fHfyYQ78rvb+P26o/JQYkxgrvyRJZBGZeiIFwIbUOUf1Cvbiz8+yOyjDcsMuk6t3FMVOZkuVBA2XL5AJhk8fJrCIX3xcmke3pnMGLHwUIyrdnSoCrJUButE3xqZjqZ9FP5kW7oRM9PcenP/mtSgWmOHNkMAQvOXhjFjwV/AtqODsJnWaSwR7ncEHJLyFHiD8qjQaS/Bcv2H1ru5ruSogrrpiEU3ZR2vOh/17ef/zSxUMu9H0Ptp7X8Pxb0in+6QAik6yp4gwody9rwY6mQFHQEIy3ejE0Try0sEMGCipEHsU5OWjNvd1SoGYT3W21PfErSA98dYKfn/HXBWGZ5QFaca0m1YTgpwDoWa6AKgkgw8t6SGG8xbWhf66IY65rilQ9yCPUiWf99nYwxb7GgYhisnL9drgniytAywRn70+7LW3tvcUPo5vCtFizOWu07819rkMtWRDn/pLB4H8Vb/2iolfrMSHu2M=
Authorization: AWS4-HMAC-SHA256 Credential=ASIA3SRZT7HAGTILH6UO/20240311/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-security-token,Signature=bed23ed9d617beca47d3658309ac201cda8c88881db4705e1a83d311b091fa19

CURL_INFO_HEADER_IN: HTTP/1.1 400 Bad Request
CURL_INFO_HEADER_IN: x-amz-bucket-region: ca-central-1
CURL_INFO_HEADER_IN: x-amz-request-id: MVDYKQ21AH0F81H5
CURL_INFO_HEADER_IN: x-amz-id-2: Dq8rlT3eYuVrnCelDGWx6MKBmY3Af5qiVBHywM0LocWrLFgshpKgtlP0rvVUVt1TMUmWBEZ60Rk=
CURL_INFO_HEADER_IN: Content-Type: application/xml
CURL_INFO_HEADER_IN: Transfer-Encoding: chunked
CURL_INFO_HEADER_IN: Date: Mon, 11 Mar 2024 20:50:25 GMT
CURL_INFO_HEADER_IN: Server: AmazonS3
CURL_INFO_HEADER_IN: Connection: close
CURL_INFO_HEADER_IN:
CURL_INFO_TEXT: Closing connection
CURL_INFO_TEXT: TLSv1.2 (IN), TLS alert, close notify (256):
CURL_INFO_TEXT: TLSv1.2 (OUT), TLS alert, close notify (256):
S3: Switching to region ca-central-1
CURL_INFO_TEXT: Couldn't find host geostore-[...].s3.amazonaws.com in the .netrc file; using defaults
CURL_INFO_TEXT: Hostname geostore-[...].s3.amazonaws.com was found in DNS cache
CURL_INFO_TEXT:   Trying 16.12.4.20:443...
CURL_INFO_TEXT: Connected to geostore-[...].s3.amazonaws.com (16.12.4.20) port 443
CURL_INFO_TEXT: ALPN: curl offers h2,http/1.1
CURL_INFO_TEXT: TLSv1.3 (OUT), TLS handshake, Client hello (1):
CURL_INFO_TEXT: TLSv1.3 (IN), TLS handshake, Server hello (2):
CURL_INFO_TEXT: TLSv1.2 (IN), TLS handshake, Certificate (11):
CURL_INFO_TEXT: TLSv1.2 (IN), TLS handshake, Server key exchange (12):
CURL_INFO_TEXT: TLSv1.2 (IN), TLS handshake, Server finished (14):
CURL_INFO_TEXT: TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
CURL_INFO_TEXT: TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
CURL_INFO_TEXT: TLSv1.2 (OUT), TLS handshake, Finished (20):
CURL_INFO_TEXT: TLSv1.2 (IN), TLS handshake, Finished (20):
CURL_INFO_TEXT: SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256 / secp256r1 / rsaEncryption
CURL_INFO_TEXT: ALPN: server accepted http/1.1
CURL_INFO_TEXT: Server certificate:
CURL_INFO_TEXT:  subject: CN=*.s3.amazonaws.com
CURL_INFO_TEXT:  start date: Oct 10 00:00:00 2023 GMT
CURL_INFO_TEXT:  expire date: Jul  3 23:59:59 2024 GMT
CURL_INFO_TEXT:  subjectAltName: host "geostore-[...].s3.amazonaws.com" matched cert's "*.s3.amazonaws.com"
CURL_INFO_TEXT:  issuer: C=US; O=Amazon; CN=Amazon RSA 2048 M01
CURL_INFO_TEXT:  SSL certificate verify ok.
CURL_INFO_TEXT:   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
CURL_INFO_TEXT:   Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
CURL_INFO_TEXT:   Certificate level 2: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
CURL_INFO_TEXT: using HTTP/1.x
CURL_INFO_HEADER_OUT: GET /?delimiter=%2F&prefix=data%2Fpath%2Fto%2Ffolder%2FQC_VaudreuilDorion_Planet_20230507_psscene_analytic_sr_udm2%2F HTTP/1.1
Host: geostore-[...].s3.amazonaws.com
User-Agent: GDAL/3.8.4
Accept: */*
x-amz-date: 20240311T205025Z
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Security-Token: IQoJb3JpZ2luX2VjEC0aDGNhLWNlbnRyYWwtMSJHMEUCIExCY/OCd2/ZXuksmdYIx89LxlciJq6ZtWtrL7NmFi96AiEA8Bezr4um5CmUKm0y737u1doyADmlSUVdaGAMDJtt4PkqmwMINhABGgw3OTU3NjM1MzgzNjgiDJT8ohbJa3nebyh2Uir4AlLpejvZwaLMa60l/KEMA9S25Jct7l/575t7YD4xCsI80iAcjoow6FBy7b7dc8DKtBDt51WNifryLsfP4itx9t7NSB5C3pUa+lD08UpbQjCJa9P3ezTcUSa8ivnDhqRObBEi4snSK6NFayT9MlFJKgIswM+qnPdYaWmMsdBhaMubMbDts+EXb0XJU5AWZvmMDVp/zgYSR63sGz38fsWrmA1nBzGNzMX+HC7FI8FYXAIb5lq7fHfyYQ78rvb+P26o/JQYkxgrvyRJZBGZeiIFwIbUOUf1Cvbiz8+yOyjDcsMuk6t3FMVOZkuVBA2XL5AJhk8fJrCIX3xcmke3pnMGLHwUIyrdnSoCrJUButE3xqZjqZ9FP5kW7oRM9PcenP/mtSgWmOHNkMAQvOXhjFjwV/AtqODsJnWaSwR7ncEHJLyFHiD8qjQaS/Bcv2H1ru5ruSogrrpiEU3ZR2vOh/17ef/zSxUMu9H0Ptp7X8Pxb0in+6QAik6yp4gwody9rwY6mQFHQEIy3ejE0Try0sEMGCipEHsU5OWjNvd1SoGYT3W21PfErSA98dYKfn/HXBWGZ5QFaca0m1YTgpwDoWa6AKgkgw8t6SGG8xbWhf66IY65rilQ9yCPUiWf99nYwxb7GgYhisnL9drgniytAywRn70+7LW3tvcUPo5vCtFizOWu07819rkMtWRDn/pLB4H8Vb/2iolfrMSHu2M=
Authorization: AWS4-HMAC-SHA256 Credential=ASIA3SRZT7HAGTILH6UO/20240311/ca-central-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-security-token,Signature=f31165c5cb0f534660ed1c7eea699b1e9a957687d012ca5d519f5b247cc9fd78

CURL_INFO_HEADER_IN: HTTP/1.1 200 OK
CURL_INFO_HEADER_IN: x-amz-id-2: nocFk5QyNxZ0LwWbTl/E7lBezi0cKDPD4E0k6gKR/dI6jjP4FWzdGiXXsRUVTcZnH2CS0rThINc=
CURL_INFO_HEADER_IN: x-amz-request-id: MVDJ1QR91FWRQ2JW
CURL_INFO_HEADER_IN: Date: Mon, 11 Mar 2024 20:50:26 GMT
CURL_INFO_HEADER_IN: x-amz-bucket-region: ca-central-1
CURL_INFO_HEADER_IN: Content-Type: application/xml
CURL_INFO_HEADER_IN: Transfer-Encoding: chunked
CURL_INFO_HEADER_IN: Server: AmazonS3
CURL_INFO_HEADER_IN:
CURL_INFO_TEXT: Connection #1 to host geostore-[...].s3.amazonaws.com left intact
S3: Downloading 0-16383 (https://geostore-[...].s3.amazonaws.com/path/to/folder/QC_VaudreuilDorion_Planet_20230507_psscene_analytic_sr_udm2/QC_VaudreuilDorion_PS_20230507-B.tif)...
CURL_INFO_TEXT: Couldn't find host geostore-[...].s3.amazonaws.com in the .netrc file; using defaults
CURL_INFO_TEXT: Found bundle for host: 0x55d8abc069a0 [serially]
CURL_INFO_TEXT: Can not multiplex, even if we wanted to
CURL_INFO_TEXT: Re-using existing connection with host geostore-[...].s3.amazonaws.com
CURL_INFO_HEADER_OUT: GET /path/to/folder/QC_VaudreuilDorion_Planet_20230507_psscene_analytic_sr_udm2/QC_VaudreuilDorion_PS_20230507-B.tif HTTP/1.1
Host: geostore-[...].s3.amazonaws.com
User-Agent: GDAL/3.8.4
Accept: */*
Range: bytes=0-16383
x-amz-date: 20240311T205025Z
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Security-Token: IQoJb3JpZ2luX2VjEC0aDGNhLWNlbnRyYWwtMSJHMEUCIExCY/OCd2/ZXuksmdYIx89LxlciJq6ZtWtrL7NmFi96AiEA8Bezr4um5CmUKm0y737u1doyADmlSUVdaGAMDJtt4PkqmwMINhABGgw3OTU3NjM1MzgzNjgiDJT8ohbJa3nebyh2Uir4AlLpejvZwaLMa60l/KEMA9S25Jct7l/575t7YD4xCsI80iAcjoow6FBy7b7dc8DKtBDt51WNifryLsfP4itx9t7NSB5C3pUa+lD08UpbQjCJa9P3ezTcUSa8ivnDhqRObBEi4snSK6NFayT9MlFJKgIswM+qnPdYaWmMsdBhaMubMbDts+EXb0XJU5AWZvmMDVp/zgYSR63sGz38fsWrmA1nBzGNzMX+HC7FI8FYXAIb5lq7fHfyYQ78rvb+P26o/JQYkxgrvyRJZBGZeiIFwIbUOUf1Cvbiz8+yOyjDcsMuk6t3FMVOZkuVBA2XL5AJhk8fJrCIX3xcmke3pnMGLHwUIyrdnSoCrJUButE3xqZjqZ9FP5kW7oRM9PcenP/mtSgWmOHNkMAQvOXhjFjwV/AtqODsJnWaSwR7ncEHJLyFHiD8qjQaS/Bcv2H1ru5ruSogrrpiEU3ZR2vOh/17ef/zSxUMu9H0Ptp7X8Pxb0in+6QAik6yp4gwody9rwY6mQFHQEIy3ejE0Try0sEMGCipEHsU5OWjNvd1SoGYT3W21PfErSA98dYKfn/HXBWGZ5QFaca0m1YTgpwDoWa6AKgkgw8t6SGG8xbWhf66IY65rilQ9yCPUiWf99nYwxb7GgYhisnL9drgniytAywRn70+7LW3tvcUPo5vCtFizOWu07819rkMtWRDn/pLB4H8Vb/2iolfrMSHu2M=
Authorization: AWS4-HMAC-SHA256 Credential=ASIA3SRZT7HAGTILH6UO/20240311/ca-central-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-security-token,Signature=215490c01b45ae01fb9fdba16175466e1e72319a6a91e1283c8e4df34ce75e0b

CURL_INFO_HEADER_IN: HTTP/1.1 206 Partial Content
CURL_INFO_HEADER_IN: x-amz-id-2: rRxy5NAPwiuBxhPI4qNjqrEP/BaKPKLXz/m2c8XTw74xhasGZzFQr/zKG2Tbk5zjUgC+3ErpcCc=
CURL_INFO_HEADER_IN: x-amz-request-id: MVDJT93X78TKRGXZ
CURL_INFO_HEADER_IN: Date: Mon, 11 Mar 2024 20:50:26 GMT
CURL_INFO_HEADER_IN: Last-Modified: Wed, 28 Feb 2024 20:20:31 GMT
CURL_INFO_HEADER_IN: ETag: "fe2ec9b7ea2ba006b93df5f5797b3b21-68"
CURL_INFO_HEADER_IN: x-amz-server-side-encryption: AES256
CURL_INFO_HEADER_IN: x-amz-version-id: adC24dVp2o_rhrLV_2SrbHM6iWDPZiMx
CURL_INFO_HEADER_IN: Accept-Ranges: bytes
CURL_INFO_HEADER_IN: Content-Range: bytes 0-16383/562331138
CURL_INFO_HEADER_IN: Content-Type: binary/octet-stream
CURL_INFO_HEADER_IN: Server: AmazonS3
CURL_INFO_HEADER_IN: Content-Length: 16384
CURL_INFO_HEADER_IN:
CURL_INFO_TEXT: Connection #1 to host geostore-[...].s3.amazonaws.com left intact
S3: Got response_code=206
GDAL: GDALOpen(/vsis3/geostore-[...]/path/to/folder/QC_VaudreuilDorion_Planet_20230507_psscene_analytic_sr_udm2/QC_VaudreuilDorion_PS_20230507-B.tif, this=0x55d8abcf6e40) succeeds as GTiff.
Driver: GTiff/GeoTIFF
GDAL: GDALDefaultOverviews::OverviewScan()
PROJ: proj_as_wkt: Open of /opt/conda/envs/rio_env/share/proj failed
PROJ: proj_as_wkt: Open of /opt/conda/envs/rio_env/share/proj failed
PROJ: proj_as_wkt: Open of /opt/conda/envs/rio_env/share/proj failed
PROJ: pj_obj_create: Open of /opt/conda/envs/rio_env/share/proj failed
PROJ: createGeodeticReferenceFrame: Open of /opt/conda/envs/rio_env/share/proj failed
Warning 1: PROJ: proj_create_from_database: Open of /opt/conda/envs/rio_env/share/proj failed
Warning 1: The definition of projected CRS EPSG:32618 got from GeoTIFF keys is not the same as the one from the EPSG registry, which may cause issues during reprojection operations. Set GTIFF_SRS_SOURCE configuration option to EPSG to use official parameters (overriding the ones from GeoTIFF keys), or to GEOKEYS to use custom values from GeoTIFF keys and drop the EPSG code.
Files: /vsis3/geostore-[...]/path/to/folder/QC_VaudreuilDorion_Planet_20230507_psscene_analytic_sr_udm2/QC_VaudreuilDorion_PS_20230507-B.tif
Size is 24216, 16456
PROJ: proj_as_wkt: Open of /opt/conda/envs/rio_env/share/proj failed
Coordinate System is:
PROJCRS["WGS 84 / UTM zone 18N",
    BASEGEOGCRS["WGS 84",
        DATUM["World Geodetic System 1984",
            ELLIPSOID["WGS 84",6378137,298.257223563,
                LENGTHUNIT["metre",1,
                    ID["EPSG",9001]]]],
        PRIMEM["Greenwich",0,
            ANGLEUNIT["degree",0.0174532925199433,
                ID["EPSG",9122]]]],
    CONVERSION["Transverse Mercator",
        METHOD["Transverse Mercator",
            ID["EPSG",9807]],
        PARAMETER["Latitude of natural origin",0,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8801]],
        PARAMETER["Longitude of natural origin",-75,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8802]],
        PARAMETER["Scale factor at natural origin",0.9996,
            SCALEUNIT["unity",1],
            ID["EPSG",8805]],
        PARAMETER["False easting",500000,
            LENGTHUNIT["metre",1],
            ID["EPSG",8806]],
        PARAMETER["False northing",0,
            LENGTHUNIT["metre",1],
            ID["EPSG",8807]]],
    CS[Cartesian,2],
        AXIS["(E)",east,
            ORDER[1],
            LENGTHUNIT["metre",1,
                ID["EPSG",9001]]],
        AXIS["(N)",north,
            ORDER[2],
            LENGTHUNIT["metre",1,
                ID["EPSG",9001]]]]
Data axis to CRS axis mapping: 1,2
Origin = (528336.000000000000000,5060133.000000000000000)
Pixel Size = (3.000000000000000,-3.000000000000000)
Metadata:
  AREA_OR_POINT=Area
  TIFFTAG_DATETIME=2023:05:07 00:00:00
Image Structure Metadata:
  INTERLEAVE=BAND
  LAYOUT=COG
GTiff: ScanDirectories()
GTiff: Opened band mask.
GTiff: Opened 12108x8228 overview.
GTiff: Opened 6054x4114 overview.
GTiff: Opened 3027x2057 overview.
GTiff: Opened 1513x1028 overview.
GTiff: Opened 756x514 overview.
GTiff: Opened 378x257 overview.
GTiff: Opened band mask for 12108x8228 overview.
GTiff: Opened band mask for 6054x4114 overview.
GTiff: Opened band mask for 3027x2057 overview.
GTiff: Opened band mask for 1513x1028 overview.
GTiff: Opened band mask for 756x514 overview.
GTiff: Opened band mask for 378x257 overview.
PROJ: proj_as_wkt: Open of /opt/conda/envs/rio_env/share/proj failed
PROJ: proj_as_wkt: Open of /opt/conda/envs/rio_env/share/proj failed
PROJ: proj_as_wkt: Open of /opt/conda/envs/rio_env/share/proj failed
PROJ: proj_as_wkt: Open of /opt/conda/envs/rio_env/share/proj failed
PROJ: proj_as_wkt: Open of /opt/conda/envs/rio_env/share/proj failed
PROJ: proj_as_wkt: Open of /opt/conda/envs/rio_env/share/proj failed
PROJ: proj_as_proj_string: Open of /opt/conda/envs/rio_env/share/proj failed
PROJ: proj_as_proj_string: Open of /opt/conda/envs/rio_env/share/proj failed
PROJ: proj_create: Open of /opt/conda/envs/rio_env/share/proj failed
PROJ: proj_create: Open of /opt/conda/envs/rio_env/share/proj failed
PROJ: proj_create_operation_factory_context: Open of /opt/conda/envs/rio_env/share/proj failed
PROJ: pj_obj_create: Open of /opt/conda/envs/rio_env/share/proj failed
PROJ: proj_coordoperation_is_instantiable: Open of /opt/conda/envs/rio_env/share/proj failed
Corner Coordinates:
Upper Left  (  528336.000, 5060133.000) ( 74d38' 9.80"W, 45d41'39.01"N)
Lower Left  (  528336.000, 5010765.000) ( 74d38'20.05"W, 45d14'59.31"N)
Upper Right (  600984.000, 5060133.000) ( 73d42'11.24"W, 45d41'14.60"N)
Lower Right (  600984.000, 5010765.000) ( 73d42'47.78"W, 45d14'35.27"N)
Center      (  564660.000, 5035449.000) ( 74d10'22.14"W, 45d28'10.47"N)
Band 1 Block=512x512 Type=Byte, ColorInterp=Gray
  Overviews: 12108x8228, 6054x4114, 3027x2057, 1513x1028, 756x514, 378x257
  Mask Flags: PER_DATASET
  Overviews of mask band: 12108x8228, 6054x4114, 3027x2057, 1513x1028, 756x514, 378x257
GDAL: GDALClose(/vsis3/geostore-[...]/path/to/folder/QC_VaudreuilDorion_Planet_20230507_psscene_analytic_sr_udm2/QC_VaudreuilDorion_PS_20230507-B.tif, this=0x55d8abcf6e40)
GDAL: In GDALDestroy - unloading GDAL shared library.

(doens't work) From docker container in EC2 instance

HTTP: libcurl/7.86.0 OpenSSL/3.0.7 zlib/1.2.13 libssh2/1.10.0 nghttp2/1.47.0
CURL_INFO_TEXT: Couldn't find host geostore-[...].s3.amazonaws.com in the (nil) file; using defaults
CURL_INFO_TEXT:   Trying 52.95.146.3:443...
CURL_INFO_TEXT: Connected to geostore-[...].s3.amazonaws.com (52.95.146.3) port 443 (#0)
CURL_INFO_TEXT: ALPN: offers h2
CURL_INFO_TEXT: ALPN: offers http/1.1
CURL_INFO_TEXT:  CAfile: /etc/ssl/certs/ca-certificates.crt
CURL_INFO_TEXT:  CApath: none
CURL_INFO_TEXT: TLSv1.0 (OUT), TLS header, Certificate Status (22):
CURL_INFO_TEXT: TLSv1.3 (OUT), TLS handshake, Client hello (1):
CURL_INFO_TEXT: TLSv1.2 (IN), TLS header, Certificate Status (22):
CURL_INFO_TEXT: TLSv1.3 (IN), TLS handshake, Server hello (2):
CURL_INFO_TEXT: TLSv1.2 (IN), TLS header, Certificate Status (22):
CURL_INFO_TEXT: TLSv1.2 (IN), TLS handshake, Certificate (11):
CURL_INFO_TEXT: TLSv1.2 (IN), TLS header, Certificate Status (22):
CURL_INFO_TEXT: TLSv1.2 (IN), TLS handshake, Server key exchange (12):
CURL_INFO_TEXT: TLSv1.2 (IN), TLS header, Certificate Status (22):
CURL_INFO_TEXT: TLSv1.2 (IN), TLS handshake, Server finished (14):
CURL_INFO_TEXT: TLSv1.2 (OUT), TLS header, Certificate Status (22):
CURL_INFO_TEXT: TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
CURL_INFO_TEXT: TLSv1.2 (OUT), TLS header, Finished (20):
CURL_INFO_TEXT: TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
CURL_INFO_TEXT: TLSv1.2 (OUT), TLS header, Certificate Status (22):
CURL_INFO_TEXT: TLSv1.2 (OUT), TLS handshake, Finished (20):
CURL_INFO_TEXT: TLSv1.2 (IN), TLS header, Finished (20):
CURL_INFO_TEXT: TLSv1.2 (IN), TLS header, Certificate Status (22):
CURL_INFO_TEXT: TLSv1.2 (IN), TLS handshake, Finished (20):
CURL_INFO_TEXT: SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
CURL_INFO_TEXT: ALPN: server accepted http/1.1
CURL_INFO_TEXT: Server certificate:
CURL_INFO_TEXT:  subject: CN=*.s3.amazonaws.com
CURL_INFO_TEXT:  start date: Oct 10 00:00:00 2023 GMT
CURL_INFO_TEXT:  expire date: Jul  3 23:59:59 2024 GMT
CURL_INFO_TEXT:  subjectAltName: host "geostore-[...].s3.amazonaws.com" matched cert's "*.s3.amazonaws.com"
CURL_INFO_TEXT:  issuer: C=US; O=Amazon; CN=Amazon RSA 2048 M01
CURL_INFO_TEXT:  SSL certificate verify ok.
CURL_INFO_TEXT: TLSv1.2 (OUT), TLS header, Supplemental data (23):
CURL_INFO_HEADER_OUT: GET /?delimiter=%2F&prefix=data%2Fpath%2Fto%2Ffolder%2FQC_VaudreuilDorion_Planet_20230507_psscene_analytic_sr_udm2%2F HTTP/1.1
Host: geostore-[...].s3.amazonaws.com
Accept: */*
x-amz-date: 20240312T133540Z
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Security-Token: IQoJb3JpZ2luX2VjED4aDGNhLWNlbnRyYWwtMSJHMEUCIQCI4R+2oM+KuBcgKVa2yi1ESf6Fjl5+m0NQ9IgwXAn1ogIgNpbyt5dIh6PEITDgGq6tVtbLuF/UTZLrofmUnugD9sQqnAMIRxABGgw3OTU3NjM1MzgzNjgiDF09QFapEKm/9cchRCr5Ahu1cJ7+3Lj9NjlooS+dFpaorAy5M8Qlzl0IhVJjOJmBbtuu6dUUv6PBVS9ha2zhPUhobMlJTgqDT6DZmEKtcEuYA8I2ors6uw4STeNWAAfOFICCXgu3J/PowoDbO6tc0uG39h9Q52Zjrp6C0p9rqCtQcuBLvbNEM8IuvyWjJJ92kBnLBd6b0Cxl2YPQuqRzfSAg+hbVtyIkd997+qyfAAGjqsOfl6Xqbzp1w+0OOWmvMBkcFdAazUW2m63KmJr6onMKcje6mWK/bvOm/eD2NWjy/Mj4sHj/Y2tEdUcWRtzsEDBfHJcLjE6aHpdDSoBAfqILZopRqfxmizKqF2UOUdYCvfOGTAGIjN93wWbU47meUD5D4QakGo9N1WzeO6XFHUFurIGB3fcm/BsYNy2gA+lLSbavhq4W7+dNQs0VFCVLHYbDwZAeITbZFLi2nJPPtc05BZkS87D2fmv+tnV/SpVPG3TVnCRtgD8p8iZkPhTXxZN8CMO1+5NuMPGzwa8GOpkBoMysirxj4dgLMOzFM2MJQ0EmmZ73QV8iXlteTf7epYlRT73/bwn4LIBqVIUI6sN+HnB3NEKj5OOD2yT8fSJURHqTIfLgLgnvbnP7FL6sQ2nRSZ7fD3xSvq9Cjcb0IRA/D2cYO774ICv8mV7AlQg3UkNzPzYaFeiHTyRaHjXkjcZCl5XY0A4Np01BCW6T3wZGFBkBvaoW9YTR
Authorization: AWS4-HMAC-SHA256 Credential=ASIA3SRZT7HAEL3X3Z5A/20240312/ca-central-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-security-token,Signature=e55183bf616d2df4b82f328f81e428b1f351954780d8ff00d147aa0a4ea52ecc

CURL_INFO_TEXT: TLSv1.2 (IN), TLS header, Supplemental data (23):
CURL_INFO_TEXT: Mark bundle as not supporting multiuse
CURL_INFO_HEADER_IN: HTTP/1.1 403 Forbidden
CURL_INFO_HEADER_IN: x-amz-bucket-region: ca-central-1
CURL_INFO_HEADER_IN: x-amz-request-id: GXEAFXHX51T16Q3F
CURL_INFO_HEADER_IN: x-amz-id-2: +pOaGkJKpBIH2fJbJX7dJNzeZuHFJq9e/bY2m3gBapaJZGknq3DKKiFj5yRpZQabxviROchsb+I=
CURL_INFO_HEADER_IN: Content-Type: application/xml
CURL_INFO_HEADER_IN: Transfer-Encoding: chunked
CURL_INFO_HEADER_IN: Date: Tue, 12 Mar 2024 13:35:39 GMT
CURL_INFO_HEADER_IN: Server: AmazonS3
CURL_INFO_HEADER_IN: 
CURL_INFO_TEXT: TLSv1.2 (IN), TLS header, Supplemental data (23):
CURL_INFO_TEXT: Connection #0 to host geostore-[...].s3.amazonaws.com left intact
S3: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>GXEAFXHX51T16Q3F</RequestId><HostId>+pOaGkJKpBIH2fJbJX7dJNzeZuHFJq9e/bY2m3gBapaJZGknq3DKKiFj5yRpZQabxviROchsb+I=</HostId></Error>
CURL_INFO_TEXT: Couldn't find host geostore-[...].s3.amazonaws.com in the (nil) file; using defaults
CURL_INFO_TEXT: Found bundle for host: 0x56252cc6a340 [serially]
CURL_INFO_TEXT: Can not multiplex, even if we wanted to
CURL_INFO_TEXT: Re-using existing connection #0 with host geostore-[...].s3.amazonaws.com
CURL_INFO_TEXT: Connected to geostore-[...].s3.amazonaws.com (52.95.146.3) port 443 (#0)
CURL_INFO_TEXT: TLSv1.2 (OUT), TLS header, Supplemental data (23):
CURL_INFO_HEADER_OUT: GET /path/to/folder/QC_VaudreuilDorion_Planet_20230507_psscene_analytic_sr_udm2/QC_VaudreuilDorion_PS_20230507-B.tif HTTP/1.1
Host: geostore-[...].s3.amazonaws.com
Accept: */*
Range: bytes=0-16383
x-amz-date: 20240312T133540Z
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Security-Token: IQoJb3JpZ2luX2VjED4aDGNhLWNlbnRyYWwtMSJHMEUCIQCI4R+2oM+KuBcgKVa2yi1ESf6Fjl5+m0NQ9IgwXAn1ogIgNpbyt5dIh6PEITDgGq6tVtbLuF/UTZLrofmUnugD9sQqnAMIRxABGgw3OTU3NjM1MzgzNjgiDF09QFapEKm/9cchRCr5Ahu1cJ7+3Lj9NjlooS+dFpaorAy5M8Qlzl0IhVJjOJmBbtuu6dUUv6PBVS9ha2zhPUhobMlJTgqDT6DZmEKtcEuYA8I2ors6uw4STeNWAAfOFICCXgu3J/PowoDbO6tc0uG39h9Q52Zjrp6C0p9rqCtQcuBLvbNEM8IuvyWjJJ92kBnLBd6b0Cxl2YPQuqRzfSAg+hbVtyIkd997+qyfAAGjqsOfl6Xqbzp1w+0OOWmvMBkcFdAazUW2m63KmJr6onMKcje6mWK/bvOm/eD2NWjy/Mj4sHj/Y2tEdUcWRtzsEDBfHJcLjE6aHpdDSoBAfqILZopRqfxmizKqF2UOUdYCvfOGTAGIjN93wWbU47meUD5D4QakGo9N1WzeO6XFHUFurIGB3fcm/BsYNy2gA+lLSbavhq4W7+dNQs0VFCVLHYbDwZAeITbZFLi2nJPPtc05BZkS87D2fmv+tnV/SpVPG3TVnCRtgD8p8iZkPhTXxZN8CMO1+5NuMPGzwa8GOpkBoMysirxj4dgLMOzFM2MJQ0EmmZ73QV8iXlteTf7epYlRT73/bwn4LIBqVIUI6sN+HnB3NEKj5OOD2yT8fSJURHqTIfLgLgnvbnP7FL6sQ2nRSZ7fD3xSvq9Cjcb0IRA/D2cYO774ICv8mV7AlQg3UkNzPzYaFeiHTyRaHjXkjcZCl5XY0A4Np01BCW6T3wZGFBkBvaoW9YTR
Authorization: AWS4-HMAC-SHA256 Credential=ASIA3SRZT7HAEL3X3Z5A/20240312/ca-central-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-security-token,Signature=09e88b4a2d8585453b469910196814f85637d151cc7060065da4e682dd5c9f16

CURL_INFO_TEXT: TLSv1.2 (IN), TLS header, Supplemental data (23):
CURL_INFO_TEXT: Mark bundle as not supporting multiuse
CURL_INFO_HEADER_IN: HTTP/1.1 403 Forbidden
CURL_INFO_HEADER_IN: x-amz-request-id: GXEA4YWRYS2YN2TX
CURL_INFO_HEADER_IN: x-amz-id-2: vrAUZBdk+p2nXy4tg9XSz3XKdix24iSoqwYCGUdsg/g8mecU0RuWdGK458oYgnxfB9+VsRqUt3c=
CURL_INFO_HEADER_IN: Content-Type: application/xml
CURL_INFO_HEADER_IN: Transfer-Encoding: chunked
CURL_INFO_HEADER_IN: Date: Tue, 12 Mar 2024 13:35:39 GMT
CURL_INFO_HEADER_IN: Server: AmazonS3
CURL_INFO_HEADER_IN: 
CURL_INFO_TEXT: TLSv1.2 (IN), TLS header, Supplemental data (23):
CURL_INFO_TEXT: Connection #0 to host geostore-[...].s3.amazonaws.com left intact
S3: GetFileSize(https://geostore-[...].s3.amazonaws.com/path/to/folder/QC_VaudreuilDorion_Planet_20230507_psscene_analytic_sr_udm2/QC_VaudreuilDorion_PS_20230507-B.tif): response_code=403, server error msg=<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>GXEA4YWRYS2YN2TX</RequestId><HostId>vrAUZBdk+p2nXy4tg9XSz3XKdix24iSoqwYCGUdsg/g8mecU0RuWdGK458oYgnxfB9+VsRqUt3c=</HostId></Error>
Warning 1: HTTP response code on https://geostore-[...].s3.amazonaws.com/path/to/folder/QC_VaudreuilDorion_Planet_20230507_psscene_analytic_sr_udm2/QC_VaudreuilDorion_PS_20230507-B.tif: 403
CURL_INFO_TEXT: Couldn't find host geostore-[...].s3.amazonaws.com in the (nil) file; using defaults
CURL_INFO_TEXT: Found bundle for host: 0x56252cc6a340 [serially]
CURL_INFO_TEXT: Can not multiplex, even if we wanted to
CURL_INFO_TEXT: Re-using existing connection #0 with host geostore-[...].s3.amazonaws.com
CURL_INFO_TEXT: Connected to geostore-[...].s3.amazonaws.com (52.95.146.3) port 443 (#0)
CURL_INFO_TEXT: TLSv1.2 (OUT), TLS header, Supplemental data (23):
CURL_INFO_HEADER_OUT: GET /?delimiter=%2F&max-keys=100&prefix=data%2Fpath%2Fto%2Ffolder%2FQC_VaudreuilDorion_Planet_20230507_psscene_analytic_sr_udm2%2FQC_VaudreuilDorion_PS_20230507-B.tif%2F HTTP/1.1
Host: geostore-[...].s3.amazonaws.com
Accept: */*
x-amz-date: 20240312T133540Z
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Security-Token: IQoJb3JpZ2luX2VjED4aDGNhLWNlbnRyYWwtMSJHMEUCIQCI4R+2oM+KuBcgKVa2yi1ESf6Fjl5+m0NQ9IgwXAn1ogIgNpbyt5dIh6PEITDgGq6tVtbLuF/UTZLrofmUnugD9sQqnAMIRxABGgw3OTU3NjM1MzgzNjgiDF09QFapEKm/9cchRCr5Ahu1cJ7+3Lj9NjlooS+dFpaorAy5M8Qlzl0IhVJjOJmBbtuu6dUUv6PBVS9ha2zhPUhobMlJTgqDT6DZmEKtcEuYA8I2ors6uw4STeNWAAfOFICCXgu3J/PowoDbO6tc0uG39h9Q52Zjrp6C0p9rqCtQcuBLvbNEM8IuvyWjJJ92kBnLBd6b0Cxl2YPQuqRzfSAg+hbVtyIkd997+qyfAAGjqsOfl6Xqbzp1w+0OOWmvMBkcFdAazUW2m63KmJr6onMKcje6mWK/bvOm/eD2NWjy/Mj4sHj/Y2tEdUcWRtzsEDBfHJcLjE6aHpdDSoBAfqILZopRqfxmizKqF2UOUdYCvfOGTAGIjN93wWbU47meUD5D4QakGo9N1WzeO6XFHUFurIGB3fcm/BsYNy2gA+lLSbavhq4W7+dNQs0VFCVLHYbDwZAeITbZFLi2nJPPtc05BZkS87D2fmv+tnV/SpVPG3TVnCRtgD8p8iZkPhTXxZN8CMO1+5NuMPGzwa8GOpkBoMysirxj4dgLMOzFM2MJQ0EmmZ73QV8iXlteTf7epYlRT73/bwn4LIBqVIUI6sN+HnB3NEKj5OOD2yT8fSJURHqTIfLgLgnvbnP7FL6sQ2nRSZ7fD3xSvq9Cjcb0IRA/D2cYO774ICv8mV7AlQg3UkNzPzYaFeiHTyRaHjXkjcZCl5XY0A4Np01BCW6T3wZGFBkBvaoW9YTR
Authorization: AWS4-HMAC-SHA256 Credential=ASIA3SRZT7HAEL3X3Z5A/20240312/ca-central-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-security-token,Signature=5b29e8a414238a671f62352e0c793ab4b5b68df317ed3dd3c75d0e528715a012

CURL_INFO_TEXT: TLSv1.2 (IN), TLS header, Supplemental data (23):
CURL_INFO_TEXT: Mark bundle as not supporting multiuse
CURL_INFO_HEADER_IN: HTTP/1.1 403 Forbidden
CURL_INFO_HEADER_IN: x-amz-bucket-region: ca-central-1
CURL_INFO_HEADER_IN: x-amz-request-id: GXE49BWV92K9N6EC
CURL_INFO_HEADER_IN: x-amz-id-2: qNwLsjUjzAv49mCRZg+UTPi9mrFo2F7xirFByHWRrRlDmRQ2QrEfpiSLjFT2/xBAmhYAa+UurJE=
CURL_INFO_HEADER_IN: Content-Type: application/xml
CURL_INFO_HEADER_IN: Transfer-Encoding: chunked
CURL_INFO_HEADER_IN: Date: Tue, 12 Mar 2024 13:35:39 GMT
CURL_INFO_HEADER_IN: Server: AmazonS3
CURL_INFO_HEADER_IN: 
CURL_INFO_TEXT: TLSv1.2 (IN), TLS header, Supplemental data (23):
CURL_INFO_TEXT: TLSv1.2 (IN), TLS header, Supplemental data (23):
CURL_INFO_TEXT: Connection #0 to host geostore-[...].s3.amazonaws.com left intact
S3: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>GXE49BWV92K9N6EC</RequestId><HostId>qNwLsjUjzAv49mCRZg+UTPi9mrFo2F7xirFByHWRrRlDmRQ2QrEfpiSLjFT2/xBAmhYAa+UurJE=</HostId></Error>
ERROR 14: Access Denied
gdalinfo failed - unable to open '/vsis3/geostore-[...]/path/to/folder/QC_VaudreuilDorion_Planet_20230507_psscene_analytic_sr_udm2/QC_VaudreuilDorion_PS_20230507-B.tif'.

@remtav
Copy link
Author

remtav commented Mar 12, 2024

@rouault
for some reason, in one case, the request returns
CURL_INFO_HEADER_IN: HTTP/1.1 200 OK
and in the other:
CURL_INFO_HEADER_IN: HTTP/1.1 403 Forbidden

Any ideas?

@rouault
Copy link
Member

rouault commented Mar 12, 2024

Any ideas?

well, maybe the AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN you provide in the failing case aren't valid? How do you retrieve them?

@remtav
Copy link
Author

remtav commented Mar 12, 2024 •
edited
Loading

The same key id, secret and session token are used in both cases. I tested on one infra and the other within a few minutes with these same credentials. The tokens are delivered by a team internally who manages the AWS infrastructure. The duration chosen in this case was 36h.

@rouault
Copy link
Member

rouault commented Mar 12, 2024

Perhaps enable the debug mode of the AWS CLI utilities to log network requests to see if they do something particular on the platform where this fails with GDAL? /me gumbles against cloud providers to make things more complicated than needed

@remtav
Copy link
Author

remtav commented Mar 18, 2024 •
edited
Loading

Perhaps enable the debug mode of the AWS CLI utilities to log network requests to see if they do something particular on the platform where this fails with GDAL? /me gumbles against cloud providers to make things more complicated than needed

EDIT: comparing both outputs line by line, I see no major difference. I will try the equivalent exercise with a download operation.

Here's the output with debug mode for an "ls" operation from EC2 instance:

2024-03-18 13:49:47,659 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.15.30 Python/3.11.8 Linux/6.2.0-1017-aws source/x86_64.debian.12
2024-03-18 13:49:47,659 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['s3', '--debug', 'ls', 's3://geostore-[...]/path/to/Planetscope/']
2024-03-18 13:49:47,670 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_s3 at 0x7f4befe5eb60>
2024-03-18 13:49:47,670 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_ddb at 0x7f4befe76f20>
2024-03-18 13:49:47,670 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.configure.configure.ConfigureCommand'>>
2024-03-18 13:49:47,670 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x7f4bf000c900>
2024-03-18 13:49:47,670 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x7f4bf000da80>
2024-03-18 13:49:47,670 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function alias_opsworks_cm at 0x7f4befc75620>
2024-03-18 13:49:47,670 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_history_commands at 0x7f4befed62a0>
2024-03-18 13:49:47,671 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.devcommands.CLIDevCommand'>>
2024-03-18 13:49:47,671 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_waiters at 0x7f4befc754e0>
2024-03-18 13:49:47,671 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x7f4befd24fd0>>
2024-03-18 13:49:47,671 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /opt/bitnami/aws-cli/venv/lib/python3.11/site-packages/awscli/data/cli.json
2024-03-18 13:49:47,672 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_types at 0x7f4befd9de40>
2024-03-18 13:49:47,672 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function no_sign_request at 0x7f4befd9e160>
2024-03-18 13:49:47,672 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_verify_ssl at 0x7f4befd9e0c0>
2024-03-18 13:49:47,672 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_read_timeout at 0x7f4befd9e2a0>
2024-03-18 13:49:47,672 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_connect_timeout at 0x7f4befd9e200>
2024-03-18 13:49:47,672 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <built-in method update of dict object at 0x7f4befd1e340>
2024-03-18 13:49:47,673 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.15.30 Python/3.11.8 Linux/6.2.0-1017-aws source/x86_64.debian.12 prompt/off
2024-03-18 13:49:47,673 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['s3', '--debug', 'ls', 's3://geostore-[...]/path/to/Planetscope/']
2024-03-18 13:49:47,673 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_timestamp_parser at 0x7f4befe5f380>
2024-03-18 13:49:47,673 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function register_uri_param_handler at 0x7f4bf0990900>
2024-03-18 13:49:47,673 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_binary_formatter at 0x7f4befcdbce0>
2024-03-18 13:49:47,673 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function no_pager_handler at 0x7f4bf0a51c60>
2024-03-18 13:49:47,673 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x7f4bf09b4860>
2024-03-18 13:49:47,675 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/
2024-03-18 13:49:47,676 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function attach_history_handler at 0x7f4befeb6a20>
2024-03-18 13:49:47,676 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_json_file_cache at 0x7f4bf005ae80>
2024-03-18 13:49:47,676 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3: calling handler <function add_waiters at 0x7f4befc754e0>
2024-03-18 13:49:47,676 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x7f4befd24fd0>>
2024-03-18 13:49:47,676 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3_ls: calling handler <function add_waiters at 0x7f4befc754e0>
2024-03-18 13:49:47,676 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3_ls: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x7f4befd24fd0>>
2024-03-18 13:49:47,677 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.paths: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f4bf09101d0>
2024-03-18 13:49:47,677 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.anonymous: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f4bf09101d0>
2024-03-18 13:49:47,677 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.page-size: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f4bf09101d0>
2024-03-18 13:49:47,677 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.human-readable: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f4bf09101d0>
2024-03-18 13:49:47,677 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.ls: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7f4bf0fd6290>
2024-03-18 13:49:47,677 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.summarize: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f4bf09101d0>
2024-03-18 13:49:47,677 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.ls: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7f4bf0fd6290>
2024-03-18 13:49:47,677 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.request-payer: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f4bf09101d0>
2024-03-18 13:49:47,678 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2024-03-18 13:49:47,678 - MainThread - botocore.credentials - INFO - Found credentials in environment variables.
2024-03-18 13:49:47,679 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /opt/bitnami/aws-cli/venv/lib/python3.11/site-packages/awscli/botocore/data/endpoints.json
2024-03-18 13:49:47,691 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f4bf1a66660>
2024-03-18 13:49:47,702 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /opt/bitnami/aws-cli/venv/lib/python3.11/site-packages/awscli/botocore/data/s3/2006-03-01/service-2.json
2024-03-18 13:49:47,717 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /opt/bitnami/aws-cli/venv/lib/python3.11/site-packages/awscli/botocore/data/s3/2006-03-01/endpoint-rule-set-1.json
2024-03-18 13:49:47,720 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /opt/bitnami/aws-cli/venv/lib/python3.11/site-packages/awscli/botocore/data/partitions.json
2024-03-18 13:49:47,723 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7f4bf20c89a0>
2024-03-18 13:49:47,723 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7f4bf20c8720>
2024-03-18 13:49:47,723 - MainThread - botocore.configprovider - DEBUG - Looking for endpoint for s3 via: environment_service
2024-03-18 13:49:47,723 - MainThread - botocore.configprovider - DEBUG - Looking for endpoint for s3 via: environment_global
2024-03-18 13:49:47,723 - MainThread - botocore.configprovider - DEBUG - Looking for endpoint for s3 via: config_service
2024-03-18 13:49:47,723 - MainThread - botocore.configprovider - DEBUG - Looking for endpoint for s3 via: config_global
2024-03-18 13:49:47,723 - MainThread - botocore.configprovider - DEBUG - No configured endpoint found.
2024-03-18 13:49:47,725 - MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60)
2024-03-18 13:49:47,726 - MainThread - botocore.utils - DEBUG - Registering S3 region redirector handler
2024-03-18 13:49:47,726 - MainThread - botocore.utils - DEBUG - Registering S3Express Identity Resolver
2024-03-18 13:49:47,736 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /opt/bitnami/aws-cli/venv/lib/python3.11/site-packages/awscli/botocore/data/s3/2006-03-01/paginators-1.json
2024-03-18 13:49:47,736 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /opt/bitnami/aws-cli/venv/lib/python3.11/site-packages/awscli/botocore/data/s3/2006-03-01/paginators-1.sdk-extras.json
2024-03-18 13:49:47,737 - MainThread - botocore.hooks - DEBUG - Event before-endpoint-resolution.s3: calling handler <function customize_endpoint_resolver_builtins at 0x7f4bf1a87420>
2024-03-18 13:49:47,737 - MainThread - botocore.hooks - DEBUG - Event before-endpoint-resolution.s3: calling handler <bound method S3RegionRedirectorv2.redirect_from_cache of <botocore.utils.S3RegionRedirectorv2 object at 0x7f4beeda3f50>>
2024-03-18 13:49:47,737 - MainThread - botocore.regions - DEBUG - Calling endpoint provider with parameters: {'Bucket': 'geostore-[...]', 'Region': 'ca-central-1', 'UseFIPS': False, 'UseDualStack': False, 'ForcePathStyle': False, 'Accelerate': False, 'UseGlobalEndpoint': False, 'Prefix': 'path/to/Planetscope/', 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True}
2024-03-18 13:49:47,738 - MainThread - botocore.regions - DEBUG - Endpoint provider result: https://geostore-[...].s3.ca-central-1.amazonaws.com
2024-03-18 13:49:47,738 - MainThread - botocore.regions - DEBUG - Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "None"
2024-03-18 13:49:47,738 - MainThread - botocore.regions - DEBUG - Selected auth type "v4" as "v4" with signing context params: {'region': 'ca-central-1', 'signing_name': 's3', 'disableDoubleEncoding': True}
2024-03-18 13:49:47,738 - MainThread - botocore.hooks - DEBUG - Event provide-client-params.s3.ListObjectsV2: calling handler <function base64_decode_input_blobs at 0x7f4befcdbd80>
2024-03-18 13:49:47,738 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <function set_list_objects_encoding_type_url at 0x7f4bf1a86660>
2024-03-18 13:49:47,738 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <function validate_bucket_name at 0x7f4bf1a851c0>
2024-03-18 13:49:47,738 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <function remove_bucket_from_url_paths_from_model at 0x7f4bf1a87240>
2024-03-18 13:49:47,738 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <bound method S3RegionRedirectorv2.annotate_request_context of <botocore.utils.S3RegionRedirectorv2 object at 0x7f4beeda3f50>>
2024-03-18 13:49:47,738 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <bound method S3ExpressIdentityResolver.inject_signing_cache_key of <botocore.utils.S3ExpressIdentityResolver object at 0x7f4beeda8fd0>>
2024-03-18 13:49:47,738 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <function generate_idempotent_uuid at 0x7f4bf1a84fe0>
2024-03-18 13:49:47,739 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.ListObjectsV2: calling handler <function add_expect_header at 0x7f4bf1a85580>
2024-03-18 13:49:47,739 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.ListObjectsV2: calling handler <bound method S3ExpressIdentityResolver.apply_signing_cache_key of <botocore.utils.S3ExpressIdentityResolver object at 0x7f4beeda8fd0>>
2024-03-18 13:49:47,739 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.ListObjectsV2: calling handler <function inject_api_version_header_if_needed at 0x7f4bf1a86ac0>
2024-03-18 13:49:47,739 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=ListObjectsV2) with params: {'url_path': '?list-type=2', 'query_string': {'prefix': 'path/to/Planetscope/', 'delimiter': '/', 'encoding-type': 'url'}, 'method': 'GET', 'headers': {'User-Agent': 'aws-cli/2.15.30 Python/3.11.8 Linux/6.2.0-1017-aws source/x86_64.debian.12 prompt/off command/s3.ls'}, 'body': b'', 'auth_path': '/geostore-[...]?list-type=2', 'url': 'https://geostore-[...].s3.ca-central-1.amazonaws.com/?list-type=2&prefix=data%2Fpath%2Fto%2Fto%2F&delimiter=%2F&encoding-type=url', 'context': {'client_region': 'ca-central-1', 'client_config': <botocore.config.Config object at 0x7f4bf09104d0>, 'has_streaming_input': False, 'auth_type': 'v4', 'signing': {'region': 'ca-central-1', 'signing_name': 's3', 'disableDoubleEncoding': True}, 'endpoint_properties': {'authSchemes': [{'disableDoubleEncoding': True, 'name': 'sigv4', 'signingName': 's3', 'signingRegion': 'ca-central-1'}]}, 'encoding_type_auto_set': True, 's3_redirect': {'redirected': False, 'bucket': 'geostore-[...]', 'params': {'Bucket': 'geostore-[...]', 'Prefix': 'path/to/Planetscope/', 'Delimiter': '/', 'EncodingType': 'url'}}, 'S3Express': {'bucket_name': 'geostore-[...]'}}}
2024-03-18 13:49:47,739 - MainThread - botocore.hooks - DEBUG - Event request-created.s3.ListObjectsV2: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f4befd26c50>>
2024-03-18 13:49:47,739 - MainThread - botocore.hooks - DEBUG - Event choose-signer.s3.ListObjectsV2: calling handler <function set_operation_specific_signer at 0x7f4bf1a84ea0>
2024-03-18 13:49:47,739 - MainThread - botocore.hooks - DEBUG - Event before-sign.s3.ListObjectsV2: calling handler <function remove_arn_from_signing_path at 0x7f4bf1a87380>
2024-03-18 13:49:47,739 - MainThread - botocore.hooks - DEBUG - Event before-sign.s3.ListObjectsV2: calling handler <bound method S3ExpressIdentityResolver.resolve_s3express_identity of <botocore.utils.S3ExpressIdentityResolver object at 0x7f4beeda8fd0>>
2024-03-18 13:49:47,740 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2024-03-18 13:49:47,740 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
GET
/
delimiter=%2F&encoding-type=url&list-type=2&prefix=data%2Fpath%2Fto%2Fto%2F
host:geostore-[...].s3.ca-central-1.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20240318T134947Z
x-amz-security-token:IQoJb3JpZ2luX2VjEM7//////////wEaDGNhLWNlbnRyYWwtMSJIMEYCIQDMrh4d9yYNHFcuWPJUE3z0HonQsGaZQsRpkw66Ew942gIhAM6R1XTQhnzOaYr6eYOSWPGxezD/9rNTBU+DrM12epaAKscDCNf//////////wEQARoMNzk1NzYzNTM4MzY4IgysFVlohHJN701Li6oqmwOlPbEr6th/nCGNr+Xud497m3RJkYaov+FYrG2a/2kcdowOPNWv4ZdyhvOEjTdZDB80XaN8jme1TaKsw7TjF1QoqT1Pu3EeA70OVW3PWt7qkkhIaVUMU2BuUj8Ikx1jnY7blobGZxyNO3HuMi8fMagb57n3bAHYnETmLjV7cr+pNQkyOHcpBGskr+ZiK/KUIqyEU4ga7O7zNGRsLoEOrPD7Uj1kIiQrsDHYVqHZdT6Wfatd/BvAp+YxmgQuX4CInteLQNS1RkttVDLgiLD63FAX+MTrB8jQ4jj8BZeATl5+dwNKOJx5JrOq8IxqOpdFMzEBAdy136rxWdIk/FGWBb0OkuAD7H6aycjCJufgP3Tk4o26L1aoJbWjTw89raS26RqkK8Z/TQr6e5I10DnuuPDSGaDxim6NSJoZEPrMDgrg7ckn4Gh3ACXgKVQvnaacQ9HRSaPyQ5/qbEqreHxjTJ6m051ccfqu0nrRgR9X5ZVaVBikxLL3dqULvLrwO9lcmkdqBXrn3tBy04Qp5iknCq65nq9uXcywK0zz5q8ws4rhrwY6mAFHSWUE4iYoXhgRR13r00CyI9JzctBixwll7YVq7jvBO9x/9CJv6Fl51cVyPJ4aYG1GrEL3H/vruEpVSc/QHww2gofXtSrVUYR0TJHcs4K6P6TjKbmLwNKRZ9sjnAvYd4D3VDVsWIomS2VHQLzCxvXNpL8it8Eu6pOzLIxieHyBtNz11clcdRcdL/uaefxwkTJIsXOPAn3eQA==

host;x-amz-content-sha256;x-amz-date;x-amz-security-token
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2024-03-18 13:49:47,740 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20240318T134947Z
20240318/ca-central-1/s3/aws4_request
4ac20095c6a5324d539bec70989541ab2ab0b2b18253eed5dfce8b845e1f372f
2024-03-18 13:49:47,740 - MainThread - botocore.auth - DEBUG - Signature:
162ee958b78beee0b848d8aecad975eda08a46b288f44fc416fafc213163b85a
2024-03-18 13:49:47,740 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=GET, url=https://geostore-[...].s3.ca-central-1.amazonaws.com/?list-type=2&prefix=data%2Fpath%2Fto%2Fto%2F&delimiter=%2F&encoding-type=url, headers={'User-Agent': b'aws-cli/2.15.30 Python/3.11.8 Linux/6.2.0-1017-aws source/x86_64.debian.12 prompt/off command/s3.ls', 'X-Amz-Date': b'20240318T134947Z', 'X-Amz-Security-Token': b'IQoJb3JpZ2luX2VjEM7//////////wEaDGNhLWNlbnRyYWwtMSJIMEYCIQDMrh4d9yYNHFcuWPJUE3z0HonQsGaZQsRpkw66Ew942gIhAM6R1XTQhnzOaYr6eYOSWPGxezD/9rNTBU+DrM12epaAKscDCNf//////////wEQARoMNzk1NzYzNTM4MzY4IgysFVlohHJN701Li6oqmwOlPbEr6th/nCGNr+Xud497m3RJkYaov+FYrG2a/2kcdowOPNWv4ZdyhvOEjTdZDB80XaN8jme1TaKsw7TjF1QoqT1Pu3EeA70OVW3PWt7qkkhIaVUMU2BuUj8Ikx1jnY7blobGZxyNO3HuMi8fMagb57n3bAHYnETmLjV7cr+pNQkyOHcpBGskr+ZiK/KUIqyEU4ga7O7zNGRsLoEOrPD7Uj1kIiQrsDHYVqHZdT6Wfatd/BvAp+YxmgQuX4CInteLQNS1RkttVDLgiLD63FAX+MTrB8jQ4jj8BZeATl5+dwNKOJx5JrOq8IxqOpdFMzEBAdy136rxWdIk/FGWBb0OkuAD7H6aycjCJufgP3Tk4o26L1aoJbWjTw89raS26RqkK8Z/TQr6e5I10DnuuPDSGaDxim6NSJoZEPrMDgrg7ckn4Gh3ACXgKVQvnaacQ9HRSaPyQ5/qbEqreHxjTJ6m051ccfqu0nrRgR9X5ZVaVBikxLL3dqULvLrwO9lcmkdqBXrn3tBy04Qp5iknCq65nq9uXcywK0zz5q8ws4rhrwY6mAFHSWUE4iYoXhgRR13r00CyI9JzctBixwll7YVq7jvBO9x/9CJv6Fl51cVyPJ4aYG1GrEL3H/vruEpVSc/QHww2gofXtSrVUYR0TJHcs4K6P6TjKbmLwNKRZ9sjnAvYd4D3VDVsWIomS2VHQLzCxvXNpL8it8Eu6pOzLIxieHyBtNz11clcdRcdL/uaefxwkTJIsXOPAn3eQA==', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=ASIA3SRZT7HAEDHIXK2N/20240318/ca-central-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=162ee958b78beee0b848d8aecad975eda08a46b288f44fc416fafc213163b85a'}>
2024-03-18 13:49:47,741 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): geostore-[...].s3.ca-central-1.amazonaws.com:443
2024-03-18 13:49:47,811 - MainThread - urllib3.connectionpool - DEBUG - https://geostore-[...].s3.ca-central-1.amazonaws.com:443 "GET /?list-type=2&prefix=data%2Fpath%2Fto%2Fto%2F&delimiter=%2F&encoding-type=url HTTP/1.1" 200 None
2024-03-18 13:49:47,812 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amz-id-2': 'VL/hob848q0nQhx+9wvApYbvPDsDmZcifXFLd0poB7tt8s4K4iLiZ0jJa8YkWS0ldHJvkAvjVNM=', 'x-amz-request-id': '0P6Q686W4HJ411GP', 'Date': 'Mon, 18 Mar 2024 13:49:48 GMT', 'x-amz-bucket-region': 'ca-central-1', 'Content-Type': 'application/xml', 'Transfer-Encoding': 'chunked', 'Server': 'AmazonS3'}
2024-03-18 13:49:47,812 - MainThread - botocore.parsers - DEBUG - Response body:
b'<?xml version="1.0" encoding="UTF-8"?>\n<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>geostore-[...]</Name><Prefix>path/to/Planetscope/</Prefix><KeyCount>3</KeyCount><MaxKeys>1000</MaxKeys><Delimiter>/</Delimiter><EncodingType>url</EncodingType><IsTruncated>false</IsTruncated><Contents><Key>path/to/Planetscope/</Key><LastModified>2024-02-28T20:13:35.000Z</LastModified><ETag>&quot;d41d8cd98f00b204e9800998ecf8427e&quot;</ETag><Size>0</Size><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>path/to/Planetscope/BC_Abbotsford_Planet_20210706_psscene_analytic_sr_udm2.zip</Key><LastModified>2024-02-28T19:58:19.000Z</LastModified><ETag>&quot;86e8f8448cf4f75c93235e5d679c636c-207&quot;</ETag><Size>1080699069</Size><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>path/to/Planetscope/QC_VaudreuilDorion_Planet_20230507_psscene_analytic_sr_udm2.zip</Key><LastModified>2024-02-28T20:13:35.000Z</LastModified><ETag>&quot;88fe416ba594f1e155cbc31e9189fbc9-338&quot;</ETag><Size>1771301886</Size><StorageClass>STANDARD</StorageClass></Contents></ListBucketResult>'
2024-03-18 13:49:47,813 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListObjectsV2: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x7f4beeda8bd0>>
2024-03-18 13:49:47,813 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.
2024-03-18 13:49:47,813 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListObjectsV2: calling handler <bound method S3RegionRedirectorv2.redirect_from_error of <botocore.utils.S3RegionRedirectorv2 object at 0x7f4beeda3f50>>
2024-03-18 13:49:47,813 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.ListObjectsV2: calling handler <function decode_list_object_v2 at 0x7f4bf1a867a0>
2024-03-18 13:49:47,813 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.ListObjectsV2: calling handler <function enhance_error_msg at 0x7f4befe5efc0>
2024-03-18 13:49:47,813 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.ListObjectsV2: calling handler <bound method RetryQuotaChecker.release_retry_quota of <botocore.retries.standard.RetryQuotaChecker object at 0x7f4beed3ac10>>

output from HPC (using same docker container, converted to Apptainer):

2024-03-18 14:16:19,041 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.15.30 Python/3.11.8 Linux/5.4.0-171-generic source/x86_64.debian.12
2024-03-18 14:16:19,042 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['s3', '--debug', 'ls', 's3://geostore-[...]/path/to/Planetscope/']
2024-03-18 14:16:19,069 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_s3 at 0x7fa8d5612de0>
2024-03-18 14:16:19,069 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_ddb at 0x7fa8d58331a0>
2024-03-18 14:16:19,069 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.configure.configure.ConfigureCommand'>>
2024-03-18 14:16:19,069 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x7fa8d59bcb80>
2024-03-18 14:16:19,069 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x7fa8d59bdd00>
2024-03-18 14:16:19,069 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function alias_opsworks_cm at 0x7fa8d56258a0>
2024-03-18 14:16:19,069 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_history_commands at 0x7fa8d5886520>
2024-03-18 14:16:19,069 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.devcommands.CLIDevCommand'>>
2024-03-18 14:16:19,069 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_waiters at 0x7fa8d5625760>
2024-03-18 14:16:19,069 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x7fa8d56d4b10>>
2024-03-18 14:16:19,070 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /opt/bitnami/aws-cli/venv/lib/python3.11/site-packages/awscli/data/cli.json
2024-03-18 14:16:19,072 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_types at 0x7fa8d57520c0>
2024-03-18 14:16:19,072 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function no_sign_request at 0x7fa8d57523e0>
2024-03-18 14:16:19,073 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_verify_ssl at 0x7fa8d5752340>
2024-03-18 14:16:19,073 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_read_timeout at 0x7fa8d5752520>
2024-03-18 14:16:19,073 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_connect_timeout at 0x7fa8d5752480>
2024-03-18 14:16:19,073 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <built-in method update of dict object at 0x7fa8d56ce3c0>
2024-03-18 14:16:19,073 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.15.30 Python/3.11.8 Linux/5.4.0-171-generic source/x86_64.debian.12 prompt/off
2024-03-18 14:16:19,074 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['s3', '--debug', 'ls', 's3://geostore-[...]/path/to/Planetscope/']
2024-03-18 14:16:19,074 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_timestamp_parser at 0x7fa8d5613600>
2024-03-18 14:16:19,074 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function register_uri_param_handler at 0x7fa8d638cb80>
2024-03-18 14:16:19,074 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_binary_formatter at 0x7fa8d568bf60>
2024-03-18 14:16:19,074 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function no_pager_handler at 0x7fa8d6445ee0>
2024-03-18 14:16:19,074 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x7fa8d63acae0>
2024-03-18 14:16:19,077 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/
2024-03-18 14:16:19,080 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function attach_history_handler at 0x7fa8d5866ca0>
2024-03-18 14:16:19,080 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_json_file_cache at 0x7fa8d580f100>
2024-03-18 14:16:19,080 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3: calling handler <function add_waiters at 0x7fa8d5625760>
2024-03-18 14:16:19,080 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x7fa8d56d4b10>>
2024-03-18 14:16:19,081 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3_ls: calling handler <function add_waiters at 0x7fa8d5625760>
2024-03-18 14:16:19,081 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3_ls: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x7fa8d56d4b10>>
2024-03-18 14:16:19,082 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.paths: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fa8d4d1aa90>
2024-03-18 14:16:19,082 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.anonymous: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fa8d4d1aa90>
2024-03-18 14:16:19,082 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.page-size: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fa8d4d1aa90>
2024-03-18 14:16:19,082 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.human-readable: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fa8d4d1aa90>
2024-03-18 14:16:19,082 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.ls: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7fa8d7e1f7d0>
2024-03-18 14:16:19,082 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.summarize: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fa8d4d1aa90>
2024-03-18 14:16:19,082 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.ls: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7fa8d7e1f7d0>
2024-03-18 14:16:19,083 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.request-payer: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fa8d4d1aa90>
2024-03-18 14:16:19,083 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2024-03-18 14:16:19,083 - MainThread - botocore.credentials - INFO - Found credentials in environment variables.
2024-03-18 14:16:19,084 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /opt/bitnami/aws-cli/venv/lib/python3.11/site-packages/awscli/botocore/data/endpoints.json
2024-03-18 14:16:19,114 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x7fa8d739a8e0>
2024-03-18 14:16:19,142 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /opt/bitnami/aws-cli/venv/lib/python3.11/site-packages/awscli/botocore/data/s3/2006-03-01/service-2.json
2024-03-18 14:16:19,182 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /opt/bitnami/aws-cli/venv/lib/python3.11/site-packages/awscli/botocore/data/s3/2006-03-01/endpoint-rule-set-1.json
2024-03-18 14:16:19,188 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /opt/bitnami/aws-cli/venv/lib/python3.11/site-packages/awscli/botocore/data/partitions.json
2024-03-18 14:16:19,194 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7fa8d79f8c20>
2024-03-18 14:16:19,194 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7fa8d79f89a0>
2024-03-18 14:16:19,194 - MainThread - botocore.configprovider - DEBUG - Looking for endpoint for s3 via: environment_service
2024-03-18 14:16:19,194 - MainThread - botocore.configprovider - DEBUG - Looking for endpoint for s3 via: environment_global
2024-03-18 14:16:19,194 - MainThread - botocore.configprovider - DEBUG - Looking for endpoint for s3 via: config_service
2024-03-18 14:16:19,194 - MainThread - botocore.configprovider - DEBUG - Looking for endpoint for s3 via: config_global
2024-03-18 14:16:19,194 - MainThread - botocore.configprovider - DEBUG - No configured endpoint found.
2024-03-18 14:16:19,198 - MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60)
2024-03-18 14:16:19,202 - MainThread - botocore.utils - DEBUG - Registering S3 region redirector handler
2024-03-18 14:16:19,202 - MainThread - botocore.utils - DEBUG - Registering S3Express Identity Resolver
2024-03-18 14:16:19,228 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /opt/bitnami/aws-cli/venv/lib/python3.11/site-packages/awscli/botocore/data/s3/2006-03-01/paginators-1.json
2024-03-18 14:16:19,228 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /opt/bitnami/aws-cli/venv/lib/python3.11/site-packages/awscli/botocore/data/s3/2006-03-01/paginators-1.sdk-extras.json
2024-03-18 14:16:19,229 - MainThread - botocore.hooks - DEBUG - Event before-endpoint-resolution.s3: calling handler <function customize_endpoint_resolver_builtins at 0x7fa8d73bf6a0>
2024-03-18 14:16:19,229 - MainThread - botocore.hooks - DEBUG - Event before-endpoint-resolution.s3: calling handler <bound method S3RegionRedirectorv2.redirect_from_cache of <botocore.utils.S3RegionRedirectorv2 object at 0x7fa8d45783d0>>
2024-03-18 14:16:19,229 - MainThread - botocore.regions - DEBUG - Calling endpoint provider with parameters: {'Bucket': 'geostore-[...]', 'Region': 'ca-central-1', 'UseFIPS': False, 'UseDualStack': False, 'ForcePathStyle': False, 'Accelerate': False, 'UseGlobalEndpoint': False, 'Prefix': 'path/to/Planetscope/', 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True}
2024-03-18 14:16:19,230 - MainThread - botocore.regions - DEBUG - Endpoint provider result: https://geostore-[...].s3.ca-central-1.amazonaws.com
2024-03-18 14:16:19,231 - MainThread - botocore.regions - DEBUG - Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "None"
2024-03-18 14:16:19,231 - MainThread - botocore.regions - DEBUG - Selected auth type "v4" as "v4" with signing context params: {'region': 'ca-central-1', 'signing_name': 's3', 'disableDoubleEncoding': True}
2024-03-18 14:16:19,231 - MainThread - botocore.hooks - DEBUG - Event provide-client-params.s3.ListObjectsV2: calling handler <function base64_decode_input_blobs at 0x7fa8d5698040>
2024-03-18 14:16:19,231 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <function set_list_objects_encoding_type_url at 0x7fa8d73be8e0>
2024-03-18 14:16:19,231 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <function validate_bucket_name at 0x7fa8d73bd440>
2024-03-18 14:16:19,231 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <function remove_bucket_from_url_paths_from_model at 0x7fa8d73bf4c0>
2024-03-18 14:16:19,231 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <bound method S3RegionRedirectorv2.annotate_request_context of <botocore.utils.S3RegionRedirectorv2 object at 0x7fa8d45783d0>>
2024-03-18 14:16:19,231 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <bound method S3ExpressIdentityResolver.inject_signing_cache_key of <botocore.utils.S3ExpressIdentityResolver object at 0x7fa8d4578b10>>
2024-03-18 14:16:19,231 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <function generate_idempotent_uuid at 0x7fa8d73bd260>
2024-03-18 14:16:19,232 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.ListObjectsV2: calling handler <function add_expect_header at 0x7fa8d73bd800>
2024-03-18 14:16:19,232 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.ListObjectsV2: calling handler <bound method S3ExpressIdentityResolver.apply_signing_cache_key of <botocore.utils.S3ExpressIdentityResolver object at 0x7fa8d4578b10>>
2024-03-18 14:16:19,232 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.ListObjectsV2: calling handler <function inject_api_version_header_if_needed at 0x7fa8d73bed40>
2024-03-18 14:16:19,232 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=ListObjectsV2) with params: {'url_path': '?list-type=2', 'query_string': {'prefix': 'path/to/Planetscope/', 'delimiter': '/', 'encoding-type': 'url'}, 'method': 'GET', 'headers': {'User-Agent': 'aws-cli/2.15.30 Python/3.11.8 Linux/5.4.0-171-generic source/x86_64.debian.12 prompt/off command/s3.ls'}, 'body': b'', 'auth_path': '/geostore-[...]?list-type=2', 'url': 'https://geostore-[...].s3.ca-central-1.amazonaws.com/?list-type=2&prefix=data%2Fpath%2Fto%2Ffolder%2F&delimiter=%2F&encoding-type=url', 'context': {'client_region': 'ca-central-1', 'client_config': <botocore.config.Config object at 0x7fa8d803e390>, 'has_streaming_input': False, 'auth_type': 'v4', 'signing': {'region': 'ca-central-1', 'signing_name': 's3', 'disableDoubleEncoding': True}, 'endpoint_properties': {'authSchemes': [{'disableDoubleEncoding': True, 'name': 'sigv4', 'signingName': 's3', 'signingRegion': 'ca-central-1'}]}, 'encoding_type_auto_set': True, 's3_redirect': {'redirected': False, 'bucket': 'geostore-[...]', 'params': {'Bucket': 'geostore-[...]', 'Prefix': 'path/to/Planetscope/', 'Delimiter': '/', 'EncodingType': 'url'}}, 'S3Express': {'bucket_name': 'geostore-[...]'}}}
2024-03-18 14:16:19,232 - MainThread - botocore.hooks - DEBUG - Event request-created.s3.ListObjectsV2: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fa8d4a004d0>>
2024-03-18 14:16:19,232 - MainThread - botocore.hooks - DEBUG - Event choose-signer.s3.ListObjectsV2: calling handler <function set_operation_specific_signer at 0x7fa8d73bd120>
2024-03-18 14:16:19,232 - MainThread - botocore.hooks - DEBUG - Event before-sign.s3.ListObjectsV2: calling handler <function remove_arn_from_signing_path at 0x7fa8d73bf600>
2024-03-18 14:16:19,232 - MainThread - botocore.hooks - DEBUG - Event before-sign.s3.ListObjectsV2: calling handler <bound method S3ExpressIdentityResolver.resolve_s3express_identity of <botocore.utils.S3ExpressIdentityResolver object at 0x7fa8d4578b10>>
2024-03-18 14:16:19,233 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2024-03-18 14:16:19,233 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
GET
/
delimiter=%2F&encoding-type=url&list-type=2&prefix=data%2Fpath%2Fto%2Ffolder%2F
host:geostore-[...].s3.ca-central-1.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20240318T141619Z
x-amz-security-token:IQoJb3JpZ2luX2VjEM7//////////wEaDGNhLWNlbnRyYWwtMSJIMEYCIQDMrh4d9yYNHFcuWPJUE3z0HonQsGaZQsRpkw66Ew942gIhAM6R1XTQhnzOaYr6eYOSWPGxezD/9rNTBU+DrM12epaAKscDCNf//////////wEQARoMNzk1NzYzNTM4MzY4IgysFVlohHJN701Li6oqmwOlPbEr6th/nCGNr+Xud497m3RJkYaov+FYrG2a/2kcdowOPNWv4ZdyhvOEjTdZDB80XaN8jme1TaKsw7TjF1QoqT1Pu3EeA70OVW3PWt7qkkhIaVUMU2BuUj8Ikx1jnY7blobGZxyNO3HuMi8fMagb57n3bAHYnETmLjV7cr+pNQkyOHcpBGskr+ZiK/KUIqyEU4ga7O7zNGRsLoEOrPD7Uj1kIiQrsDHYVqHZdT6Wfatd/BvAp+YxmgQuX4CInteLQNS1RkttVDLgiLD63FAX+MTrB8jQ4jj8BZeATl5+dwNKOJx5JrOq8IxqOpdFMzEBAdy136rxWdIk/FGWBb0OkuAD7H6aycjCJufgP3Tk4o26L1aoJbWjTw89raS26RqkK8Z/TQr6e5I10DnuuPDSGaDxim6NSJoZEPrMDgrg7ckn4Gh3ACXgKVQvnaacQ9HRSaPyQ5/qbEqreHxjTJ6m051ccfqu0nrRgR9X5ZVaVBikxLL3dqULvLrwO9lcmkdqBXrn3tBy04Qp5iknCq65nq9uXcywK0zz5q8ws4rhrwY6mAFHSWUE4iYoXhgRR13r00CyI9JzctBixwll7YVq7jvBO9x/9CJv6Fl51cVyPJ4aYG1GrEL3H/vruEpVSc/QHww2gofXtSrVUYR0TJHcs4K6P6TjKbmLwNKRZ9sjnAvYd4D3VDVsWIomS2VHQLzCxvXNpL8it8Eu6pOzLIxieHyBtNz11clcdRcdL/uaefxwkTJIsXOPAn3eQA==

host;x-amz-content-sha256;x-amz-date;x-amz-security-token
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2024-03-18 14:16:19,233 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20240318T141619Z
20240318/ca-central-1/s3/aws4_request
08f52be0cbc7a9461b3c8f34591db3f64580f0604f5afcef27ff5b4e1fede78f
2024-03-18 14:16:19,233 - MainThread - botocore.auth - DEBUG - Signature:
2b2f1dc5e5d5e925896d08ae6c0dae47a8b24096c7fe7a129ef7324ed9fe0717
2024-03-18 14:16:19,233 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=GET, url=https://geostore-[...].s3.ca-central-1.amazonaws.com/?list-type=2&prefix=data%2Fpath%2Fto%2Fto%2F&delimiter=%2F&encoding-type=url, headers={'User-Agent': b'aws-cli/2.15.30 Python/3.11.8 Linux/5.4.0-171-generic source/x86_64.debian.12 prompt/off command/s3.ls', 'X-Amz-Date': b'20240318T141619Z', 'X-Amz-Security-Token': b'IQoJb3JpZ2luX2VjEM7//////////wEaDGNhLWNlbnRyYWwtMSJIMEYCIQDMrh4d9yYNHFcuWPJUE3z0HonQsGaZQsRpkw66Ew942gIhAM6R1XTQhnzOaYr6eYOSWPGxezD/9rNTBU+DrM12epaAKscDCNf//////////wEQARoMNzk1NzYzNTM4MzY4IgysFVlohHJN701Li6oqmwOlPbEr6th/nCGNr+Xud497m3RJkYaov+FYrG2a/2kcdowOPNWv4ZdyhvOEjTdZDB80XaN8jme1TaKsw7TjF1QoqT1Pu3EeA70OVW3PWt7qkkhIaVUMU2BuUj8Ikx1jnY7blobGZxyNO3HuMi8fMagb57n3bAHYnETmLjV7cr+pNQkyOHcpBGskr+ZiK/KUIqyEU4ga7O7zNGRsLoEOrPD7Uj1kIiQrsDHYVqHZdT6Wfatd/BvAp+YxmgQuX4CInteLQNS1RkttVDLgiLD63FAX+MTrB8jQ4jj8BZeATl5+dwNKOJx5JrOq8IxqOpdFMzEBAdy136rxWdIk/FGWBb0OkuAD7H6aycjCJufgP3Tk4o26L1aoJbWjTw89raS26RqkK8Z/TQr6e5I10DnuuPDSGaDxim6NSJoZEPrMDgrg7ckn4Gh3ACXgKVQvnaacQ9HRSaPyQ5/qbEqreHxjTJ6m051ccfqu0nrRgR9X5ZVaVBikxLL3dqULvLrwO9lcmkdqBXrn3tBy04Qp5iknCq65nq9uXcywK0zz5q8ws4rhrwY6mAFHSWUE4iYoXhgRR13r00CyI9JzctBixwll7YVq7jvBO9x/9CJv6Fl51cVyPJ4aYG1GrEL3H/vruEpVSc/QHww2gofXtSrVUYR0TJHcs4K6P6TjKbmLwNKRZ9sjnAvYd4D3VDVsWIomS2VHQLzCxvXNpL8it8Eu6pOzLIxieHyBtNz11clcdRcdL/uaefxwkTJIsXOPAn3eQA==', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=ASIA3SRZT7HAEDHIXK2N/20240318/ca-central-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=2b2f1dc5e5d5e925896d08ae6c0dae47a8b24096c7fe7a129ef7324ed9fe0717'}>
2024-03-18 14:16:19,234 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): geostore-[...].s3.ca-central-1.amazonaws.com:443
2024-03-18 14:16:19,494 - MainThread - urllib3.connectionpool - DEBUG - https://geostore-[...].s3.ca-central-1.amazonaws.com:443 "GET /?list-type=2&prefix=data%2Fpath%2Fto%2Fto%2F&delimiter=%2F&encoding-type=url HTTP/1.1" 200 None
2024-03-18 14:16:19,494 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amz-id-2': 'Ssnm05wqhGeiKs6phxaqQJ0UU1Ga34epgckxxXjMCWrpYTwOjFdWKAtnp+n5hgPMmj+3VLAalkk=', 'x-amz-request-id': '3HS7PQKYWNC16DCD', 'Date': 'Mon, 18 Mar 2024 14:16:20 GMT', 'x-amz-bucket-region': 'ca-central-1', 'Content-Type': 'application/xml', 'Transfer-Encoding': 'chunked', 'Server': 'AmazonS3'}
2024-03-18 14:16:19,494 - MainThread - botocore.parsers - DEBUG - Response body:
b'<?xml version="1.0" encoding="UTF-8"?>\n<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>geostore-[...]</Name><Prefix>path/to/Planetscope/</Prefix><KeyCount>3</KeyCount><MaxKeys>1000</MaxKeys><Delimiter>/</Delimiter><EncodingType>url</EncodingType><IsTruncated>false</IsTruncated><Contents><Key>path/to/Planetscope/</Key><LastModified>2024-02-28T20:13:35.000Z</LastModified><ETag>&quot;d41d8cd98f00b204e9800998ecf8427e&quot;</ETag><Size>0</Size><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>path/to/Planetscope/BC_Abbotsford_Planet_20210706_psscene_analytic_sr_udm2.zip</Key><LastModified>2024-02-28T19:58:19.000Z</LastModified><ETag>&quot;86e8f8448cf4f75c93235e5d679c636c-207&quot;</ETag><Size>1080699069</Size><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>path/to/Planetscope/QC_VaudreuilDorion_Planet_20230507_psscene_analytic_sr_udm2.zip</Key><LastModified>2024-02-28T20:13:35.000Z</LastModified><ETag>&quot;88fe416ba594f1e155cbc31e9189fbc9-338&quot;</ETag><Size>1771301886</Size><StorageClass>STANDARD</StorageClass></Contents></ListBucketResult>'
2024-03-18 14:16:19,497 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListObjectsV2: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x7fa8d4578950>>
2024-03-18 14:16:19,497 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.
2024-03-18 14:16:19,497 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListObjectsV2: calling handler <bound method S3RegionRedirectorv2.redirect_from_error of <botocore.utils.S3RegionRedirectorv2 object at 0x7fa8d45783d0>>
2024-03-18 14:16:19,497 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.ListObjectsV2: calling handler <function decode_list_object_v2 at 0x7fa8d73bea20>
2024-03-18 14:16:19,497 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.ListObjectsV2: calling handler <function enhance_error_msg at 0x7fa8d5613240>
2024-03-18 14:16:19,497 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.ListObjectsV2: calling handler <bound method RetryQuotaChecker.release_retry_quota of <botocore.retries.standard.RetryQuotaChecker object at 0x7fa8d4578350>>

@remtav
Copy link
Author

remtav commented Mar 18, 2024 •
edited
Loading

Repeating with a "cp" operation, the only difference I see in comparing outputs is the following (which appears only from EC2 instance):

[INFO] [2024-03-18T14:31:05Z] [00007f2f23fcc740] [S3Client] - id=0x55b004866650: Detected Amazon EC2 with nitro as the current environment.
[INFO] [2024-03-18T14:31:05Z] [00007f2f23fcc740] [S3Client] - id=0x55b004866650: Determined instance type to be m6i.xlarge, from dmi info. Caching.

Complete output from EC2 instance (HPC's output is identical, except for two lines above):

2024-03-18 14:31:05,557 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.15.30 Python/3.11.8 Linux/6.2.0-1017-aws source/x86_64.debian.12
2024-03-18 14:31:05,557 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['s3', '--debug', 'cp', 's3://geostore-[...]/path/to/schema_tache_info_suivi_prod.json', '/tmp']
2024-03-18 14:31:05,569 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_s3 at 0x7f2f2105ab60>
2024-03-18 14:31:05,569 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_ddb at 0x7f2f2107ef20>
2024-03-18 14:31:05,569 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.configure.configure.ConfigureCommand'>>
2024-03-18 14:31:05,569 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x7f2f21204900>
2024-03-18 14:31:05,569 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x7f2f21205a80>
2024-03-18 14:31:05,569 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function alias_opsworks_cm at 0x7f2f20e71620>
2024-03-18 14:31:05,569 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_history_commands at 0x7f2f210d22a0>
2024-03-18 14:31:05,569 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.devcommands.CLIDevCommand'>>
2024-03-18 14:31:05,569 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_waiters at 0x7f2f20e714e0>
2024-03-18 14:31:05,569 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x7f2f20f20f90>>
2024-03-18 14:31:05,569 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /opt/bitnami/aws-cli/venv/lib/python3.11/site-packages/awscli/data/cli.json
2024-03-18 14:31:05,570 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_types at 0x7f2f20f99e40>
2024-03-18 14:31:05,570 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function no_sign_request at 0x7f2f20f9a160>
2024-03-18 14:31:05,571 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_verify_ssl at 0x7f2f20f9a0c0>
2024-03-18 14:31:05,571 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_read_timeout at 0x7f2f20f9a2a0>
2024-03-18 14:31:05,571 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_connect_timeout at 0x7f2f20f9a200>
2024-03-18 14:31:05,571 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <built-in method update of dict object at 0x7f2f20f1a2c0>
2024-03-18 14:31:05,571 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.15.30 Python/3.11.8 Linux/6.2.0-1017-aws source/x86_64.debian.12 prompt/off
2024-03-18 14:31:05,571 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['s3', '--debug', 'cp', 's3://geostore-[...]/path/to/schema_tache_info_suivi_prod.json', '/tmp']
2024-03-18 14:31:05,571 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_timestamp_parser at 0x7f2f2105b380>
2024-03-18 14:31:05,571 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function register_uri_param_handler at 0x7f2f213f8900>
2024-03-18 14:31:05,572 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_binary_formatter at 0x7f2f20ed7ce0>
2024-03-18 14:31:05,572 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function no_pager_handler at 0x7f2f21c49c60>
2024-03-18 14:31:05,572 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x7f2f2141c860>
2024-03-18 14:31:05,573 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/
2024-03-18 14:31:05,574 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function attach_history_handler at 0x7f2f210b2a20>
2024-03-18 14:31:05,574 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_json_file_cache at 0x7f2f21256e80>
2024-03-18 14:31:05,575 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3: calling handler <function add_waiters at 0x7f2f20e714e0>
2024-03-18 14:31:05,575 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x7f2f20f20f90>>
2024-03-18 14:31:05,575 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3_cp: calling handler <function add_waiters at 0x7f2f20e714e0>
2024-03-18 14:31:05,575 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3_cp: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x7f2f20f20f90>>
2024-03-18 14:31:05,576 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.paths: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f2f212a5e90>
2024-03-18 14:31:05,576 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.dryrun: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f2f212a5e90>
2024-03-18 14:31:05,576 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7f2f21c7a790>
2024-03-18 14:31:05,577 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.quiet: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f2f212a5e90>
2024-03-18 14:31:05,577 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7f2f21c7a790>
2024-03-18 14:31:05,577 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.anonymous: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f2f212a5e90>
2024-03-18 14:31:05,577 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.acl: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f2f212a5e90>
2024-03-18 14:31:05,577 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.follow-symlinks: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f2f212a5e90>
2024-03-18 14:31:05,577 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7f2f21c7a790>
2024-03-18 14:31:05,577 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.anonymous: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f2f212a5e90>
2024-03-18 14:31:05,577 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f2f212a5e90>
2024-03-18 14:31:05,577 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse-c: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f2f212a5e90>
2024-03-18 14:31:05,577 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse-c-key: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f2f212a5e90>
2024-03-18 14:31:05,577 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse-kms-key-id: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f2f212a5e90>
2024-03-18 14:31:05,577 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse-c-copy-source: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f2f212a5e90>
2024-03-18 14:31:05,577 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse-c-copy-source-key: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f2f212a5e90>
2024-03-18 14:31:05,577 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.storage-class: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f2f212a5e90>
2024-03-18 14:31:05,577 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.grants: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f2f212a5e90>
2024-03-18 14:31:05,577 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.website-redirect: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f2f212a5e90>
2024-03-18 14:31:05,577 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.content-type: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f2f212a5e90>
2024-03-18 14:31:05,577 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.cache-control: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f2f212a5e90>
2024-03-18 14:31:05,577 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.content-disposition: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f2f212a5e90>
2024-03-18 14:31:05,577 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.content-encoding: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f2f212a5e90>
2024-03-18 14:31:05,578 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.content-language: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f2f212a5e90>
2024-03-18 14:31:05,578 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.expires: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f2f212a5e90>
2024-03-18 14:31:05,578 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.source-region: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f2f212a5e90>
2024-03-18 14:31:05,578 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.only-show-errors: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f2f212a5e90>
2024-03-18 14:31:05,578 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7f2f21c7a790>
2024-03-18 14:31:05,578 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.anonymous: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f2f212a5e90>
2024-03-18 14:31:05,578 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.page-size: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f2f212a5e90>
2024-03-18 14:31:05,578 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.ignore-glacier-warnings: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f2f212a5e90>
2024-03-18 14:31:05,578 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7f2f21c7a790>
2024-03-18 14:31:05,578 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.force-glacier-transfer: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f2f212a5e90>
2024-03-18 14:31:05,578 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7f2f21c7a790>
2024-03-18 14:31:05,578 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.request-payer: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f2f212a5e90>
2024-03-18 14:31:05,578 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.metadata: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f2f212a5e90>
2024-03-18 14:31:05,578 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.copy-props: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f2f212a5e90>
2024-03-18 14:31:05,578 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7f2f21c7a790>
2024-03-18 14:31:05,578 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.metadata-directive: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f2f212a5e90>
2024-03-18 14:31:05,578 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.expected-size: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f2f212a5e90>
2024-03-18 14:31:05,578 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.anonymous: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f2f212a5e90>
2024-03-18 14:31:05,579 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2024-03-18 14:31:05,579 - MainThread - botocore.credentials - INFO - Found credentials in environment variables.
2024-03-18 14:31:05,580 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /opt/bitnami/aws-cli/venv/lib/python3.11/site-packages/awscli/botocore/data/endpoints.json
2024-03-18 14:31:05,591 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f2f22c1e660>
2024-03-18 14:31:05,604 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /opt/bitnami/aws-cli/venv/lib/python3.11/site-packages/awscli/botocore/data/s3/2006-03-01/service-2.json
2024-03-18 14:31:05,620 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /opt/bitnami/aws-cli/venv/lib/python3.11/site-packages/awscli/botocore/data/s3/2006-03-01/endpoint-rule-set-1.json
2024-03-18 14:31:05,623 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /opt/bitnami/aws-cli/venv/lib/python3.11/site-packages/awscli/botocore/data/partitions.json
2024-03-18 14:31:05,626 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7f2f2327c9a0>
2024-03-18 14:31:05,626 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7f2f2327c720>
2024-03-18 14:31:05,626 - MainThread - botocore.configprovider - DEBUG - Looking for endpoint for s3 via: environment_service
2024-03-18 14:31:05,627 - MainThread - botocore.configprovider - DEBUG - Looking for endpoint for s3 via: environment_global
2024-03-18 14:31:05,627 - MainThread - botocore.configprovider - DEBUG - Looking for endpoint for s3 via: config_service
2024-03-18 14:31:05,627 - MainThread - botocore.configprovider - DEBUG - Looking for endpoint for s3 via: config_global
2024-03-18 14:31:05,627 - MainThread - botocore.configprovider - DEBUG - No configured endpoint found.
2024-03-18 14:31:05,628 - MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60)
2024-03-18 14:31:05,630 - MainThread - botocore.utils - DEBUG - Registering S3 region redirector handler
2024-03-18 14:31:05,630 - MainThread - botocore.utils - DEBUG - Registering S3Express Identity Resolver
2024-03-18 14:31:05,630 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f2f22c1e660>
2024-03-18 14:31:05,631 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7f2f2327c9a0>
2024-03-18 14:31:05,631 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7f2f2327c720>
2024-03-18 14:31:05,631 - MainThread - botocore.configprovider - DEBUG - Looking for endpoint for s3 via: environment_service
2024-03-18 14:31:05,631 - MainThread - botocore.configprovider - DEBUG - Looking for endpoint for s3 via: environment_global
2024-03-18 14:31:05,631 - MainThread - botocore.configprovider - DEBUG - Looking for endpoint for s3 via: config_service
2024-03-18 14:31:05,631 - MainThread - botocore.configprovider - DEBUG - Looking for endpoint for s3 via: config_global
2024-03-18 14:31:05,631 - MainThread - botocore.configprovider - DEBUG - No configured endpoint found.
2024-03-18 14:31:05,632 - MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60)
2024-03-18 14:31:05,634 - MainThread - botocore.utils - DEBUG - Registering S3 region redirector handler
2024-03-18 14:31:05,634 - MainThread - botocore.utils - DEBUG - Registering S3Express Identity Resolver
2024-03-18 14:31:05,634 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f2f22c1e660>
2024-03-18 14:31:05,635 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7f2f2327c9a0>
2024-03-18 14:31:05,635 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7f2f2327c720>
2024-03-18 14:31:05,635 - MainThread - botocore.configprovider - DEBUG - Looking for endpoint for s3 via: environment_service
2024-03-18 14:31:05,635 - MainThread - botocore.configprovider - DEBUG - Looking for endpoint for s3 via: environment_global
2024-03-18 14:31:05,635 - MainThread - botocore.configprovider - DEBUG - Looking for endpoint for s3 via: config_service
2024-03-18 14:31:05,635 - MainThread - botocore.configprovider - DEBUG - Looking for endpoint for s3 via: config_global
2024-03-18 14:31:05,635 - MainThread - botocore.configprovider - DEBUG - No configured endpoint found.
2024-03-18 14:31:05,637 - MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60)
2024-03-18 14:31:05,638 - MainThread - botocore.utils - DEBUG - Registering S3 region redirector handler
2024-03-18 14:31:05,639 - MainThread - botocore.utils - DEBUG - Registering S3Express Identity Resolver
[INFO] [2024-03-18T14:31:05Z] [00007f2f23fcc740] [S3Client] - id=0x55b004866650: Detected Amazon EC2 with nitro as the current environment.
[INFO] [2024-03-18T14:31:05Z] [00007f2f23fcc740] [S3Client] - id=0x55b004866650: Determined instance type to be m6i.xlarge, from dmi info. Caching.
2024-03-18 14:31:05,639 - MainThread - awscli.customizations.s3.factory - DEBUG - S3 CRT client optimized for system: False
2024-03-18 14:31:05,639 - MainThread - awscli.customizations.s3.factory - DEBUG - Auto resolved s3 transfer client to: classic
2024-03-18 14:31:05,639 - MainThread - awscli.customizations.s3.factory - DEBUG - Using a multipart threshold of 8388608 and a part size of 8388608
2024-03-18 14:31:05,640 - MainThread - botocore.hooks - DEBUG - Event before-endpoint-resolution.s3: calling handler <function customize_endpoint_resolver_builtins at 0x7f2f22c3b420>
2024-03-18 14:31:05,640 - MainThread - botocore.hooks - DEBUG - Event before-endpoint-resolution.s3: calling handler <bound method S3RegionRedirectorv2.redirect_from_cache of <botocore.utils.S3RegionRedirectorv2 object at 0x7f2f1bd3ec90>>
2024-03-18 14:31:05,640 - MainThread - botocore.regions - DEBUG - Calling endpoint provider with parameters: {'Bucket': 'geostore-[...]', 'Region': 'ca-central-1', 'UseFIPS': False, 'UseDualStack': False, 'ForcePathStyle': False, 'Accelerate': False, 'UseGlobalEndpoint': False, 'Key': 'path/to/schema_tache_info_suivi_prod.json', 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True}
2024-03-18 14:31:05,641 - MainThread - botocore.regions - DEBUG - Endpoint provider result: https://geostore-[...].s3.ca-central-1.amazonaws.com
2024-03-18 14:31:05,641 - MainThread - botocore.regions - DEBUG - Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "None"
2024-03-18 14:31:05,641 - MainThread - botocore.regions - DEBUG - Selected auth type "v4" as "v4" with signing context params: {'region': 'ca-central-1', 'signing_name': 's3', 'disableDoubleEncoding': True}
2024-03-18 14:31:05,641 - MainThread - botocore.hooks - DEBUG - Event provide-client-params.s3.HeadObject: calling handler <function base64_decode_input_blobs at 0x7f2f20ed7d80>
2024-03-18 14:31:05,641 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.HeadObject: calling handler <function sse_md5 at 0x7f2f22c39260>
2024-03-18 14:31:05,641 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.HeadObject: calling handler <function validate_bucket_name at 0x7f2f22c391c0>
2024-03-18 14:31:05,641 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.HeadObject: calling handler <function remove_bucket_from_url_paths_from_model at 0x7f2f22c3b240>
2024-03-18 14:31:05,641 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.HeadObject: calling handler <bound method S3RegionRedirectorv2.annotate_request_context of <botocore.utils.S3RegionRedirectorv2 object at 0x7f2f1bd3ec90>>
2024-03-18 14:31:05,642 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.HeadObject: calling handler <bound method S3ExpressIdentityResolver.inject_signing_cache_key of <botocore.utils.S3ExpressIdentityResolver object at 0x7f2f1bd3f710>>
2024-03-18 14:31:05,642 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.HeadObject: calling handler <function generate_idempotent_uuid at 0x7f2f22c38fe0>
2024-03-18 14:31:05,642 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.HeadObject: calling handler <function add_expect_header at 0x7f2f22c39580>
2024-03-18 14:31:05,642 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.HeadObject: calling handler <bound method S3ExpressIdentityResolver.apply_signing_cache_key of <botocore.utils.S3ExpressIdentityResolver object at 0x7f2f1bd3f710>>
2024-03-18 14:31:05,642 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.HeadObject: calling handler <function inject_api_version_header_if_needed at 0x7f2f22c3aac0>
2024-03-18 14:31:05,642 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=HeadObject) with params: {'url_path': '/path/to/schema_tache_info_suivi_prod.json', 'query_string': {}, 'method': 'HEAD', 'headers': {'User-Agent': 'aws-cli/2.15.30 Python/3.11.8 Linux/6.2.0-1017-aws source/x86_64.debian.12 prompt/off command/s3.cp'}, 'body': b'', 'auth_path': '/geostore-[...]/path/to/schema_tache_info_suivi_prod.json', 'url': 'https://geostore-[...].s3.ca-central-1.amazonaws.com/path/to/schema_tache_info_suivi_prod.json', 'context': {'client_region': 'ca-central-1', 'client_config': <botocore.config.Config object at 0x7f2f1bd2ac10>, 'has_streaming_input': False, 'auth_type': 'v4', 'signing': {'region': 'ca-central-1', 'signing_name': 's3', 'disableDoubleEncoding': True}, 'endpoint_properties': {'authSchemes': [{'disableDoubleEncoding': True, 'name': 'sigv4', 'signingName': 's3', 'signingRegion': 'ca-central-1'}]}, 's3_redirect': {'redirected': False, 'bucket': 'geostore-[...]', 'params': {'Bucket': 'geostore-[...]', 'Key': 'path/to/schema_tache_info_suivi_prod.json'}}, 'S3Express': {'bucket_name': 'geostore-[...]'}}}
2024-03-18 14:31:05,643 - MainThread - botocore.hooks - DEBUG - Event request-created.s3.HeadObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f2f1bef3310>>
2024-03-18 14:31:05,643 - MainThread - botocore.hooks - DEBUG - Event choose-signer.s3.HeadObject: calling handler <function set_operation_specific_signer at 0x7f2f22c38ea0>
2024-03-18 14:31:05,643 - MainThread - botocore.hooks - DEBUG - Event before-sign.s3.HeadObject: calling handler <function remove_arn_from_signing_path at 0x7f2f22c3b380>
2024-03-18 14:31:05,643 - MainThread - botocore.hooks - DEBUG - Event before-sign.s3.HeadObject: calling handler <bound method S3ExpressIdentityResolver.resolve_s3express_identity of <botocore.utils.S3ExpressIdentityResolver object at 0x7f2f1bd3f710>>
2024-03-18 14:31:05,643 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2024-03-18 14:31:05,643 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
HEAD
/path/to/schema_tache_info_suivi_prod.json

host:geostore-[...].s3.ca-central-1.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20240318T143105Z
x-amz-security-token:IQoJb3JpZ2luX2VjEM7//////////wEaDGNhLWNlbnRyYWwtMSJIMEYCIQDQyA9pL9hNCyQweUEOroUI+bXgT2B/FEEHi8i43BA8PwIhAKJyfzrRZRPae59BAZjTJ0ReuTS8n1yUXf0HolejTVg/KroDCNj//////////wEQARoMNzk1NzYzNTM4MzY4Igw2zjmR8NhB5v3F7WYqjgOrJVdUi4rP8ViqSe0t0S5A/gNT6NMs8/AcWsh1w44JWIMkuSnI8IUyQLBjcNs2g0gas4fAPZ9wzBblQ37gRVcpjn9eOGyH9uDXzqdiB9OBc6aUrAtmh987Yh/zV8ByDIY9x8ANwlmtwN613s/dkIHsk8iVTxI6xg4DB9ksHTAyOUjiWL0+AWILKViWLOQJWRvvGcR+SwnCNz2hQFlQm7Wu+g8I+mcN8cA68EM7yIxFsgXE5J4/zolvO8G5UkE/9ewuZufnjIdWNHGz9eC+qiVyHdwbRRiNEAWejO4ZrkHBaaqexlCc1lAX5We1q446E51VbeZzwLw7+97eAjk0Dpx2McUIjJLzkrN5IRM9+0yHmNm7sQA8F/0/SAbmEDQQto8i7WiM8//yNIo21yXKuHaC1926mn2/6dN3Zffs256l7XvKDZCXFQKaV7sbUtKd2f25JOXFEEItzZBBwFRJOXH0vtJYnaSreOq8oOfGFqm86QvrFu9DU5aiuIkEH9ZEIQrF5jaTvzFAfBTgyc10ojD+nuGvBjqYASiwCZrP271FaVXipfCUQJdrS5jihFPY7javaPnIcQcRVWURCu7GjXjfsKsCDyKyhsi9dsdoyovO11rwr7oW+YAd1laEH/Kij40HnNia9q2DnJe0rMV6Kf7NoiAgDStTWqPxAQCqCjcGEbDUwEvrOJL4hlp8JtfodF6zz3TcxBK9IiV0yQ+obUlqeHVjnlCjqfvKSwbGamh2

host;x-amz-content-sha256;x-amz-date;x-amz-security-token
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2024-03-18 14:31:05,643 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20240318T143105Z
20240318/ca-central-1/s3/aws4_request
75fa2fac81aa2e504116e31a29ef1e390ce4777976116665c871e6a2c401569e
2024-03-18 14:31:05,643 - MainThread - botocore.auth - DEBUG - Signature:
167036067920bd46c0b1c3f7c0c25425f31ff405c27fb331f0c62ee731d2ee1e
2024-03-18 14:31:05,643 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=HEAD, url=https://geostore-[...].s3.ca-central-1.amazonaws.com/path/to/schema_tache_info_suivi_prod.json, headers={'User-Agent': b'aws-cli/2.15.30 Python/3.11.8 Linux/6.2.0-1017-aws source/x86_64.debian.12 prompt/off command/s3.cp', 'X-Amz-Date': b'20240318T143105Z', 'X-Amz-Security-Token': b'IQoJb3JpZ2luX2VjEM7//////////wEaDGNhLWNlbnRyYWwtMSJIMEYCIQDQyA9pL9hNCyQweUEOroUI+bXgT2B/FEEHi8i43BA8PwIhAKJyfzrRZRPae59BAZjTJ0ReuTS8n1yUXf0HolejTVg/KroDCNj//////////wEQARoMNzk1NzYzNTM4MzY4Igw2zjmR8NhB5v3F7WYqjgOrJVdUi4rP8ViqSe0t0S5A/gNT6NMs8/AcWsh1w44JWIMkuSnI8IUyQLBjcNs2g0gas4fAPZ9wzBblQ37gRVcpjn9eOGyH9uDXzqdiB9OBc6aUrAtmh987Yh/zV8ByDIY9x8ANwlmtwN613s/dkIHsk8iVTxI6xg4DB9ksHTAyOUjiWL0+AWILKViWLOQJWRvvGcR+SwnCNz2hQFlQm7Wu+g8I+mcN8cA68EM7yIxFsgXE5J4/zolvO8G5UkE/9ewuZufnjIdWNHGz9eC+qiVyHdwbRRiNEAWejO4ZrkHBaaqexlCc1lAX5We1q446E51VbeZzwLw7+97eAjk0Dpx2McUIjJLzkrN5IRM9+0yHmNm7sQA8F/0/SAbmEDQQto8i7WiM8//yNIo21yXKuHaC1926mn2/6dN3Zffs256l7XvKDZCXFQKaV7sbUtKd2f25JOXFEEItzZBBwFRJOXH0vtJYnaSreOq8oOfGFqm86QvrFu9DU5aiuIkEH9ZEIQrF5jaTvzFAfBTgyc10ojD+nuGvBjqYASiwCZrP271FaVXipfCUQJdrS5jihFPY7javaPnIcQcRVWURCu7GjXjfsKsCDyKyhsi9dsdoyovO11rwr7oW+YAd1laEH/Kij40HnNia9q2DnJe0rMV6Kf7NoiAgDStTWqPxAQCqCjcGEbDUwEvrOJL4hlp8JtfodF6zz3TcxBK9IiV0yQ+obUlqeHVjnlCjqfvKSwbGamh2', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=ASIA3SRZT7HANIDGZNFP/20240318/ca-central-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=167036067920bd46c0b1c3f7c0c25425f31ff405c27fb331f0c62ee731d2ee1e'}>
2024-03-18 14:31:05,644 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): geostore-[...].s3.ca-central-1.amazonaws.com:443
2024-03-18 14:31:05,722 - MainThread - urllib3.connectionpool - DEBUG - https://geostore-[...].s3.ca-central-1.amazonaws.com:443 "HEAD /path/to/schema_tache_info_suivi_prod.json HTTP/1.1" 200 0
2024-03-18 14:31:05,722 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amz-id-2': 'zoKa7Ybxm7sQRIzKWCNsT3AZbH0wUbhrUb68iZXL92ozVnn5xQo+NLokrHVr74aOxvN0G8kqGDQ=', 'x-amz-request-id': '6106H0A8QNM73801', 'Date': 'Mon, 18 Mar 2024 14:31:06 GMT', 'Last-Modified': 'Wed, 21 Jun 2023 19:57:05 GMT', 'ETag': '"99bc7b99184be3266f664f9d65a513e2"', 'x-amz-server-side-encryption': 'AES256', 'x-amz-meta-!gapi!created!': '2023-06-21T19:57:04 | GAPI geoai | [POST] /v1/amazon/fichiers', 'x-amz-version-id': 'ngo4GbOLwtelOFd9ogRlmnLwMiIA89DY', 'Accept-Ranges': 'bytes', 'Content-Type': 'binary/octet-stream', 'Server': 'AmazonS3', 'Content-Length': '2438'}
2024-03-18 14:31:05,722 - MainThread - botocore.parsers - DEBUG - Response body:
b''
2024-03-18 14:31:05,723 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.HeadObject: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x7f2f1bd3f310>>
2024-03-18 14:31:05,723 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.
2024-03-18 14:31:05,723 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.HeadObject: calling handler <bound method S3RegionRedirectorv2.redirect_from_error of <botocore.utils.S3RegionRedirectorv2 object at 0x7f2f1bd3ec90>>
2024-03-18 14:31:05,723 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.HeadObject: calling handler <function enhance_error_msg at 0x7f2f2105afc0>
2024-03-18 14:31:05,723 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.HeadObject: calling handler <bound method RetryQuotaChecker.release_retry_quota of <botocore.retries.standard.RetryQuotaChecker object at 0x7f2f1bd3edd0>>
2024-03-18 14:31:05,724 - MainThread - s3transfer.utils - DEBUG - Acquiring 0
2024-03-18 14:31:05,724 - ThreadPoolExecutor-1_0 - s3transfer.tasks - DEBUG - DownloadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7f2f1bdb9b50>}) about to wait for the following futures []
2024-03-18 14:31:05,724 - ThreadPoolExecutor-1_0 - s3transfer.tasks - DEBUG - DownloadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7f2f1bdb9b50>}) done waiting for dependent futures
2024-03-18 14:31:05,724 - ThreadPoolExecutor-1_0 - s3transfer.tasks - DEBUG - Executing task DownloadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7f2f1bdb9b50>}) with kwargs {'client': <botocore.client.S3 object at 0x7f2f1bd61ad0>, 'config': <s3transfer.manager.TransferConfig object at 0x7f2f21cfd850>, 'osutil': <s3transfer.utils.OSUtils object at 0x7f2f21c7a4d0>, 'request_executor': <s3transfer.futures.BoundedExecutor object at 0x7f2f1bda6b90>, 'transfer_future': <s3transfer.futures.TransferFuture object at 0x7f2f1bdb9b50>, 'io_executor': <s3transfer.futures.BoundedExecutor object at 0x7f2f1bda7090>}
2024-03-18 14:31:05,725 - ThreadPoolExecutor-1_0 - s3transfer.futures - DEBUG - Submitting task ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'geostore-[...]', 'key': 'path/to/schema_tache_info_suivi_prod.json', 'extra_args': {}}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f2f1bda6b90> for transfer request: 0.
2024-03-18 14:31:05,725 - ThreadPoolExecutor-1_0 - s3transfer.utils - DEBUG - Acquiring 0
2024-03-18 14:31:05,725 - ThreadPoolExecutor-0_0 - s3transfer.tasks - DEBUG - ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'geostore-[...]', 'key': 'path/to/schema_tache_info_suivi_prod.json', 'extra_args': {}}) about to wait for the following futures []
2024-03-18 14:31:05,725 - ThreadPoolExecutor-1_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2024-03-18 14:31:05,725 - ThreadPoolExecutor-0_0 - s3transfer.tasks - DEBUG - ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'geostore-[...]', 'key': 'path/to/schema_tache_info_suivi_prod.json', 'extra_args': {}}) done waiting for dependent futures
2024-03-18 14:31:05,725 - ThreadPoolExecutor-0_0 - s3transfer.tasks - DEBUG - Executing task ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'geostore-[...]', 'key': 'path/to/schema_tache_info_suivi_prod.json', 'extra_args': {}}) with kwargs {'client': <botocore.client.S3 object at 0x7f2f1bd61ad0>, 'bucket': 'geostore-[...]', 'key': 'path/to/schema_tache_info_suivi_prod.json', 'fileobj': <s3transfer.utils.DeferredOpenFile object at 0x7f2f1bdd0e10>, 'extra_args': {}, 'callbacks': [functools.partial(<bound method BaseSubscriber.on_progress of <awscli.customizations.s3.subscribers.ProvideSizeSubscriber object at 0x7f2f1bec5f10>>, future=<s3transfer.futures.TransferFuture object at 0x7f2f1bdb9b50>), functools.partial(<bound method BaseSubscriber.on_progress of <awscli.customizations.s3.results.QueuedResultSubscriber object at 0x7f2f1be7fe50>>, future=<s3transfer.futures.TransferFuture object at 0x7f2f1bdb9b50>), functools.partial(<bound method BaseSubscriber.on_progress of <awscli.customizations.s3.subscribers.DirectoryCreatorSubscriber object at 0x7f2f1be7ff50>>, future=<s3transfer.futures.TransferFuture object at 0x7f2f1bdb9b50>), functools.partial(<bound method BaseSubscriber.on_progress of <awscli.customizations.s3.subscribers.ProvideLastModifiedTimeSubscriber object at 0x7f2f1bdad150>>, future=<s3transfer.futures.TransferFuture object at 0x7f2f1bdb9b50>), functools.partial(<bound method ProgressResultSubscriber.on_progress of <awscli.customizations.s3.results.ProgressResultSubscriber object at 0x7f2f1be762d0>>, future=<s3transfer.futures.TransferFuture object at 0x7f2f1bdb9b50>), functools.partial(<bound method BaseSubscriber.on_progress of <awscli.customizations.s3.results.DoneResultSubscriber object at 0x7f2f1be75c50>>, future=<s3transfer.futures.TransferFuture object at 0x7f2f1bdb9b50>)], 'max_attempts': 5, 'download_output_manager': <s3transfer.download.DownloadFilenameOutputManager object at 0x7f2f1bdac150>, 'io_chunksize': 262144, 'bandwidth_limiter': None}
2024-03-18 14:31:05,726 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event before-endpoint-resolution.s3: calling handler <function customize_endpoint_resolver_builtins at 0x7f2f22c3b420>
2024-03-18 14:31:05,726 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event before-endpoint-resolution.s3: calling handler <bound method S3RegionRedirectorv2.redirect_from_cache of <botocore.utils.S3RegionRedirectorv2 object at 0x7f2f1bda5350>>
2024-03-18 14:31:05,726 - ThreadPoolExecutor-0_0 - botocore.regions - DEBUG - Calling endpoint provider with parameters: {'Bucket': 'geostore-[...]', 'Region': 'ca-central-1', 'UseFIPS': False, 'UseDualStack': False, 'ForcePathStyle': False, 'Accelerate': False, 'UseGlobalEndpoint': False, 'Key': 'path/to/schema_tache_info_suivi_prod.json', 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True}
2024-03-18 14:31:05,727 - ThreadPoolExecutor-0_0 - botocore.regions - DEBUG - Endpoint provider result: https://geostore-[...].s3.ca-central-1.amazonaws.com
2024-03-18 14:31:05,727 - ThreadPoolExecutor-0_0 - botocore.regions - DEBUG - Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "None"
2024-03-18 14:31:05,727 - ThreadPoolExecutor-0_0 - botocore.regions - DEBUG - Selected auth type "v4" as "v4" with signing context params: {'region': 'ca-central-1', 'signing_name': 's3', 'disableDoubleEncoding': True}
2024-03-18 14:31:05,727 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event provide-client-params.s3.GetObject: calling handler <function base64_decode_input_blobs at 0x7f2f20ed7d80>
2024-03-18 14:31:05,727 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event before-parameter-build.s3.GetObject: calling handler <function sse_md5 at 0x7f2f22c39260>
2024-03-18 14:31:05,727 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event before-parameter-build.s3.GetObject: calling handler <function validate_bucket_name at 0x7f2f22c391c0>
2024-03-18 14:31:05,727 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event before-parameter-build.s3.GetObject: calling handler <function remove_bucket_from_url_paths_from_model at 0x7f2f22c3b240>
2024-03-18 14:31:05,727 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event before-parameter-build.s3.GetObject: calling handler <bound method S3RegionRedirectorv2.annotate_request_context of <botocore.utils.S3RegionRedirectorv2 object at 0x7f2f1bda5350>>
2024-03-18 14:31:05,727 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event before-parameter-build.s3.GetObject: calling handler <bound method S3ExpressIdentityResolver.inject_signing_cache_key of <botocore.utils.S3ExpressIdentityResolver object at 0x7f2f1bda6510>>
2024-03-18 14:31:05,727 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event before-parameter-build.s3.GetObject: calling handler <function generate_idempotent_uuid at 0x7f2f22c38fe0>
2024-03-18 14:31:05,727 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event before-call.s3.GetObject: calling handler <function add_expect_header at 0x7f2f22c39580>
2024-03-18 14:31:05,727 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event before-call.s3.GetObject: calling handler <bound method S3ExpressIdentityResolver.apply_signing_cache_key of <botocore.utils.S3ExpressIdentityResolver object at 0x7f2f1bda6510>>
2024-03-18 14:31:05,727 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event before-call.s3.GetObject: calling handler <function inject_api_version_header_if_needed at 0x7f2f22c3aac0>
2024-03-18 14:31:05,727 - ThreadPoolExecutor-0_0 - botocore.endpoint - DEBUG - Making request for OperationModel(name=GetObject) with params: {'url_path': '/path/to/schema_tache_info_suivi_prod.json', 'query_string': {}, 'method': 'GET', 'headers': {'User-Agent': 'aws-cli/2.15.30 Python/3.11.8 Linux/6.2.0-1017-aws source/x86_64.debian.12 prompt/off command/s3.cp'}, 'body': b'', 'auth_path': '/geostore-[...]/path/to/schema_tache_info_suivi_prod.json', 'url': 'https://geostore-[...].s3.ca-central-1.amazonaws.com/path/to/schema_tache_info_suivi_prod.json', 'context': {'client_region': 'ca-central-1', 'client_config': <botocore.config.Config object at 0x7f2f1bd99950>, 'has_streaming_input': False, 'auth_type': 'v4', 'signing': {'region': 'ca-central-1', 'signing_name': 's3', 'disableDoubleEncoding': True}, 'endpoint_properties': {'authSchemes': [{'disableDoubleEncoding': True, 'name': 'sigv4', 'signingName': 's3', 'signingRegion': 'ca-central-1'}]}, 's3_redirect': {'redirected': False, 'bucket': 'geostore-[...]', 'params': {'Bucket': 'geostore-[...]', 'Key': 'path/to/schema_tache_info_suivi_prod.json'}}, 'S3Express': {'bucket_name': 'geostore-[...]'}}}
2024-03-18 14:31:05,728 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event request-created.s3.GetObject: calling handler <function signal_not_transferring at 0x7f2f21331260>
2024-03-18 14:31:05,728 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event request-created.s3.GetObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f2f1bd61dd0>>
2024-03-18 14:31:05,728 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event choose-signer.s3.GetObject: calling handler <function set_operation_specific_signer at 0x7f2f22c38ea0>
2024-03-18 14:31:05,728 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event before-sign.s3.GetObject: calling handler <function remove_arn_from_signing_path at 0x7f2f22c3b380>
2024-03-18 14:31:05,728 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event before-sign.s3.GetObject: calling handler <bound method S3ExpressIdentityResolver.resolve_s3express_identity of <botocore.utils.S3ExpressIdentityResolver object at 0x7f2f1bda6510>>
2024-03-18 14:31:05,728 - ThreadPoolExecutor-0_0 - botocore.auth - DEBUG - Calculating signature using v4 auth.
2024-03-18 14:31:05,728 - ThreadPoolExecutor-0_0 - botocore.auth - DEBUG - CanonicalRequest:
GET
/path/to/schema_tache_info_suivi_prod.json

host:geostore-[...].s3.ca-central-1.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20240318T143105Z
x-amz-security-token:IQoJb3JpZ2luX2VjEM7//////////wEaDGNhLWNlbnRyYWwtMSJIMEYCIQDQyA9pL9hNCyQweUEOroUI+bXgT2B/FEEHi8i43BA8PwIhAKJyfzrRZRPae59BAZjTJ0ReuTS8n1yUXf0HolejTVg/KroDCNj//////////wEQARoMNzk1NzYzNTM4MzY4Igw2zjmR8NhB5v3F7WYqjgOrJVdUi4rP8ViqSe0t0S5A/gNT6NMs8/AcWsh1w44JWIMkuSnI8IUyQLBjcNs2g0gas4fAPZ9wzBblQ37gRVcpjn9eOGyH9uDXzqdiB9OBc6aUrAtmh987Yh/zV8ByDIY9x8ANwlmtwN613s/dkIHsk8iVTxI6xg4DB9ksHTAyOUjiWL0+AWILKViWLOQJWRvvGcR+SwnCNz2hQFlQm7Wu+g8I+mcN8cA68EM7yIxFsgXE5J4/zolvO8G5UkE/9ewuZufnjIdWNHGz9eC+qiVyHdwbRRiNEAWejO4ZrkHBaaqexlCc1lAX5We1q446E51VbeZzwLw7+97eAjk0Dpx2McUIjJLzkrN5IRM9+0yHmNm7sQA8F/0/SAbmEDQQto8i7WiM8//yNIo21yXKuHaC1926mn2/6dN3Zffs256l7XvKDZCXFQKaV7sbUtKd2f25JOXFEEItzZBBwFRJOXH0vtJYnaSreOq8oOfGFqm86QvrFu9DU5aiuIkEH9ZEIQrF5jaTvzFAfBTgyc10ojD+nuGvBjqYASiwCZrP271FaVXipfCUQJdrS5jihFPY7javaPnIcQcRVWURCu7GjXjfsKsCDyKyhsi9dsdoyovO11rwr7oW+YAd1laEH/Kij40HnNia9q2DnJe0rMV6Kf7NoiAgDStTWqPxAQCqCjcGEbDUwEvrOJL4hlp8JtfodF6zz3TcxBK9IiV0yQ+obUlqeHVjnlCjqfvKSwbGamh2

host;x-amz-content-sha256;x-amz-date;x-amz-security-token
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2024-03-18 14:31:05,728 - ThreadPoolExecutor-0_0 - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20240318T143105Z
20240318/ca-central-1/s3/aws4_request
2982b49ecab408c9b84d19a135cbcf74d3255a279d368342b9646160ed5188ac
2024-03-18 14:31:05,728 - ThreadPoolExecutor-0_0 - botocore.auth - DEBUG - Signature:
d55704d09846bec88b9a48951b79b2a30af9a90ca0296838f64fd9ac1c028fc7
2024-03-18 14:31:05,729 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event request-created.s3.GetObject: calling handler <function signal_transferring at 0x7f2f21331580>
2024-03-18 14:31:05,729 - ThreadPoolExecutor-0_0 - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=True, method=GET, url=https://geostore-[...].s3.ca-central-1.amazonaws.com/path/to/schema_tache_info_suivi_prod.json, headers={'User-Agent': b'aws-cli/2.15.30 Python/3.11.8 Linux/6.2.0-1017-aws source/x86_64.debian.12 prompt/off command/s3.cp', 'X-Amz-Date': b'20240318T143105Z', 'X-Amz-Security-Token': b'IQoJb3JpZ2luX2VjEM7//////////wEaDGNhLWNlbnRyYWwtMSJIMEYCIQDQyA9pL9hNCyQweUEOroUI+bXgT2B/FEEHi8i43BA8PwIhAKJyfzrRZRPae59BAZjTJ0ReuTS8n1yUXf0HolejTVg/KroDCNj//////////wEQARoMNzk1NzYzNTM4MzY4Igw2zjmR8NhB5v3F7WYqjgOrJVdUi4rP8ViqSe0t0S5A/gNT6NMs8/AcWsh1w44JWIMkuSnI8IUyQLBjcNs2g0gas4fAPZ9wzBblQ37gRVcpjn9eOGyH9uDXzqdiB9OBc6aUrAtmh987Yh/zV8ByDIY9x8ANwlmtwN613s/dkIHsk8iVTxI6xg4DB9ksHTAyOUjiWL0+AWILKViWLOQJWRvvGcR+SwnCNz2hQFlQm7Wu+g8I+mcN8cA68EM7yIxFsgXE5J4/zolvO8G5UkE/9ewuZufnjIdWNHGz9eC+qiVyHdwbRRiNEAWejO4ZrkHBaaqexlCc1lAX5We1q446E51VbeZzwLw7+97eAjk0Dpx2McUIjJLzkrN5IRM9+0yHmNm7sQA8F/0/SAbmEDQQto8i7WiM8//yNIo21yXKuHaC1926mn2/6dN3Zffs256l7XvKDZCXFQKaV7sbUtKd2f25JOXFEEItzZBBwFRJOXH0vtJYnaSreOq8oOfGFqm86QvrFu9DU5aiuIkEH9ZEIQrF5jaTvzFAfBTgyc10ojD+nuGvBjqYASiwCZrP271FaVXipfCUQJdrS5jihFPY7javaPnIcQcRVWURCu7GjXjfsKsCDyKyhsi9dsdoyovO11rwr7oW+YAd1laEH/Kij40HnNia9q2DnJe0rMV6Kf7NoiAgDStTWqPxAQCqCjcGEbDUwEvrOJL4hlp8JtfodF6zz3TcxBK9IiV0yQ+obUlqeHVjnlCjqfvKSwbGamh2', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=ASIA3SRZT7HANIDGZNFP/20240318/ca-central-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=d55704d09846bec88b9a48951b79b2a30af9a90ca0296838f64fd9ac1c028fc7'}>
2024-03-18 14:31:05,729 - ThreadPoolExecutor-0_0 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): geostore-[...].s3.ca-central-1.amazonaws.com:443
2024-03-18 14:31:05,798 - ThreadPoolExecutor-0_0 - urllib3.connectionpool - DEBUG - https://geostore-[...].s3.ca-central-1.amazonaws.com:443 "GET /path/to/schema_tache_info_suivi_prod.json HTTP/1.1" 200 2438
2024-03-18 14:31:05,798 - ThreadPoolExecutor-0_0 - botocore.parsers - DEBUG - Response headers: {'x-amz-id-2': 'pTSaiuL6y+xvczt9oNy4q1p3Mz8EBG/HZmawzOCrMxsKXBElZZRXF0GyodT69u72gupvFEaTeaM=', 'x-amz-request-id': '6102XTK2CSNHG01D', 'Date': 'Mon, 18 Mar 2024 14:31:06 GMT', 'Last-Modified': 'Wed, 21 Jun 2023 19:57:05 GMT', 'ETag': '"99bc7b99184be3266f664f9d65a513e2"', 'x-amz-server-side-encryption': 'AES256', 'x-amz-meta-!gapi!created!': '2023-06-21T19:57:04 | GAPI geoai | [POST] /v1/amazon/fichiers', 'x-amz-version-id': 'ngo4GbOLwtelOFd9ogRlmnLwMiIA89DY', 'Accept-Ranges': 'bytes', 'Content-Type': 'binary/octet-stream', 'Server': 'AmazonS3', 'Content-Length': '2438'}
2024-03-18 14:31:05,798 - ThreadPoolExecutor-0_0 - botocore.parsers - DEBUG - Response body:
<botocore.response.StreamingBody object at 0x7f2f1bdd0810>
2024-03-18 14:31:05,799 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event needs-retry.s3.GetObject: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x7f2f1bda6150>>
2024-03-18 14:31:05,799 - ThreadPoolExecutor-0_0 - botocore.retries.standard - DEBUG - Not retrying request.
2024-03-18 14:31:05,799 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event needs-retry.s3.GetObject: calling handler <bound method S3RegionRedirectorv2.redirect_from_error of <botocore.utils.S3RegionRedirectorv2 object at 0x7f2f1bda5350>>
2024-03-18 14:31:05,799 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event after-call.s3.GetObject: calling handler <function enhance_error_msg at 0x7f2f2105afc0>
2024-03-18 14:31:05,799 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event after-call.s3.GetObject: calling handler <bound method RetryQuotaChecker.release_retry_quota of <botocore.retries.standard.RetryQuotaChecker object at 0x7f2f1bda5c10>>
2024-03-18 14:31:05,799 - ThreadPoolExecutor-0_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 0}) about to wait for the following futures []
2024-03-18 14:31:05,799 - ThreadPoolExecutor-0_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 0}) done waiting for dependent futures
2024-03-18 14:31:05,799 - ThreadPoolExecutor-0_0 - s3transfer.tasks - DEBUG - Executing task IOWriteTask(transfer_id=0, {'offset': 0}) with kwargs {'fileobj': <s3transfer.utils.DeferredOpenFile object at 0x7f2f1bdd0e10>, 'offset': 0}
2024-03-18 14:31:05,800 - ThreadPoolExecutor-0_0 - s3transfer.tasks - DEBUG - IORenameFileTask(transfer_id=0, {'final_filename': '/tmp/schema_tache_info_suivi_prod.json'}) about to wait for the following futures []
2024-03-18 14:31:05,800 - ThreadPoolExecutor-0_0 - s3transfer.tasks - DEBUG - IORenameFileTask(transfer_id=0, {'final_filename': '/tmp/schema_tache_info_suivi_prod.json'}) done waiting for dependent futures
2024-03-18 14:31:05,800 - ThreadPoolExecutor-0_0 - s3transfer.tasks - DEBUG - Executing task IORenameFileTask(transfer_id=0, {'final_filename': '/tmp/schema_tache_info_suivi_prod.json'}) with kwargs {'fileobj': <s3transfer.utils.DeferredOpenFile object at 0x7f2f1bdd0e10>, 'final_filename': '/tmp/schema_tache_info_suivi_prod.json', 'osutil': <s3transfer.utils.OSUtils object at 0x7f2f21c7a4d0>}
2024-03-18 14:31:05,800 - ThreadPoolExecutor-0_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
download: s3://geostore-[...]/path/to/schema_tache_info_suivi_prod.json to tmp/schema_tache_info_suivi_prod.json

@remtav
Copy link
Author

remtav commented Mar 27, 2024 •
edited
Loading

@rouault are there any other ways I can gain clarify on the cause?
The bug is impacting a number of our processes in production. We can temporarily have them run on our HPC, but that's not ideal. My colleague mentionned that the bug doesn't arise when the EC2 instance has permissions for the bucket via it's IAM Role. It's also something I'll try, but it couldn't be a permanent solution for us either.

@rouault
Copy link
Member

rouault commented Mar 27, 2024

are there any other ways I can gain clarify on the cause?

(the complexity of cloud authentication schemes is killing me and the more time, the less I understand them as they keep complexifying them. It is as if the cloud providers didn't want mortals to implement their protocol but use their heavy weight SDKs)

So the issue only occurs on EC2 instances? Perhaps the EC2 detection logic in GDAL is triggered then whereas it shouldn't for your particular case, and overrides the token you provided with the one of your IAM account? You might try to edit port/cpl_aws.cpp bool VSIS3HandleHelper::GetConfigurationFromEC2() and add an early "return false;" at its top. Or perhaps set the CPL_AWS_EC2_API_ROOT_URL environment variable to a non existing URL like "http://0.0.0.0/" (this may cause some timeouts, so likely not something you want as the final solution, but more as a debugging tool). But looking more closely at VSIS3HandleHelper::GetConfiguration(), given that you set AWS_SECRET_ACCESS_KEY, AWS_ACCESS_KEY_ID and AWS_SESSION_TOKEN, this code path shouldn't trigger... So its weird that the change in permissions for the IAM role has an influence. There's something weird. Are you sure the variables you think are defined are really defined?

You could also potentially build GDAL with -DDEBUG_VERBOSE (or uncomment #define DEBUG_VERBOSE 1 at the top of port/cpl_aws.cpp), and look at the "osStringToSign" trace that will be then output in debug mode, to see if it contains the same elements as the ones of the Python SDK debug traces?

@remtav
Copy link
Author

remtav commented Apr 30, 2024 •
edited
Loading

Some further testing has shown that setting the following env variable this way solved our problem:
export AWS_S3_ENDPOINT=s3.ca-central-1.amazonaws.com

Simple enough...
Thanks to @carlstlaurent and @micpilon

@remtav remtav closed this as completed Apr 30, 2024
@remtav
Copy link
Author

remtav commented Apr 30, 2024 •
edited
Loading

Details

see next post for insights and explanations: #9444 (comment)

AWS-CLI vs GDAL

The solution found mainly comes from comparing debug logs from operations performed with aws-cli vs GDAL.

Here are the full logs for both operations, followed by our analysis and details about our explanation of the issue.

Call stack from AWS-CLI

input command: aws s3 ls --debug s3://foostore-foo-fooapistore/

full log:

2024-04-30 17:16:43,486 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/1.32.94 Python/3.8.10 Linux/6.5.0-1016-aws botocore/1.34.94
2024-04-30 17:16:43,486 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['s3', 'ls', '--debug', 's3://foostore-foo-fooapistore/']
2024-04-30 17:16:43,486 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_scalar_parsers at 0x7986fcd6c310>
2024-04-30 17:16:43,486 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function register_uri_param_handler at 0x7986fd37e9d0>
2024-04-30 17:16:43,487 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x7986fd2df700>
2024-04-30 17:16:43,488 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/
2024-04-30 17:16:43,489 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function attach_history_handler at 0x7986fce9f1f0>
2024-04-30 17:16:43,489 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3: calling handler <function add_waiters at 0x7986fcd8a940>
2024-04-30 17:16:43,490 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.s3: calling handler <function update_endpoint_url at 0x7986fcd278b0>
2024-04-30 17:16:43,490 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.s3.anonymous: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7986fcc71550>
2024-04-30 17:16:43,490 - MainThread - botocore.hooks - DEBUG - Event building-command-table.ls: calling handler <function add_waiters at 0x7986fcd8a940>
2024-04-30 17:16:43,490 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.s3.ls: calling handler <function update_endpoint_url at 0x7986fcd278b0>
2024-04-30 17:16:43,491 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.paths: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7986fcc71550>
2024-04-30 17:16:43,491 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.anonymous: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7986fcc71550>
2024-04-30 17:16:43,491 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.page-size: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7986fcc71550>
2024-04-30 17:16:43,491 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.human-readable: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7986fcc71550>
2024-04-30 17:16:43,491 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.ls: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7986fd325f10>
2024-04-30 17:16:43,491 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.summarize: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7986fcc71550>
2024-04-30 17:16:43,492 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.ls: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7986fd325f10>
2024-04-30 17:16:43,492 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.request-payer: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7986fcc71550>
2024-04-30 17:16:43,492 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2024-04-30 17:16:43,492 - MainThread - botocore.credentials - INFO - Found credentials in environment variables.
2024-04-30 17:16:43,493 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/lib/python3.8/dist-packages/botocore/data/endpoints.json
2024-04-30 17:16:43,516 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/lib/python3.8/dist-packages/botocore/data/sdk-default-configuration.json
2024-04-30 17:16:43,516 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x7986fe3eee50>
2024-04-30 17:16:43,522 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/lib/python3.8/dist-packages/botocore/data/s3/2006-03-01/service-2.json.gz
2024-04-30 17:16:43,538 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/lib/python3.8/dist-packages/botocore/data/s3/2006-03-01/endpoint-rule-set-1.json.gz
2024-04-30 17:16:43,542 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/lib/python3.8/dist-packages/botocore/data/partitions.json
2024-04-30 17:16:43,544 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7986fd8891f0>
2024-04-30 17:16:43,544 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7986fd884f70>
2024-04-30 17:16:43,545 - MainThread - botocore.configprovider - DEBUG - Looking for endpoint for s3 via: environment_service
2024-04-30 17:16:43,545 - MainThread - botocore.configprovider - DEBUG - Looking for endpoint for s3 via: environment_global
2024-04-30 17:16:43,545 - MainThread - botocore.configprovider - DEBUG - Looking for endpoint for s3 via: config_service
2024-04-30 17:16:43,545 - MainThread - botocore.configprovider - DEBUG - Looking for endpoint for s3 via: config_global
2024-04-30 17:16:43,545 - MainThread - botocore.configprovider - DEBUG - No configured endpoint found.
2024-04-30 17:16:43,547 - MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60)
2024-04-30 17:16:43,549 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/lib/python3.8/dist-packages/botocore/data/_retry.json
2024-04-30 17:16:43,549 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: s3
2024-04-30 17:16:43,550 - MainThread - botocore.utils - DEBUG - Registering S3 region redirector handler
2024-04-30 17:16:43,550 - MainThread - botocore.utils - DEBUG - Registering S3Express Identity Resolver
2024-04-30 17:16:43,559 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/lib/python3.8/dist-packages/botocore/data/s3/2006-03-01/paginators-1.json
2024-04-30 17:16:43,559 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/lib/python3.8/dist-packages/botocore/data/s3/2006-03-01/paginators-1.sdk-extras.json
2024-04-30 17:16:43,560 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <function set_list_objects_encoding_type_url at 0x7986fd7ccee0>
2024-04-30 17:16:43,560 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <function validate_bucket_name at 0x7986fd7cac10>
2024-04-30 17:16:43,560 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <function remove_bucket_from_url_paths_from_model at 0x7986fd750a60>
2024-04-30 17:16:43,560 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <bound method S3RegionRedirectorv2.annotate_request_context of <botocore.utils.S3RegionRedirectorv2 object at 0x7986fc2fc040>>
2024-04-30 17:16:43,560 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <bound method ClientCreator._inject_s3_input_parameters of <botocore.client.ClientCreator object at 0x7986fd3289a0>>
2024-04-30 17:16:43,560 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <function generate_idempotent_uuid at 0x7986fd7caa60>
2024-04-30 17:16:43,560 - MainThread - botocore.hooks - DEBUG - Event before-endpoint-resolution.s3: calling handler <function customize_endpoint_resolver_builtins at 0x7986fd750c10>
2024-04-30 17:16:43,560 - MainThread - botocore.hooks - DEBUG - Event before-endpoint-resolution.s3: calling handler <bound method S3RegionRedirectorv2.redirect_from_cache of <botocore.utils.S3RegionRedirectorv2 object at 0x7986fc2fc040>>
2024-04-30 17:16:43,561 - MainThread - botocore.regions - DEBUG - Calling endpoint provider with parameters: {'Bucket': 'foostore-foo-fooapistore', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'ForcePathStyle': False, 'Accelerate': False, 'UseGlobalEndpoint': True, 'Prefix': '', 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True}
2024-04-30 17:16:43,561 - MainThread - botocore.regions - DEBUG - Endpoint provider result: https://foostore-foo-fooapistore.s3.amazonaws.com
2024-04-30 17:16:43,561 - MainThread - botocore.regions - DEBUG - Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "None"
2024-04-30 17:16:43,561 - MainThread - botocore.regions - DEBUG - Selected auth type "v4" as "v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True}
2024-04-30 17:16:43,562 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.ListObjectsV2: calling handler <function add_expect_header at 0x7986fd7caf70>
2024-04-30 17:16:43,562 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.ListObjectsV2: calling handler <bound method S3ExpressIdentityResolver.apply_signing_cache_key of <botocore.utils.S3ExpressIdentityResolver object at 0x7986fc2fc100>>
2024-04-30 17:16:43,562 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.ListObjectsV2: calling handler <function add_recursion_detection_header at 0x7986fd7ca700>
2024-04-30 17:16:43,562 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.ListObjectsV2: calling handler <function inject_api_version_header_if_needed at 0x7986fd750310>
2024-04-30 17:16:43,562 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=ListObjectsV2) with params: {'url_path': '?list-type=2', 'query_string': {'prefix': '', 'delimiter': '/', 'encoding-type': 'url'}, 'method': 'GET', 'headers': {'User-Agent': 'aws-cli/1.32.94 md/Botocore#1.34.94 ua/2.0 os/linux#6.5.0-1016-aws md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy botocore/1.34.94'}, 'body': b'', 'auth_path': '/foostore-foo-fooapistore/', 'url': 'https://foostore-foo-fooapistore.s3.amazonaws.com/?list-type=2&prefix=&delimiter=%2F&encoding-type=url', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7986fc385880>, 'has_streaming_input': False, 'auth_type': 'v4', 'encoding_type_auto_set': True, 's3_redirect': {'redirected': False, 'bucket': 'foostore-foo-fooapistore', 'params': {'Bucket': 'foostore-foo-fooapistore', 'Prefix': '', 'Delimiter': '/', 'EncodingType': 'url'}}, 'input_params': {'Bucket': 'foostore-foo-fooapistore', 'Prefix': ''}, 'signing': {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True}, 'endpoint_properties': {'authSchemes': [{'disableDoubleEncoding': True, 'name': 'sigv4', 'signingName': 's3', 'signingRegion': 'us-east-1'}]}}}
2024-04-30 17:16:43,562 - MainThread - botocore.hooks - DEBUG - Event request-created.s3.ListObjectsV2: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7986fc385a00>>
2024-04-30 17:16:43,562 - MainThread - botocore.hooks - DEBUG - Event choose-signer.s3.ListObjectsV2: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7986fd3289a0>>
2024-04-30 17:16:43,562 - MainThread - botocore.hooks - DEBUG - Event choose-signer.s3.ListObjectsV2: calling handler <function set_operation_specific_signer at 0x7986fd7ca940>
2024-04-30 17:16:43,563 - MainThread - botocore.hooks - DEBUG - Event before-sign.s3.ListObjectsV2: calling handler <function remove_arn_from_signing_path at 0x7986fd750b80>
2024-04-30 17:16:43,563 - MainThread - botocore.hooks - DEBUG - Event before-sign.s3.ListObjectsV2: calling handler <bound method S3ExpressIdentityResolver.resolve_s3express_identity of <botocore.utils.S3ExpressIdentityResolver object at 0x7986fc2fc100>>
2024-04-30 17:16:43,563 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2024-04-30 17:16:43,563 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
GET
/
delimiter=%2F&encoding-type=url&list-type=2&prefix=
host:foostore-foo-fooapistore.s3.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20240430T171643Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2024-04-30 17:16:43,563 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20240430T171643Z
20240430/us-east-1/s3/aws4_request
35a8d45a56a37fa46d06190567d01aa7add9e6930c577b0718e840e691e154a0
2024-04-30 17:16:43,563 - MainThread - botocore.auth - DEBUG - Signature:
ff92e7f70e5f524dbc345a847fd408703b740830a0480382538ef2bb056fbfc3
2024-04-30 17:16:43,563 - MainThread - botocore.hooks - DEBUG - Event request-created.s3.ListObjectsV2: calling handler <function add_retry_headers at 0x7986fd7509d0>
2024-04-30 17:16:43,563 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=GET, url=https://foostore-foo-fooapistore.s3.amazonaws.com/?list-type=2&prefix=&delimiter=%2F&encoding-type=url, headers={'User-Agent': b'aws-cli/1.32.94 md/Botocore#1.34.94 ua/2.0 os/linux#6.5.0-1016-aws md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy botocore/1.34.94', 'X-Amz-Date': b'20240430T171643Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIA3SRZT7HAMJHOOAN3/20240430/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=ff92e7f70e5f524dbc345a847fd408703b740830a0480382538ef2bb056fbfc3', 'amz-sdk-invocation-id': b'27e1b631-0f00-43a9-8686-bdcb957a8a64', 'amz-sdk-request': b'attempt=1'}>
2024-04-30 17:16:43,564 - MainThread - botocore.httpsession - DEBUG - Certificate path: /usr/local/lib/python3.8/dist-packages/botocore/cacert.pem
2024-04-30 17:16:43,564 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): foostore-foo-fooapistore.s3.amazonaws.com:443
2024-04-30 17:16:43,580 - MainThread - urllib3.connectionpool - DEBUG - https://foostore-foo-fooapistore.s3.amazonaws.com:443 "GET /?list-type=2&prefix=&delimiter=%2F&encoding-type=url HTTP/1.1" 400 None
2024-04-30 17:16:43,581 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amz-bucket-region': 'ca-central-1', 'x-amz-request-id': 'HSN36ENA6YZPHVF4', 'x-amz-id-2': 'ecJHJWtIIOCydWy10654JZEBfdiQFqbPPouTOpA+xNkW+OpTVoYpYbYOBdM9Ejidqf+XJCzQeLE=', 'Content-Type': 'application/xml', 'Transfer-Encoding': 'chunked', 'Date': 'Tue, 30 Apr 2024 17:16:42 GMT', 'Server': 'AmazonS3', 'Connection': 'close'}
2024-04-30 17:16:43,581 - MainThread - botocore.parsers - DEBUG - Response body:
b'<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>AuthorizationHeaderMalformed</Code><Message>The authorization header is malformed; the region \'us-east-1\' is wrong; expecting \'ca-central-1\'</Message><Region>ca-central-1</Region><RequestId>HSN36ENA6YZPHVF4</RequestId><HostId>ecJHJWtIIOCydWy10654JZEBfdiQFqbPPouTOpA+xNkW+OpTVoYpYbYOBdM9Ejidqf+XJCzQeLE=</HostId></Error>'
2024-04-30 17:16:43,582 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListObjectsV2: calling handler <botocore.retryhandler.RetryHandler object at 0x7986fc36bfa0>
2024-04-30 17:16:43,582 - MainThread - botocore.retryhandler - DEBUG - No retry needed.
2024-04-30 17:16:43,583 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListObjectsV2: calling handler <bound method S3RegionRedirectorv2.redirect_from_error of <botocore.utils.S3RegionRedirectorv2 object at 0x7986fc2fc040>>
2024-04-30 17:16:43,583 - MainThread - botocore.utils - DEBUG - S3 client configured for region us-east-1 but the bucket foostore-foo-fooapistore is in region ca-central-1; Please configure the proper region to avoid multiple unnecessary redirects and signing attempts.
2024-04-30 17:16:43,583 - MainThread - botocore.hooks - DEBUG - Event before-endpoint-resolution.s3: calling handler <function customize_endpoint_resolver_builtins at 0x7986fd750c10>
2024-04-30 17:16:43,583 - MainThread - botocore.hooks - DEBUG - Event before-endpoint-resolution.s3: calling handler <bound method S3RegionRedirectorv2.redirect_from_cache of <botocore.utils.S3RegionRedirectorv2 object at 0x7986fc2fc040>>
2024-04-30 17:16:43,583 - MainThread - botocore.regions - DEBUG - Calling endpoint provider with parameters: {'Bucket': 'foostore-foo-fooapistore', 'Region': 'ca-central-1', 'UseFIPS': False, 'UseDualStack': False, 'ForcePathStyle': False, 'Accelerate': False, 'UseGlobalEndpoint': True, 'Prefix': '', 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True}
2024-04-30 17:16:43,583 - MainThread - botocore.regions - DEBUG - Endpoint provider result: https://foostore-foo-fooapistore.s3.ca-central-1.amazonaws.com
2024-04-30 17:16:43,583 - MainThread - botocore.utils - DEBUG - Updating URI from https://foostore-foo-fooapistore.s3.amazonaws.com/?list-type=2&prefix=&delimiter=%2F&encoding-type=url to https://foostore-foo-fooapistore.s3.ca-central-1.amazonaws.com/?list-type=2&prefix=&delimiter=%2F&encoding-type=url
2024-04-30 17:16:43,583 - MainThread - botocore.regions - DEBUG - Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "None"
2024-04-30 17:16:43,583 - MainThread - botocore.regions - DEBUG - Selected auth type "v4" as "v4" with signing context params: {'region': 'ca-central-1', 'signing_name': 's3', 'disableDoubleEncoding': True}
2024-04-30 17:16:43,583 - MainThread - botocore.endpoint - DEBUG - Response received to retry, sleeping for 0 seconds
2024-04-30 17:16:43,585 - MainThread - botocore.hooks - DEBUG - Event request-created.s3.ListObjectsV2: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7986fc385a00>>
2024-04-30 17:16:43,585 - MainThread - botocore.hooks - DEBUG - Event choose-signer.s3.ListObjectsV2: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7986fd3289a0>>
2024-04-30 17:16:43,585 - MainThread - botocore.hooks - DEBUG - Event choose-signer.s3.ListObjectsV2: calling handler <function set_operation_specific_signer at 0x7986fd7ca940>
2024-04-30 17:16:43,585 - MainThread - botocore.hooks - DEBUG - Event before-sign.s3.ListObjectsV2: calling handler <function remove_arn_from_signing_path at 0x7986fd750b80>
2024-04-30 17:16:43,585 - MainThread - botocore.hooks - DEBUG - Event before-sign.s3.ListObjectsV2: calling handler <bound method S3ExpressIdentityResolver.resolve_s3express_identity of <botocore.utils.S3ExpressIdentityResolver object at 0x7986fc2fc100>>
2024-04-30 17:16:43,585 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2024-04-30 17:16:43,585 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
GET
/
delimiter=%2F&encoding-type=url&list-type=2&prefix=
host:foostore-foo-fooapistore.s3.ca-central-1.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20240430T171643Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2024-04-30 17:16:43,586 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20240430T171643Z
20240430/ca-central-1/s3/aws4_request
4d67aab1d0b722ab23f02342e7c0d2f517b759c74164532b2319e3586ef94076
2024-04-30 17:16:43,586 - MainThread - botocore.auth - DEBUG - Signature:
3a04dc62c87e835c9ca2af17c5037e106845cf969afb668fc29a365d764b0f35
2024-04-30 17:16:43,586 - MainThread - botocore.hooks - DEBUG - Event request-created.s3.ListObjectsV2: calling handler <function add_retry_headers at 0x7986fd7509d0>
2024-04-30 17:16:43,586 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=GET, url=https://foostore-foo-fooapistore.s3.ca-central-1.amazonaws.com/?list-type=2&prefix=&delimiter=%2F&encoding-type=url, headers={'User-Agent': b'aws-cli/1.32.94 md/Botocore#1.34.94 ua/2.0 os/linux#6.5.0-1016-aws md/arch#x86_64 lang/python#3.8.10 md/pyimpl#CPython cfg/retry-mode#legacy botocore/1.34.94', 'X-Amz-Date': b'20240430T171643Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIA3SRZT7HAMJHOOAN3/20240430/ca-central-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=3a04dc62c87e835c9ca2af17c5037e106845cf969afb668fc29a365d764b0f35', 'amz-sdk-invocation-id': b'27e1b631-0f00-43a9-8686-bdcb957a8a64', 'amz-sdk-request': b'ttl=20240430T171742Z; attempt=2; max=5'}>
2024-04-30 17:16:43,586 - MainThread - botocore.httpsession - DEBUG - Certificate path: /usr/local/lib/python3.8/dist-packages/botocore/cacert.pem
2024-04-30 17:16:43,586 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): foostore-foo-fooapistore.s3.ca-central-1.amazonaws.com:443
2024-04-30 17:16:43,640 - MainThread - urllib3.connectionpool - DEBUG - https://foostore-foo-fooapistore.s3.ca-central-1.amazonaws.com:443 "GET /?list-type=2&prefix=&delimiter=%2F&encoding-type=url HTTP/1.1" 200 None
2024-04-30 17:16:43,641 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amz-id-2': 'skWXzz77FVLr0IKhTCZf/6sga6kVPNJicXsK0c/lRprb4vghgRTVKd/PUVDc9ji2V+733iJ1wEk=', 'x-amz-request-id': 'HSN1A17X0ZWG4B84', 'Date': 'Tue, 30 Apr 2024 17:16:44 GMT', 'x-amz-bucket-region': 'ca-central-1', 'Content-Type': 'application/xml', 'Transfer-Encoding': 'chunked', 'Server': 'AmazonS3'}
2024-04-30 17:16:43,641 - MainThread - botocore.parsers - DEBUG - Response body:
b'<?xml version="1.0" encoding="UTF-8"?>\n<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>foostore-foo-fooapistore</Name><Prefix></Prefix><KeyCount>8</KeyCount><MaxKeys>1000</MaxKeys><Delimiter>/</Delimiter><EncodingType>url</EncodingType><IsTruncated>false</IsTruncated><CommonPrefixes><Prefix>foo/</Prefix></CommonPrefixes><CommonPrefixes><Prefix>admin/</Prefix></CommonPrefixes><CommonPrefixes><Prefix>apidata/</Prefix></CommonPrefixes><CommonPrefixes><Prefix>backup/</Prefix></CommonPrefixes><CommonPrefixes><Prefix>data/</Prefix></CommonPrefixes><CommonPrefixes><Prefix>external/</Prefix></CommonPrefixes><CommonPrefixes><Prefix>home/</Prefix></CommonPrefixes><CommonPrefixes><Prefix>pub/</Prefix></CommonPrefixes></ListBucketResult>'
2024-04-30 17:16:43,641 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListObjectsV2: calling handler <botocore.retryhandler.RetryHandler object at 0x7986fc36bfa0>
2024-04-30 17:16:43,641 - MainThread - botocore.retryhandler - DEBUG - No retry needed.
2024-04-30 17:16:43,641 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListObjectsV2: calling handler <bound method S3RegionRedirectorv2.redirect_from_error of <botocore.utils.S3RegionRedirectorv2 object at 0x7986fc2fc040>>
2024-04-30 17:16:43,642 - MainThread - botocore.utils - DEBUG - S3 request was previously redirected, not redirecting.
2024-04-30 17:16:43,642 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.ListObjectsV2: calling handler <function decode_list_object_v2 at 0x7986fd750040>
2024-04-30 17:16:43,642 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.ListObjectsV2: calling handler <function enhance_error_msg at 0x7986fcd67e50>
                           PRE dir1/
                           PRE dir2/
                           PRE dir3/
                           PRE backup/
                           PRE data/
                           PRE external/
                           PRE home/
                           PRE pub/

Call stack from GDAL

input command: gdalinfo /vsis3/foostore-foo-fooapistore/.../AB1-img-B.tif

full log:

HTTP: libcurl/7.68.0 GnuTLS/3.6.13 zlib/1.2.11 brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.2.0) libssh/0.9.3/openssl/zlib nghttp2/1.40.0 librtmp/2.3
* Couldn't find host foostore-foo-fooapistore.s3.amazonaws.com in the .netrc file; using defaults
*   Trying 52.95.190.156:443...
* TCP_NODELAY set
* Connected to foostore-foo-fooapistore.s3.amazonaws.com (52.95.190.156) port 443 (#0)
* found 387 certificates in /etc/ssl/certs
* ALPN, offering h2
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
*        server certificate verification OK
*        server certificate status verification SKIPPED
*        common name: *.s3.amazonaws.com (matched)
*        server certificate expiration date OK
*        server certificate activation date OK
*        certificate public key: RSA
*        certificate version: #3
*        subject: CN=*.s3.amazonaws.com
*        start date: Tue, 10 Oct 2023 00:00:00 GMT
*        expire date: Wed, 03 Jul 2024 23:59:59 GMT
*        issuer: C=US,O=Amazon,CN=Amazon RSA 2048 M01
* ALPN, server accepted to use http/1.1
> GET /?delimiter=%2F&prefix=data%2Ffoo%2Fprocessed%2Ffooprovider%2FAB1%2F HTTP/1.1
Host: foostore-foo-fooapistore.s3.amazonaws.com
Accept: */*
x-amz-date: 20240430T171810Z
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Authorization: AWS4-HMAC-SHA256 Credential=AKIA3SRZT7HAMJHOOAN3/20240430/ca-central-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=80884551ff3185c15ba1e4c781a17d6be70465e035a28b4515bdc994d5edfc95

* Mark bundle as not supporting multiuse
< HTTP/1.1 403 Forbidden
< x-amz-bucket-region: ca-central-1
< x-amz-request-id: YVXXKK4DJX46PB4J
< x-amz-id-2: nLhGuvQtUuqOevMLMTpBmvw5Wax1lEHcOkxPpX98B+mZ9bFb5+6ElnJc8towAdCx+ZbVfvNvb1I=
< Content-Type: application/xml
< Transfer-Encoding: chunked
< Date: Tue, 30 Apr 2024 17:18:10 GMT
< Server: AmazonS3
< 
* Connection #0 to host foostore-foo-fooapistore.s3.amazonaws.com left intact
S3: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>YVXXKK4DJX46PB4J</RequestId><HostId>nLhGuvQtUuqOevMLMTpBmvw5Wax1lEHcOkxPpX98B+mZ9bFb5+6ElnJc8towAdCx+ZbVfvNvb1I=</HostId></Error>
* Couldn't find host foostore-foo-fooapistore.s3.amazonaws.com in the .netrc file; using defaults
* Found bundle for host foostore-foo-fooapistore.s3.amazonaws.com: 0x5a41e1de9d40 [serially]
* Can not multiplex, even if we wanted to!
* Re-using existing connection! (#0) with host foostore-foo-fooapistore.s3.amazonaws.com
* Connected to foostore-foo-fooapistore.s3.amazonaws.com (52.95.190.156) port 443 (#0)
> GET /data/foo/processed/fooprovider/AB1/AB1-img-B.tif HTTP/1.1
Host: foostore-foo-fooapistore.s3.amazonaws.com
Accept: */*
Range: bytes=0-16383
x-amz-date: 20240430T171810Z
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Authorization: AWS4-HMAC-SHA256 Credential=AKIA3SRZT7HAMJHOOAN3/20240430/ca-central-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=3c30b652d70b1e0140830776e02d214f3db26350f8cf78eceab4db1282a1b6f1

* Mark bundle as not supporting multiuse
< HTTP/1.1 403 Forbidden
< x-amz-request-id: YVXYX5TYRZW58QXK
< x-amz-id-2: lqdGxbjJOjRQj5iUH+NqhxrkbBL3a50IGjCkrjdzWemSMLGJa1umVbcnbZtVwOWvaeDlQqYsvXU=
< Content-Type: application/xml
< Transfer-Encoding: chunked
< Date: Tue, 30 Apr 2024 17:18:10 GMT
< Server: AmazonS3
< 
* Connection #0 to host foostore-foo-fooapistore.s3.amazonaws.com left intact
Warning 1: HTTP response code on https://foostore-foo-fooapistore.s3.amazonaws.com/data/foo/processed/fooprovider/AB1/AB1-img-B.tif: 403
S3: GetFileSize(https://foostore-foo-fooapistore.s3.amazonaws.com/data/foo/processed/fooprovider/AB1/AB1-img-B.tif)=0  response_code=403
* Couldn't find host foostore-foo-fooapistore.s3.amazonaws.com in the .netrc file; using defaults
* Found bundle for host foostore-foo-fooapistore.s3.amazonaws.com: 0x5a41e1de9d40 [serially]
* Can not multiplex, even if we wanted to!
* Re-using existing connection! (#0) with host foostore-foo-fooapistore.s3.amazonaws.com
* Connected to foostore-foo-fooapistore.s3.amazonaws.com (52.95.190.156) port 443 (#0)
> GET /?delimiter=%2F&max-keys=100&prefix=data%2Ffoo%2Fprocessed%2Ffooprovider%2FAB1%2FAB1-img-B.tif%2F HTTP/1.1
Host: foostore-foo-fooapistore.s3.amazonaws.com
Accept: */*
x-amz-date: 20240430T171810Z
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Authorization: AWS4-HMAC-SHA256 Credential=AKIA3SRZT7HAMJHOOAN3/20240430/ca-central-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=aad4a7dd0c9aaf19c8ff4e2c5ba583e5824e218cd7328cc00e1295a205897adc

* Mark bundle as not supporting multiuse
< HTTP/1.1 403 Forbidden
< x-amz-bucket-region: ca-central-1
< x-amz-request-id: YVXWZ6030XS29YG1
< x-amz-id-2: 3dJx6RU8glvGuPCe7uLxJ1zv1T1XrYHpH6rxZNiu2f9GJJKgJOPY9Ep2Dj1fbBFAgLc6zWdVzpo=
< Content-Type: application/xml
< Transfer-Encoding: chunked
< Date: Tue, 30 Apr 2024 17:18:10 GMT
< Server: AmazonS3
< 
* Connection #0 to host foostore-foo-fooapistore.s3.amazonaws.com left intact
S3: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>YVXWZ6030XS29YG1</RequestId><HostId>3dJx6RU8glvGuPCe7uLxJ1zv1T1XrYHpH6rxZNiu2f9GJJKgJOPY9Ep2Dj1fbBFAgLc6zWdVzpo=</HostId></Error>
ERROR 14: Access Denied
gdalinfo failed - unable to open '/vsis3/foostore-foo-fooapistore/data/foo/processed/fooprovider/AB1/AB1-img-B.tif'.

@remtav
Copy link
Author

remtav commented Apr 30, 2024

Insights

Call stack from AWS-CLI

input command: aws s3 ls --debug s3://foostore-foo-fooapistore/

Some insight within this larger debug log:

The authorization header is malformed; the region \'us-east-1\' is wrong; expecting \'ca-central-1\'</Message><Region>ca-central-1
S3 client configured for region us-east-1 but the bucket foostore-foo-fooapistore is in region ca-central-1; Please configure the proper region to avoid multiple unnecessary redirects and signing attempts.
Calling endpoint provider with parameters: {'Bucket': 'foostore-foo-fooapistore', 'Region': 'ca-central-1', 'UseFIPS': False, 'UseDualStack': False, 'ForcePathStyle': False, 'Accelerate': False, 'UseGlobalEndpoint': True, 'Prefix': '', 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True}

Call stack from GDAL

input command: gdalinfo /vsis3/foostore-foo-fooapistore/.../AB1-img-B.tif

Small insight:

Warning 1: HTTP response code on https://foostore-foo-fooapistore.s3.amazonaws.com/data/foo/processed/fooprovider/AB1/AB1-img-B.tif: 403

The why - AWS account restrictions

What we understand is that in both cases, we seem to be trying to talk to foostore-foo-fooapistore.s3.amazonaws.com first, but AWS CLI finds a way out with foostore-foo-fooapistore.s3.ca-central-1.amazonaws.com and GDAL doesn't.

The issue is that our aws account has a policy that restricts the use of the federation token to the internal IPs (within our office's network).

Normally, this shouldn't cause a problem because there's a DNS-level redirection for the S3 endpoint so that the request doesn't leave the AWS network, but GDAL defaults to its own S3 URL and ends on a "Access denied".

It was working in the office, HPC, and VPN because all those IPs are whitelisted in the foostore-foo-fooapistore user role, whereas with AWS EC2 instances, since we never have the same outgoing (thus incoming to the bucket) IP, our IT team created a special route that is allowed. However, to use it, one must use one of the URLs:

  • s3.ca-central-1.amazonaws.com
  • *.s3.ca-central-1.amazonaws.com
  • *.s3-accesspoint.ca-central-1.amazonaws.com
  • *.s3-control.ca-central-1.amazonaws.com

The tests were performed using the main AWS keys of the account, rather than federation tokens. Therefore, this issue is not specificly linked to using federation tokens. Rather, it concerns the specific policy enforced by our IT team to restric access to AWS buckets from outside our network.

To provide a broader solution without the use of AWS_S3_ENDPOINT, GDAL could update the URI based on AWS_REGION like AWS-CLI does. @rouault

@carlstlaurent please review these details and clarify if I've got anything wrong or confused.

@rouault
Copy link
Member

rouault commented Apr 30, 2024

GDAL could update the URI based on AWS_REGION like AWS-CLI does

hum, I'm wondering if the issue hasn't been fixed recently by c3faafa . You should try GDAL 3.9.0 beta1 or beta2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@rouault @remtav