generate_presigned_url for put_bucket_cors results in invalid signature

[philipnbbc]

Describe the bug

Generating a presigned URL for the put_bucket_cors method results in a signature that AWS reports as "SignatureDoesNotMatch"

Expected Behavior

The signature should match and the method succeed.

Current Behavior

The request to AWS fails with the response text shown below (some text has been replaced with REDACTED).

Note the end of StringToSign which has "/test-bucket/?cors". The error text shown further below shows that botocore is using "/test-bucket?cors?cors" instead!

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>REDACTED</AWSAccessKeyId><StringToSign>PUT
6ExFvI55ofqo42RutDlTDg==

1685543937
x-amz-security-token:REDACTED
/test-bucket/?cors</StringToSign><SignatureProvided>REDACTED</SignatureProvided><StringToSignBytes>REDACTED</StringToSignBytes><RequestId>REDACTED</RequestId><HostId>REDACTED</HostId></Error>

The full output (with some text REDACTED) for the test code is:

2023-05-31 14:38:57,176 botocore.hooks [DEBUG] Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2023-05-31 14:38:57,177 botocore.hooks [DEBUG] Changing event name from before-call.apigateway to before-call.api-gateway
2023-05-31 14:38:57,177 botocore.hooks [DEBUG] Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2023-05-31 14:38:57,178 botocore.hooks [DEBUG] Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2023-05-31 14:38:57,178 botocore.hooks [DEBUG] Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2023-05-31 14:38:57,179 botocore.hooks [DEBUG] Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2023-05-31 14:38:57,179 botocore.hooks [DEBUG] Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2023-05-31 14:38:57,180 botocore.hooks [DEBUG] Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2023-05-31 14:38:57,180 botocore.hooks [DEBUG] Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2023-05-31 14:38:57,180 botocore.hooks [DEBUG] Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2023-05-31 14:38:57,180 botocore.hooks [DEBUG] Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2023-05-31 14:38:57,182 botocore.utils [DEBUG] IMDS ENDPOINT: http://169.254.169.254/
2023-05-31 14:38:57,183 botocore.credentials [DEBUG] Looking for credentials via: env
2023-05-31 14:38:57,183 botocore.credentials [DEBUG] Looking for credentials via: assume-role
2023-05-31 14:38:57,183 botocore.credentials [DEBUG] Looking for credentials via: assume-role-with-web-identity
2023-05-31 14:38:57,183 botocore.credentials [DEBUG] Looking for credentials via: sso
2023-05-31 14:38:57,183 botocore.credentials [DEBUG] Looking for credentials via: shared-credentials-file
2023-05-31 14:38:57,183 botocore.credentials [DEBUG] Looking for credentials via: custom-process
2023-05-31 14:38:57,766 botocore.loaders [DEBUG] Loading JSON file: /home/philipn/cors_issue/venv/lib/python3.10/site-packages/botocore/data/endpoints.json
2023-05-31 14:38:57,785 botocore.loaders [DEBUG] Loading JSON file: /home/philipn/cors_issue/venv/lib/python3.10/site-packages/botocore/data/sdk-default-configuration.json
2023-05-31 14:38:57,786 botocore.hooks [DEBUG] Event choose-service-name: calling handler <function handle_service_name_alias at 0x7fb0a77fbb50>
2023-05-31 14:38:57,790 botocore.loaders [DEBUG] Loading JSON file: /home/philipn/cors_issue/venv/lib/python3.10/site-packages/botocore/data/s3/2006-03-01/service-2.json
2023-05-31 14:38:57,801 botocore.loaders [DEBUG] Loading JSON file: /home/philipn/cors_issue/venv/lib/python3.10/site-packages/botocore/data/s3/2006-03-01/endpoint-rule-set-1.json.gz
2023-05-31 14:38:57,805 botocore.loaders [DEBUG] Loading JSON file: /home/philipn/cors_issue/venv/lib/python3.10/site-packages/botocore/data/partitions.json
2023-05-31 14:38:57,806 botocore.hooks [DEBUG] Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7fb0a7764160>
2023-05-31 14:38:57,806 botocore.hooks [DEBUG] Event creating-client-class.s3: calling handler <function lazy_call.<locals>._handler at 0x7fb0a747e170>
2023-05-31 14:38:57,816 botocore.hooks [DEBUG] Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7fb0a775beb0>
2023-05-31 14:38:57,817 botocore.endpoint [DEBUG] Setting s3 timeout as (60, 60)
2023-05-31 14:38:57,818 botocore.loaders [DEBUG] Loading JSON file: /home/philipn/cors_issue/venv/lib/python3.10/site-packages/botocore/data/_retry.json
2023-05-31 14:38:57,818 botocore.client [DEBUG] Registering retry handlers for service: s3
2023-05-31 14:38:57,818 botocore.utils [DEBUG] Registering S3 region redirector handler
2023-05-31 14:38:57,819 botocore.hooks [DEBUG] Event before-parameter-build.s3.PutBucketCors: calling handler <function validate_bucket_name at 0x7fb0a7821240>
2023-05-31 14:38:57,819 botocore.hooks [DEBUG] Event before-parameter-build.s3.PutBucketCors: calling handler <function remove_bucket_from_url_paths_from_model at 0x7fb0a7823010>
2023-05-31 14:38:57,819 botocore.hooks [DEBUG] Event before-parameter-build.s3.PutBucketCors: calling handler <bound method S3RegionRedirectorv2.annotate_request_context of <botocore.utils.S3RegionRedirectorv2 object at 0x7fb0a695a6e0>>
2023-05-31 14:38:57,819 botocore.hooks [DEBUG] Event before-parameter-build.s3.PutBucketCors: calling handler <function generate_idempotent_uuid at 0x7fb0a7821090>
2023-05-31 14:38:57,819 botocore.hooks [DEBUG] Event before-endpoint-resolution.s3: calling handler <function customize_endpoint_resolver_builtins at 0x7fb0a78231c0>
2023-05-31 14:38:57,819 botocore.hooks [DEBUG] Event before-endpoint-resolution.s3: calling handler <bound method S3RegionRedirectorv2.redirect_from_cache of <botocore.utils.S3RegionRedirectorv2 object at 0x7fb0a695a6e0>>
2023-05-31 14:38:57,819 botocore.regions [DEBUG] Calling endpoint provider with parameters: {'Bucket': 'test-bucket', 'Region': 'aws-global', 'UseFIPS': False, 'UseDualStack': False, 'ForcePathStyle': False, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True}
2023-05-31 14:38:57,819 botocore.regions [DEBUG] Endpoint provider result: https://test-bucket.s3.amazonaws.com
2023-05-31 14:38:57,819 botocore.regions [DEBUG] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "None"
2023-05-31 14:38:57,819 botocore.regions [DEBUG] Selected auth type "v4" as "v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True}
2023-05-31 14:38:57,820 botocore.hooks [DEBUG] Event choose-signer.s3.PutBucketCors: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7fb0a749a110>>
2023-05-31 14:38:57,820 botocore.hooks [DEBUG] Event before-sign.s3.PutBucketCors: calling handler <function remove_arn_from_signing_path at 0x7fb0a7823130>
2023-05-31 14:38:57,820 botocore.auth [DEBUG] Calculating signature using hmacv1 auth.
2023-05-31 14:38:57,820 botocore.auth [DEBUG] HTTP request method: PUT
2023-05-31 14:38:57,820 botocore.auth [DEBUG] StringToSign:
PUT
6ExFvI55ofqo42RutDlTDg==

1685543937
x-amz-security-token:REDACTED
/test-bucket?cors?cors
2023-05-31 14:38:57,822 urllib3.connectionpool [DEBUG] Starting new HTTPS connection (1): test-bucket.s3.amazonaws.com:443
2023-05-31 14:38:58,239 urllib3.connectionpool [DEBUG] https://test-bucket.s3.amazonaws.com:443 "PUT /?cors&AWSAccessKeyId=REDACTED&Signature=REDACTED&x-amz-security-token=REDACTED&Expires=1685543937 HTTP/1.1" 307 None
2023-05-31 14:38:58,245 urllib3.connectionpool [DEBUG] Starting new HTTPS connection (1): test-bucket.s3-eu-west-1.amazonaws.com:443
2023-05-31 14:38:58,471 urllib3.connectionpool [DEBUG] https://test-bucket.s3-eu-west-1.amazonaws.com:443 "PUT /?cors&AWSAccessKeyId=REDACTED&Signature=REDACTED&x-amz-security-token=REDACTED&Expires=1685543937 HTTP/1.1" 403 None
2023-05-31 14:38:58,494 charset_normalizer [DEBUG] Encoding detection: utf_8 is most likely the one.
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>REDACTED</AWSAccessKeyId><StringToSign>PUT
6ExFvI55ofqo42RutDlTDg==

1685543937
x-amz-security-token:REDACTED
/test-bucket/?cors</StringToSign><SignatureProvided>REDACTED</SignatureProvided><StringToSignBytes>REDACTED</StringToSignBytes><RequestId>REDACTED</RequestId><HostId>REDACTED</HostId></Error>

Reproduction Steps

The code copied below can be used to reproduce the error and includes a hack to "fix" it The code generates a presigned URL and then makes a request. The MODIFY_FUNC variable toggles a "fix" in the botocore.signers.generate_presigned_url() function to get it to work.

The code assumes that there exists a bucket named "test-bucket" in the "eu-west-1" region. The name and region can be changed as required.

Set MODIFY_FUNC = False to see the request failing Set it to MODIFY_FUNC = True to see the test passing.

import boto3
import requests
import hashlib
import base64
import sys

BUCKET_NAME = "test-bucket"
PROFILE_NAME = None
REGION_NAME = "eu-west-1"
ENDPOINT_URL = None
DEBUG_OUTPUT = True

MODIFY_FUNC = False

# This modifies the code from botocore.signers.generate_presigned_url() to
# change the "auth_path" in the request dict from "/{BUCKET_NAME}?cors" to "/{BUCKET_NAME}/"
def modify_generate_presigned_url(client):
    from botocore.exceptions import UnknownClientMethodError
    from botocore.utils import ArnParser

    def _should_use_global_endpoint(client):
        if client.meta.partition != 'aws':
            return False
        s3_config = client.meta.config.s3
        if s3_config:
            if s3_config.get('use_dualstack_endpoint', False):
                return False
            if (
                s3_config.get('us_east_1_regional_endpoint') == 'regional'
                and client.meta.config.region_name == 'us-east-1'
            ):
                return False
        return True

    def mod_generate_presigned_url(ClientMethod, Params=None, ExpiresIn=3600, HttpMethod=None):
        client_method = ClientMethod
        params = Params
        if params is None:
            params = {}
        expires_in = ExpiresIn
        http_method = HttpMethod
        context = {
            'is_presign_request': True,
            'use_global_endpoint': _should_use_global_endpoint(client),
        }

        request_signer = client._request_signer

        try:
            operation_name = client._PY_TO_OP_NAME[client_method]
        except KeyError:
            raise UnknownClientMethodError(method_name=client_method)

        operation_model = client.meta.service_model.operation_model(operation_name)
        params = client._emit_api_params(
            api_params=params,
            operation_model=operation_model,
            context=context,
        )
        bucket_is_arn = ArnParser.is_arn(params.get('Bucket', ''))
        endpoint_url, additional_headers = client._resolve_endpoint_ruleset(
            operation_model,
            params,
            context,
            ignore_signing_region=(not bucket_is_arn),
        )

        request_dict = client._convert_to_request_dict(
            api_params=params,
            operation_model=operation_model,
            endpoint_url=endpoint_url,
            context=context,
            headers=additional_headers,
            set_user_agent_header=False,
        )

        # MODIFICATION
        print(f"Changing auth_path in request_dict from '{request_dict['auth_path']}' to '{'/' + BUCKET_NAME + '/'}'")
        request_dict["auth_path"] = "/" + BUCKET_NAME + "/"

        # Switch out the http method if user specified it.
        if http_method is not None:
            request_dict['method'] = http_method

        # Generate the presigned url.
        return request_signer.generate_presigned_url(
            request_dict=request_dict,
            expires_in=expires_in,
            operation_name=operation_name,
        )

    client.generate_presigned_url = mod_generate_presigned_url


def test():
    if DEBUG_OUTPUT:
        boto3.set_stream_logger('')

    session = boto3.Session(profile_name=PROFILE_NAME)
    client = session.client(
        "s3",
        endpoint_url=ENDPOINT_URL,
        region_name=REGION_NAME
    )
    if MODIFY_FUNC:
        modify_generate_presigned_url(client)

    cors_xml = """\
<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
   <CORSRule>
      <AllowedMethod>GET</AllowedMethod>
      <AllowedOrigin>*</AllowedOrigin>
   </CORSRule>
</CORSConfiguration>
"""
    cors_data = cors_xml.encode('utf-8')
    cors_data_md5 = base64.b64encode(hashlib.md5(cors_data).digest()).decode('utf-8')

    cors_presigned_url = client.generate_presigned_url(
        'put_bucket_cors',
        Params={
            'Bucket': BUCKET_NAME,
            'CORSConfiguration': {
                'CORSRules': [
                    {
                        'AllowedMethods': ['GET'],
                        'AllowedOrigins': ['*']
                    }
                ]
            },
            'ContentMD5': cors_data_md5
        },
        ExpiresIn=3600
    )

    cors_headers = {
        "Content-Length": str(len(cors_data)),
        "Content-MD5": cors_data_md5
    }
    resp = requests.put(
        cors_presigned_url,
        data=cors_data,
        headers=cors_headers
    )
    if resp.status_code != 200:
        print(resp.text, file=sys.stderr)


test()

Possible Solution

See the test code. Modifying auth_path property in the request_dict in the botocore.signers.generate_presigned_url() function results a correct StringToSign and it succeeds.

Note however that this is meant as a quick hack and it is unclear what the proper fix should be.

Additional Information/Context

No response

SDK version used

botocore 1.29.143, boto3 1.26.143

Environment details (OS name and version, etc.)

Python 3.10 virtualenv

[tim-finnigan]

Hi @philipnbbc thanks for reaching out. Could you share more details on your use case? I'm not sure that generating a presigned url is intended to support the put_bucket_cors command. As mentioned in the boto3 documentation on presigned urls, the main purpose of presigned URLs is to grant a user temporary access to an S3 object. I tried to find similar issues related to the use case you described but was not able to.

[philipnbbc]

Our use case is that we have a service that indexes content in S3 and provides information that clients can use to read and write content directly to S3 using pre-signed URLs. The pre-signed URLs allow clients to not have AWS credentials and it avoids the service having to handle the object and bucket requests (with retries etc.), becoming a bottleneck in the S3 interactions.

[tim-finnigan]

Thanks for following up, I brought this issue up for discussion with the team and think it requires more investigation. Marked this for further review.

Quick update: after further investigation by the team, this issue appears to be due to recent changes involving endpoint rulesets. A fix will need to be implemented for this issue, for now we will leave the issue open and continue tracking the bug here.

[tim-finnigan]

It looks like this can now be closed since #2971 was merged.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.