generate_presigned_url for put_bucket_cors fails for non-AWS region

[philipnbbc]

Describe the bug

Using generate_presigned_url for the put_bucket_cors method fails for a non-AWS region.

Expected Behavior

It should succeed and return the presigned URL string.

Current Behavior

It produces these DEBUG log messages and finally the error stack trace:

2023-05-17 17:12:16,749 botocore.hooks [DEBUG] Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2023-05-17 17:12:16,750 botocore.hooks [DEBUG] Changing event name from before-call.apigateway to before-call.api-gateway
2023-05-17 17:12:16,751 botocore.hooks [DEBUG] Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2023-05-17 17:12:16,752 botocore.hooks [DEBUG] Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2023-05-17 17:12:16,752 botocore.hooks [DEBUG] Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2023-05-17 17:12:16,752 botocore.hooks [DEBUG] Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2023-05-17 17:12:16,752 botocore.hooks [DEBUG] Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2023-05-17 17:12:16,754 botocore.hooks [DEBUG] Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2023-05-17 17:12:16,754 botocore.hooks [DEBUG] Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2023-05-17 17:12:16,754 botocore.hooks [DEBUG] Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2023-05-17 17:12:16,754 botocore.hooks [DEBUG] Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2023-05-17 17:12:16,756 botocore.utils [DEBUG] IMDS ENDPOINT: http://169.254.169.254/
2023-05-17 17:12:16,757 botocore.credentials [DEBUG] Looking for credentials via: env
2023-05-17 17:12:16,757 botocore.credentials [INFO] Found credentials in environment variables.
2023-05-17 17:12:16,758 botocore.loaders [DEBUG] Loading JSON file: /test/venv/lib/python3.10/site-packages/botocore/data/endpoints.json
2023-05-17 17:12:16,764 botocore.loaders [DEBUG] Loading JSON file: /test/venv/lib/python3.10/site-packages/botocore/data/sdk-default-configuration.json
2023-05-17 17:12:16,764 botocore.hooks [DEBUG] Event choose-service-name: calling handler <function handle_service_name_alias at 0x7fd022730040>
2023-05-17 17:12:16,769 botocore.loaders [DEBUG] Loading JSON file: /test/venv/lib/python3.10/site-packages/botocore/data/s3/2006-03-01/service-2.json
2023-05-17 17:12:16,781 botocore.loaders [DEBUG] Loading JSON file: /test/venv/lib/python3.10/site-packages/botocore/data/s3/2006-03-01/endpoint-rule-set-1.json.gz
2023-05-17 17:12:16,784 botocore.loaders [DEBUG] Loading JSON file: /test/venv/lib/python3.10/site-packages/botocore/data/partitions.json
2023-05-17 17:12:16,785 botocore.hooks [DEBUG] Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7fd02267c670>
2023-05-17 17:12:16,785 botocore.hooks [DEBUG] Event creating-client-class.s3: calling handler <function lazy_call.<locals>._handler at 0x7fd023737eb0>
2023-05-17 17:12:16,795 botocore.hooks [DEBUG] Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7fd02267c430>
2023-05-17 17:12:16,796 botocore.regions [DEBUG] Creating a regex based endpoint for s3, other
2023-05-17 17:12:16,796 botocore.endpoint [DEBUG] Setting s3 timeout as (60, 60)
2023-05-17 17:12:16,798 botocore.loaders [DEBUG] Loading JSON file: /test/venv/lib/python3.10/site-packages/botocore/data/_retry.json
2023-05-17 17:12:16,798 botocore.client [DEBUG] Registering retry handlers for service: s3
2023-05-17 17:12:16,798 botocore.utils [DEBUG] Registering S3 region redirector handler
2023-05-17 17:12:16,798 botocore.hooks [DEBUG] Event before-parameter-build.s3.PutBucketCors: calling handler <function validate_bucket_name at 0x7fd0227316c0>
2023-05-17 17:12:16,798 botocore.hooks [DEBUG] Event before-parameter-build.s3.PutBucketCors: calling handler <function remove_bucket_from_url_paths_from_model at 0x7fd022733490>
2023-05-17 17:12:16,798 botocore.hooks [DEBUG] Event before-parameter-build.s3.PutBucketCors: calling handler <bound method S3RegionRedirectorv2.annotate_request_context of <botocore.utils.S3RegionRedirectorv2 object at 0x7fd021c60c40>>
2023-05-17 17:12:16,798 botocore.hooks [DEBUG] Event before-parameter-build.s3.PutBucketCors: calling handler <function generate_idempotent_uuid at 0x7fd022731510>
2023-05-17 17:12:16,799 botocore.hooks [DEBUG] Event before-endpoint-resolution.s3: calling handler <function customize_endpoint_resolver_builtins at 0x7fd022733640>
2023-05-17 17:12:16,799 botocore.hooks [DEBUG] Event before-endpoint-resolution.s3: calling handler <bound method S3RegionRedirectorv2.redirect_from_cache of <botocore.utils.S3RegionRedirectorv2 object at 0x7fd021c60c40>>
2023-05-17 17:12:16,799 botocore.regions [DEBUG] Calling endpoint provider with parameters: {'Bucket': 'test', 'Region': 'other', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://example.com/', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': False, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True}
2023-05-17 17:12:16,799 botocore.regions [DEBUG] Endpoint provider result: https://example.com/test
2023-05-17 17:12:16,799 botocore.regions [DEBUG] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "None"
2023-05-17 17:12:16,800 botocore.regions [DEBUG] Selected auth type "v4" as "v4" with signing context params: {'region': 'other', 'signing_name': 's3', 'disableDoubleEncoding': True}
2023-05-17 17:12:16,800 botocore.hooks [DEBUG] Event choose-signer.s3.PutBucketCors: calling handler <function set_operation_specific_signer at 0x7fd0227313f0>
2023-05-17 17:12:16,800 botocore.hooks [DEBUG] Event before-sign.s3.PutBucketCors: calling handler <function remove_arn_from_signing_path at 0x7fd0227335b0>
Traceback (most recent call last):
  File "/test/test_case.py", line 11, in <module>
    presigned_url = client.generate_presigned_url(
  File "/test/venv/lib/python3.10/site-packages/botocore/signers.py", line 686, in generate_presigned_url
    return request_signer.generate_presigned_url(
  File "/test/venv/lib/python3.10/site-packages/botocore/signers.py", line 327, in generate_presigned_url
    self.sign(
  File "/test/venv/lib/python3.10/site-packages/botocore/signers.py", line 189, in sign
    auth.add_auth(request)
  File "/test/venv/lib/python3.10/site-packages/botocore/auth.py", line 423, in add_auth
    self._modify_request_before_signing(request)
  File "/test/venv/lib/python3.10/site-packages/botocore/auth.py", line 587, in _modify_request_before_signing
    query_dict.update(_get_body_as_dict(request))
  File "/test/venv/lib/python3.10/site-packages/botocore/auth.py", line 96, in _get_body_as_dict
    data = json.loads(data.decode('utf-8'))
  File "/home/philipn/.pyenv/versions/3.10/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/home/philipn/.pyenv/versions/3.10/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/home/philipn/.pyenv/versions/3.10/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Reproduction Steps

It can be reproduced using this code below. Change the region_name to eu-west-1 for example and it succeeds.

import boto3

boto3.set_stream_logger("")

client = boto3.client(
    "s3",
    endpoint_url="https://example.com/",
    region_name="other"  # Change to "eu-west-1" and it succeeds
)

presigned_url = client.generate_presigned_url(
    "put_bucket_cors",
    Params={
        "Bucket": "test",
        "CORSConfiguration": {
            "CORSRules": [
                {
                    "AllowedMethods": ["GET"],
                    "AllowedOrigins": ["*"]
                }
            ]
        }
    },
    ExpiresIn=3600
)
print(presigned_url)

Possible Solution

The workaround prior to botocore version 1.28.0 was to add config=botocore.config.Config(signature_version="s3") to the session.client() call. Unfortunately something changed from botocore version 1.27.96 to 1.28.0 and the S3 service now returns an "AccessDenied" / 403 error.

Additional Information/Context

The target of this presigned URL is a Ceph / Radosgw s3 deployment, hence the change in endpoint URL and region.

SDK version used

botocore 1.29.135

Environment details (OS name and version, etc.)

Code was tested in a Python 3.10.9 virtualenv

[tim-finnigan]

Hi @philipnbbc thanks for reaching out. Here is the CHANGELOG entry for botocore version 1.28.0. As noted there, updates to the endpoint rule system were made:

feature:Endpoints: Implemented new endpoint ruleset system to dynamically derive endpoints and settings for services

Here is the rule set for S3 that was added. This may be some sort of third-party compatibility issue. Could you explain more what you mean by using a non-AWS region?

[philipnbbc]

Hi @tim-finnigan. It had another closer look and have created #2962 after finding that a request using the presigned URL fails for AWS S3 (e.g. "eu-west-1").

By non-AWS region I mean our Openstack deployment which has a Ceph / Rados gateway S3 storage backend. The region name it uses is "default".

If I set the region name to "default" then it returns the error reported in this issue. If I set it to "eu-west-1" (with or without the fix from #2962) the presigned URL is generated but our deployment reports "AccessDenied" - which probably makes sense as the region is incorrect.

The secondary issue is that it used to work for botocore version <1.28.0 using a V2 signature (config=botocore.config.Config(signature_version="s3")). It now (>= 1.28.0) returns an "AccessDenied".

[tim-finnigan]

Hi @philipnbbc thanks for following up. Regarding v2 signatures, it notes here in the S3 documentation:

Amazon S3 supports only AWS Signature Version 4 in most AWS Regions. In some of the older AWS Regions, Amazon S3 supports both Signature Version 4 and Signature Version 2. However, Signature Version 2 is being turned off (deprecated). For more information about the end of support for Signature Version 2, see AWS Signature Version 2 Turned Off (Deprecated) for Amazon S3.

You can refer to this page for regions mapping to valid S3 endpoints. I'm not familiar with OpenStack but for using boto3 you should configure a valid region for your requests. The AccessDenied error could also be a permissions issue, for example not having the right role attached. But if all else is the same between <1.28.0 and >= 1.28.0 then maybe it is some incompatibility issue with OpenStack and the new endpoint ruleset introduced in 1.28.0, in which case OpenStack would need to address that.

[github-actions]

Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.

[philipnbbc]

I'm aware that the V2 signature is deprecated, which is why I wanted to instead see in this issue whether the V4 signature in pre-signed URLs can be made to work for put_bucket_cors as it already does for create_bucket for example. It looks like I need to investigate it on the Ceph side to see what works there (and compare it with what works on AWS) and write the code to sign the request.

[tim-finnigan]

Thanks for following up. I'm not familiar with Ceph but yes this seems to be some kind of third-party incompatibility regarding how regions are handled. If you have any updates on what you have found please let us know. I'm still looking into the related issue you opened (#2962).

[philipnbbc]

Thanks for your help. I'll probably not have the time to get an update on findings within 5 days, but will re-open an issue once I do.