generate_presigned_url for s3 endpoint results in access denied

[dahlinPL]

Describe the bug

When using example from https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-presigned-urls.html to generate presigned url for file download from S3, it always ends in AccessDenied.

Expected Behavior

To generate working presigned URL.

Current Behavior

Trying to reach file using generated presigned URL, results in AccessDenied error returned by AWS.

Reproduction Steps

Out of the box example code mentioned about will fail due to missing AWS region.

Not adding region ends in AuthorizationQueryParametersError with message Error parsing the X-Amz-Credential parameter; the region 'us-east-1' is wrong; expecting '{my_region_name}'

Below sample code with region added:

import  logging
import boto3
from botocore.exceptions import ClientError


def create_presigned_url(bucket_name, object_name, expiration=3600):
    s3_client = boto3.client("s3", 
            config=boto3.session.Config(
                signature_version="s3v4",
                region_name=AWS_REGION_NAME,
            ),
    )
    try:
        response = s3_client.generate_presigned_url("get_object",
                                                    Params={"Bucket": bucket_name,
                                                            "Key": object_name},
                                                    ExpiresIn=expiration)
    except ClientError as e:
        logging.error(e)
        return None
    return response

URL generated with this code is not working. My observation is that constructed URL is different that URL constructed while generating presigned URL from AWS console (using this link I'm able to download file).

URL generated using AWS console:
https://{bucket_name}.s3.{region}.amazonaws.com/{object_key}

URL generated by boto3:
https://{bucket_name}.s3.amazonaws.com/{object_key}
As you can see region is missing. Trying to reach object ends in AccessDenied.

adding endpoint_url as https://{bucket_name}.s3.{region}.amazonaws.com into client config as results in URL:
https://{bucket_name}.s3.{region}.amazonaws.com/{bucket_name}/{object_key} and AccessDenied

adding endpoint_url as https://{bucket_name}.s3.amazonaws.com into client config as results in URL:
https://s3.{region}.amazonaws.com/{bucket_name}/{object_key} and AccessDenied

combining above options with s3.addressing_style and/or s3.inject_host_prefix results in AccessDenied

adding 'ResponseContentDisposition': 'inline' to generate_presigned_url Params, as this parameter exists in working URL generated from AWS console change nothing.

Tried with eu-central-1 and ap-northeast-2 regions. Results are the same.

Possible Solution

It seems that without adding endpoint_url, endpoint provider using wrong input to generate endpoint:

Calling endpoint provider with parameters: {'Bucket': 'bucket_name', 'Region': 'aws-global', 'UseFIPS': False, 'UseDualStack': False, 'ForcePathStyle': False, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True}
Endpoint provider result: https://bucket_name.s3.amazonaws.com
Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4"
Selected auth type "v4" as "s3v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True}
Calculating signature using v4 auth.
(...)
CanonicalRequest:
GET
/bucket_name/pizza.jpg

As you can see, for some reason Region is set to aws-global and it results in URL https://bucket_name.s3.amazonaws.com. Later it results in using us-east-1 region for signing instead of required local region.

When I add endpoint_url as https://{bucket_name}.s3.{region}.amazonaws.com/ it looks like region is properly passed into args:

Calling endpoint provider with parameters: {'Bucket': 'bucket_name', 'Region': 'eu-central-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://bucket_name.s3.region.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': False, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True}
Endpoint provider result: https:/bucket_name.s3.region.amazonaws.com/bucket_name
Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "s3v4"
Selected auth type "v4" as "s3v4" with signing context params: {'region': 'region', 'signing_name': 's3', 'disableDoubleEncoding': True}
(...)
Calculating signature using v4 auth.
CanonicalRequest:
GET
/bucket_name/pizza.jpg

however as you can see, constructed URL is bad, since it include bucket name as prefix and postfix in URL.

Additional Information/Context

No response

SDK version used

1.26.133

Environment details (OS name and version, etc.)

Django 3.2.18 on docker with python:3.10-slim-bullseye on AWS kubernetes

[tim-finnigan]

Hi @dahlinPL thanks for reaching out. I tried to reproduce the issue but was not able to. I could generate a working presigned URL for an object in an eu-central-1 bucket using the example code.

A region is excluded from the example as you mentioned but there are a few ways you could configure a region: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html.

Can you check that you have the necessary IAM policies in place to access the object? Here is more info on this topic from the S3 User Guide: https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html

[dahlinPL]

Dear @tim-finnigan ,

thanks for quick answers. I double checked my deployment details in kubernetes and reason was wrong IAM role attached. I was fooled by no error returned, and assumed that if I received presigned URL, used credentials (which was instance role in that case) has proper permissions.

Thanks for your help, and sorry for taking your time!
Have a nice day,
Marcin

[gatto-c]

Hi @dahlinPL ,

I am facing the same issue you described. Could you tell me what IAM role you needed to add and what permissions the role had that fixed the issue?

Thanks,
Chris

[dahlinPL]

Hi @dahlinPL ,

I am facing the same issue you described. Could you tell me what IAM role you needed to add and what permissions the role had that fixed the issue?

Thanks, Chris

Hi @gatto-c ,

I've attached custom IAM role with custom S3 bucket permissions. Policy basically should looks like this (in my case there are also other permissions but I guess they are not needed in that case):

{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Sid": "VisualEditor0",
        "Effect": "Allow",
        "Action": [
            "s3:PutObject"        ],
        "Resource": "arn:aws:s3:::your_bucket/*"
      }
    ]
}

and mine IAM role has also trust relationship set for security reasons:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::account_id:oidc-provider/oidc.region.amazonaws.com/id/someid"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.region.amazonaws.com/id/someid:aud": "sts.amazonaws.com",
                    "oidc.eks.region.amazonaws.com/id/someid:sub": "system:serviceaccount:somerolename"
                }
            }
        }
    ]
}

[gatto-c]

Thanks @dahlinPL This worked, much appreciated!

[SatinderSidhu]

You also have to provide GetObject access to IAM tole

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject"
],
"Resource": "arn:aws:s3:::/"
}
]
}