Support Requester Pays S3 Buckets

[senorkrabs]

Is your idea related to a problem? Please describe.
It appears that the s3 methods in Data Wrangler don't support retrieving data in a bucket with Requester Pays enabled.

I verified the below code works non a bucket with Requester Pays disabled, but produces an AccessDenied error when it is enabled on the same bucket.

df = wr.s3.read_parquet('{}/{}'.format(path, "markets/")
ClientError: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

Describe the solution you'd like
Add an argument to s3 methods to indicate/acknowledge requester pays in the S3 API request.

P.S. Don't attach files. Please, prefer add code snippets directly in the message body.

[igorborgest]

Hi @senorkrabs, thanks for reaching out.

I've just added support for requester pays on most functions/features we have (commit above). You just need to pass s3_additional_kwargs={"RequestPayer": "requester"} as argument.

e.g.

wr.s3.to_parquet(
    df=df,
    path=f"s3://{BUCKET}/test/",
    mode="overwrite",
    dataset=True,
    s3_additional_kwargs={"RequestPayer": "requester"}
)

df = wr.s3.read_parquet(
    path=f"s3://{BUCKET}/test/",
    dataset=True,
    s3_additional_kwargs={"RequestPayer": "requester"}
)

P.S. If you face some specific function still w/o resquester pays support, please open a new issue here o GitHub and we will address that.

P.P.S. It will be release on version 2.4.0 next week.

[igorborgest]

Released on version 2.4.0.

[senorkrabs]

Nice! Thanks for adding this! I recommend this package to everyone!

[tykishore-snowflake]

I am trying to integrate Sage Maker Data wrangler flow with S3 Bucket (Access point) and getting an error. I dont see an option to pass requestor pay parameter in datawrangler flow. Is it possible? Please note that Sage Maker is in a different account (Consumer) than the S3 Bucket ( Producer Account).