Writing to S3 fails if the user is authenticated with `aws sso login`

[jayceslesar]

Apache Iceberg version

0.5.0 (latest release)

Please describe the bug 🐞

I have noticed that when I explicitly export my AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY that things work as expected but as soon as I unset those and use aws sso login to authenticate myself I get

File "pyarrow/error.pxi", line 115, in pyarrow.lib.check_status
OSError: When testing for existence of bucket 'my-iceberg-bucket': AWS Error ACCESS_DENIED during HeadBucket operation: No response body.

[jayceslesar]

Not the scope of this repo I dont think, but for other wondering I am able to do eval $(aws configure export-credentials --format env)....not super sure how to work that into this repo or if it is necessary but wanted to share

[jayceslesar]

This is confirmed an upstream bug in pyarrow 13.0.0 see apache/arrow#38013

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[github-actions]

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'