[C++] Automatic S3 region selection

[asfimport]

Currently, PyArrow and ArrowCpp need to be provided the region of the S3 file/bucket, else it defaults to using 'us-east-1'. Ideally, PyArrow and ArrowCpp can automatically detect the region and get the files, etc. For instance, s3fs and boto3 can read and write files without having to specify the region explicitly. Similar functionality to auto-detect the region would be great to have in PyArrow and ArrowCpp.

Environment: macOS, Linux.
Reporter: Sahil Gupta / @sahil1105
Assignee: Antoine Pitrou / @pitrou

PRs and other links:

• GitHub Pull Request #8205

_{Note: This issue was originally created as ARROW-9775. Please see the migration documentation for further details.}

[asfimport]

Antoine Pitrou / @pitrou:
Do you want to submit a PR with the desired changes?

[asfimport]

Antoine Pitrou / @pitrou:
It seems it can be determined through a HEAD request on a bucket:
aws/aws-cli#2431

This is how boto does it:
https://github.com/boto/botocore/pull/936/files

A S3Client is bound to a region, so some care will be needed in the implementation.

[asfimport]

Antoine Pitrou / @pitrou:
Notes to self:

• the "x-amz-bucket-region" header is always populated on return if the bucket exists, even if it's the same region
• the HTTP status can be 301 for a public bucket, 403 if authentication will be required
• HTTP status 404 means the bucket doesn't exist in any region

[asfimport]

Antoine Pitrou / @pitrou:
Issue resolved by pull request 8205
#8205