GH-46214: [C++] Improve S3 client initialization

[pitrou]

Rationale for this change

The default constructor of the S3ClientConfiguration class in the AWS SDK issues spurious EC2 metadata requests, even though we later set up the configuration values ourselves.

What changes are included in this PR?

1. Avoid spurious EC2 metadata calls by disabling "IMDS" in the S3ClientConfiguration constructor
2. Change the smart defaults from "legacy" to "standard" (see https://docs.aws.amazon.com/sdkref/latest/guide/feature-smart-config-defaults.html)
3. Let the user configure the smart defaults in S3Options

Benchmarks on my local work machine:

• on git main:

>>> %time fs.S3FileSystem(anonymous=True)
CPU times: user 316 μs, sys: 1.01 ms, total: 1.33 ms
Wall time: 2 s
<pyarrow._s3fs.S3FileSystem at 0x7d2182772fb0>
>>> %time fs.S3FileSystem(access_key='key', secret_key='secret')
CPU times: user 1.43 ms, sys: 0 ns, total: 1.43 ms
Wall time: 2 s
<pyarrow._s3fs.S3FileSystem at 0x7d22602337b0>
>>> %time fs.S3FileSystem()
CPU times: user 6.72 ms, sys: 3.27 ms, total: 9.99 ms
Wall time: 12 s
<pyarrow._s3fs.S3FileSystem at 0x7d226003bc30>

• on this PR:

>>> %time fs.S3FileSystem(anonymous=True)
CPU times: user 199 μs, sys: 0 ns, total: 199 μs
Wall time: 203 μs
<pyarrow._s3fs.S3FileSystem at 0x7d6c401c01b0>
>>> %time fs.S3FileSystem(access_key='key', secret_key='secret')
CPU times: user 198 μs, sys: 10 μs, total: 208 μs
Wall time: 212 μs
<pyarrow._s3fs.S3FileSystem at 0x7d6c4b2e2c30>
>>> %time fs.S3FileSystem()
CPU times: user 13.5 ms, sys: 3.52 ms, total: 17.1 ms
Wall time: 14 s
<pyarrow._s3fs.S3FileSystem at 0x7d6c40158df0>

Are these changes tested?

By existing CI tests and configurations.

Are there any user-facing changes?

The default S3 settings are potentially changed. Hopefully this will not trigger any regression in behavior.

• GitHub Issue: [C++] S3FileSystem construction causes IDMS lookups for region even when specified #46214

[pitrou]

cc @apmorton

[pitrou]

@github-actions crossbow submit -g python -g cpp

[pitrou]

@github-actions crossbow submit wheelcp313*

[pitrou]

@github-actions crossbow submit -g python -g cpp

[pitrou]

@github-actions crossbow submit -g python -g cpp

[github-actions]

Revision: 80bf6ee

Submitted crossbow builds: ursacomputing/crossbow @ actions-e43f4463ab

Task	Status
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
example-cpp-tutorial
example-python-minimal-build-fedora-conda
example-python-minimal-build-ubuntu-venv
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-conda-python-3.10
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-latest-numpy-latest
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-latest-numpy-1.26
test-conda-python-3.11-pandas-latest-numpy-latest
test-conda-python-3.11-pandas-nightly-numpy-nightly
test-conda-python-3.11-pandas-upstream_devel-numpy-nightly
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.12-cpython-debug
test-conda-python-3.13
test-conda-python-3.9
test-conda-python-3.9-pandas-1.1.3-numpy-1.19.5
test-conda-python-emscripten
test-cuda-cpp-ubuntu-22.04-cuda-11.7.1
test-cuda-python-ubuntu-22.04-cuda-11.7.1
test-debian-12-cpp-amd64
test-debian-12-cpp-i386
test-debian-12-python-3-amd64
test-debian-12-python-3-i386
test-fedora-39-cpp
test-fedora-39-python-3
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-20
test-ubuntu-22.04-cpp-bundled
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-22.04-python-3
test-ubuntu-22.04-python-313-freethreading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-bundled-offline
test-ubuntu-24.04-cpp-gcc-13-bundled
test-ubuntu-24.04-cpp-gcc-14
test-ubuntu-24.04-cpp-minimal-with-formats
test-ubuntu-24.04-cpp-thread-sanitizer
test-ubuntu-24.04-python-3

[kou]

+1

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit 6080ca6.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.