[CI][Python] Some nightly python tests fail due to ACCESS DENIED to S3 bucket

[asfimport]

Several nightly tests fail with:

 =================================== FAILURES ===================================
____________________________ test_s3fs_wrong_region ____________________________    @pytest.mark.s3
    def test_s3fs_wrong_region():
        from pyarrow.fs import S3FileSystem
    
        # wrong region for bucket
        fs = S3FileSystem(region='eu-north-1')
    
        msg = ("When getting information for bucket 'voltrondata-labs-datasets': "
               r"AWS Error UNKNOWN \(HTTP status 301\) during HeadBucket "
               "operation: No response body. Looks like the configured region is "
               "'eu-north-1' while the bucket is located in 'us-east-2'."
               "|NETWORK_CONNECTION")
        with pytest.raises(OSError, match=msg) as exc:
            fs.get_file_info("voltrondata-labs-datasets")
    
        # Sometimes fails on unrelated network error, so next call would also fail.
        if 'NETWORK_CONNECTION' in str(exc.value):
            return
    
        fs = S3FileSystem(region='us-east-2')
>       fs.get_file_info("voltrondata-labs-datasets")opt/conda/envs/arrow/lib/python3.7/site-packages/pyarrow/tests/test_fs.py:1339: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pyarrow/_fs.pyx:571: in pyarrow._fs.FileSystem.get_file_info
    ???
pyarrow/error.pxi:144: in pyarrow.lib.pyarrow_internal_check_status
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >   ???
E   OSError: When getting information for bucket 'voltrondata-labs-datasets': AWS Error ACCESS_DENIED during HeadBucket operation: No response body. 

I can't seem to be able to reproduce locally but is pretty consistent:

• test-conda-python-3.10
• test-conda-python-3.11
• test-conda-python-3.7
• test-conda-python-3.7-pandas-latest
• test-conda-python-3.8
• test-conda-python-3.8-pandas-latest
• test-conda-python-3.8-pandas-nightly
• test-conda-python-3.9
• test-conda-python-3.9-pandas-upstream_devel

Reporter: Raúl Cumplido / @raulcd
Assignee: Miles Granger / @milesgranger

PRs and other links:

• GitHub Pull Request #14716

_{Note: This issue was originally created as ARROW-18392. Please see the migration documentation for further details.}

[asfimport]

Antoine Pitrou / @pitrou:
Issue resolved by pull request 14716
#14716