[Python][CI] Some nightly jobs are failing due to ACCESS_DENIED to S3 bucket

[asfimport]

The following nitghly failures:

• test-conda-python-3.10

• test-conda-python-3.7

• test-conda-python-3.7-pandas-0.24

• test-conda-python-3.7-pandas-latest

• test-conda-python-3.8

• test-conda-python-3.8-pandas-latest

• test-conda-python-3.8-pandas-nightly

• test-conda-python-3.9

• test-conda-python-3.9-pandas-master

• test-debian-11-python-3

  Failed Python test_s3_real_aws_region_selection with ACCESS_DENIED:

   =================================== FAILURES ===================================
  ______________________ test_s3_real_aws_region_selection _______________________    @pytest.mark.s3
      def test_s3_real_aws_region_selection():
          # Taken from a registry of open S3-hosted datasets
          # at https://github.com/awslabs/open-data-registry
          fs, path = FileSystem.from_uri('s3://mf-nwp-models/README.txt')
          assert fs.region == 'eu-west-1'
  >       with fs.open_input_stream(path) as f:opt/conda/envs/arrow/lib/python3.10/site-packages/pyarrow/tests/test_fs.py:1660: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
  pyarrow/_fs.pyx:805: in pyarrow._fs.FileSystem.open_input_stream
      ???
  pyarrow/error.pxi:144: in pyarrow.lib.pyarrow_internal_check_status
      ???
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >   ???
  E   OSError: When reading information for key 'README.txt' in bucket 'mf-nwp-models': AWS Error ACCESS_DENIED during HeadObject operation: No response body.pyarrow/error.pxi:115: OSError 

Reporter: Raúl Cumplido / @raulcd
Assignee: Jacob Wujciak / @assignUser

_{Note: This issue was originally created as ARROW-17791. Please see the migration documentation for further details.}

[asfimport]

Antoine Pitrou / @pitrou:
I've seen that too. Can you restart some of those jobs to see if it's sporadic?

[asfimport]

Raúl Cumplido / @raulcd:
I've re-run the test-conda-python-3.10 one and it failed on the Retry too:

https://github.com/ursacomputing/crossbow/actions/runs/3094438413

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
It has been failing for multiple days, so doesn't seem sporadic

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
For the test-conda-python-3.8-pandas-latest build:

• last successful: https://github.com/ursacomputing/crossbow/actions/runs/3064659326/jobs/4947970745

• first failing: https://github.com/ursacomputing/crossbow/actions/runs/3071533959/jobs/4962296245

  Checking potential differences between both: for the conda environment itself, only the thrift build version changed, and certifi/ca-certificates updated from 2022.6.15.1 to 2022.9.14

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
I was checking further differences between both runs based on the logs, and the failing build has those additional env variables set:

env:
  ...
  AWS_SECRET_ACCESS_KEY: ***
  AWS_ACCESS_KEY_ID: ***
  SCCACHE_BUCKET: ***
  SCCACHE_S3_KEY_PREFIX: sccache

and those are not present in the working build. So probably something to do with the sccache change? (#13556, cc @assignUser)

[asfimport]

Antoine Pitrou / @pitrou:
Ahah, looks like we'll need to use non-standard env var names for sccache (such as SCCACHE_S3_ACCESS_KEY, SCCACHE_S3_SECRET_KEY)

[asfimport]

Jacob Wujciak / @assignUser:
Renaming the envvars is not an option because sccache would not detect them then. But we found the issue, the sccache user needs explicit permission to access any bucket. We have now added this and will add any other buckets that need to be accessed in jobs that use sccache.

Successful run here: https://github.com/ursacomputing/crossbow/actions/runs/3094438413/jobs/5047216106