`DefaultAWSCredentialsProviderChain` does not work with `credential_process` and named profiles

[stevenmanton]

Describe the bug

Using named profiles along with credential_process appears to be incompatible with the DefaultAWSCredentialsProviderChain class.

Expected Behavior

Using AWS profiles should be compatible with the credential_process.

Current Behavior

In using the PyArrow library I came across an issue where using credential_process to authenticate fails only when using a named AWS profile. That library simply calls to DefaultAWSCredentialsProviderChain, so it seems most likely an issue with the AWS SDK itself.

In short, suppose that I have a config file as follows:

[default]
region = us-east-1
credential_process = /path/to/get-creds.sh

[dev]
region = us-east-1
credential_process = /path/to/get-creds.sh

These commands both work, which validates that both accounts have access to S3:

aws s3 ls s3://bucket/path --profile default
aws s3 ls s3://bucket/path --profile dev

Since PyArrow uses DefaultAWSCredentialsProviderChain under the hood, we should be able to set the environment variables to control which profile is used. Let's use a simple script to test the behavior:

# script.py
import pyarrow.dataset as ds

dataset = ds.dataset("s3://bucket/path/")  # authenticates with `DefaultAWSCredentialsProviderChain`

Now, here's the strangeness:

# Check that no environment variables are set in this shell:
env | grep AWS

# This works:
python ./script.py
# So does this:
AWS_PROFILE=default python ./script.py
# But this fails:
AWS_PROFILE=dev python ./script.py

The error is:

Traceback (most recent call last):
  File "./script.py", line 3, in <module>
    dataset = ds.dataset("s3://bucket/path/")
  File "/home/antonstv/miniconda3/envs/pdna/lib/python3.8/site-packages/pyarrow/dataset.py", line 763, in dataset
    return _filesystem_dataset(source, **kwargs)
  File "/home/antonstv/miniconda3/envs/pdna/lib/python3.8/site-packages/pyarrow/dataset.py", line 446, in _filesystem_dataset
    fs, paths_or_selector = _ensure_single_source(source, filesystem)
  File "/home/antonstv/miniconda3/envs/pdna/lib/python3.8/site-packages/pyarrow/dataset.py", line 413, in _ensure_single_source
    file_info = filesystem.get_file_info(path)
  File "pyarrow/_fs.pyx", line 571, in pyarrow._fs.FileSystem.get_file_info
  File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 115, in pyarrow.lib.check_status
OSError: When getting information for key 'path' in bucket 'bucket': AWS Error ACCESS_DENIED during HeadObject operation: No response body.

I've also confirmed that the above script works if instead of credential_process, I used fixed credentials that were created by the credential process.

Reproduction Steps

See above.

Possible Solution

No response

Additional Information/Context

No response

AWS CPP SDK version used

I'm unsure; pyarrow is 12.0.1

Compiler and Version used

I'm unsure

Operating System and version

I'm unsure

[yasminetalby]

Hello @stevenmanton ,

Thank you very much for your submission.
I am not very familiar with pyarrow but I will attempt to reproduce this issue using the AWS CPP SDK directly.

In the meantime, could you please attempt to set the dev profile environment variable as the default profile and see if you are able to successfully use credential_process? You can do so by running: export AWS_PROFILE=dev (See).

Best regards,

Yasmine

[stevenmanton]

Thanks @yasminetalby!

I tried this and I get the same error:

export AWS_PROFILE=dev
python script.py

[yasminetalby]

Hello @stevenmanton ,

I hope that you are doing well. My apologies for the delay of my answer.
I have attempted with both the AWS CLI and AWS CPP SDK to use credential_process within the default profile and a "dev" profile as defined above and was able to run a list object request successfully so there is no issue with credential_process using different AWS Profile in the CPP SDK.

I wanted to double check with you have provided an AWS Region within your environment and how your ~/.aws/config file looks like? (I suspect that this behavior might be caused by a difference in region setting within the two profiles)

Best regards,

Yasmine

[stevenmanton]

Ok, I think I figured it out! This issue appears to originate in the difference in format between ~/.aws/config and ~/.aws/credentials. In the former, profiles appear with profile, e.g. [profile default] or [profile dev]. In the latter, the profile is omitted: [default] or [dev]. (I'm not sure why the difference, but it's there.) It seems that many tools ignore this distinction, but something about this SDK seems to take it into account.

In my case, changing my ~/.aws/config to the following works:

[profile default]
region = us-east-1
credential_process = /path/to/get-creds.sh

[profile dev]
region = us-east-1
credential_process = /path/to/get-creds.sh

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[yasminetalby]

Hello Steve,

Awesome! I am glad it is working for you now.
As a side note, the format of the configuration and credential files has the following requirement:

Depending on the file, profile names use the following format:

• Config file: [default] [profile user1]
• Credentials file: [default] [user1]
• Do not use the word profile when creating an entry in the credentials file.
• All entries in a section take the general form of setting_name=value.
• Lines can be commented out by starting the line with a hashtag character (#).

(See documentation for the format of the configuration and credential files)

Linking this documentation for future reference!

Best regards,

Yasmine