Credentials for aws s3 objects source

[btheu]

Hi,

While trying to use s3 objects source i got following error:

WARN dataflow::source::s3: unable to list bucket mybucket1: Couldn't find AWS credentials in environment, credentials file, or IAM role. (1 retries remaining)

Current /root/.aws/credentials content is as follow (after secret substitution):

[default]
role_arn=arn:aws:iam::<id>:role/<rolename>
role_session_name=materialize
source_profile=staging

[staging]
aws_access_key_id=<secret>
aws_secret_access_key=<secret>

As validation, aws s3 ls command is successful on AWS CLI2 with this credentials file.

What should looks like credentials file for using s3 objects source from Materialize ?

regards,

[quodlibetor]

That looks reasonable and should be picked up. To double check, is materialized running as root?

[btheu]

Unfortunately, my provided aws_access need to assume this role in order create/read any aws resources. (firm policy)

[quodlibetor]

Okay, that is a very reasonable policy, and yes I have confirmed that these are the two problems:

• Our credentials library isn't correctly handling the split profile
• We aren't assuming the role in the credentials file

I'm not 100% certain what the source of the first issue is so I'm not sure how complex the fix to it is. The second issue is reasonably straightforward to fix, although I believe that the credentials parser is also not handling that so the underlying fix might take a bit.

The fastest workaround right now that I can think of is to use the aws sts cli to create the appropriate credentials (including the session token, which should be converted to aws_session_token = ), and inline them into the default profile in your credentials file. Running this command will emit some json with credentials for the intended role to assume:

aws --profile default sts assume-role --role-arn arn:aws:iam::,,, --role-session-name materialize

Unfortunately those credentials will expire (by default about once an hour) so this only works for experimentation. A more long-term workaround is to configure an EC2 instance that has an InstanceProfile that assumes the correct role.

[quodlibetor]

I've created #5895 to track the implementation of roles for AWS sources.

[benesch]

There are some serious deficiencies with the way Rusoto handles credential profiles. Somewhat covered in rusoto/rusoto#1897. Ugh.

[btheu]

Thanks for the reply, i'll give token credentials a try