Add support for AWS Auth with WebIdentity/OIDC

[jujugrrr]

If you run an AWS EKS cluster, Kubernetes pods and want to assume an IAM role, the official way to do it is through EKS-> IAM service mapping. It relies on AWS STS WebIdentity:

https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html

It's not an EKS specific feature but an OIDC one. It is supported by AWS SKDs/CLIS https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html

s3cmd ls s3://XXX-bucket

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    An unexpected error has occurred.
  Please try reproducing the error using
  the latest s3cmd code from the git master
  branch found at:
    https://github.com/s3tools/s3cmd
  and have a look at the known issues list:
    https://github.com/s3tools/s3cmd/wiki/Common-known-issues-and-their-solutions
  If the error persists, please report the
  following lines (removing any private
  info as necessary) to:
   s3tools-bugs@lists.sourceforge.net


!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Invoked as: /usr/bin/s3cmd ls s3://XXX-bucket
Problem: <class 'TypeError: can only concatenate str (not "bytes") to str
S3cmd:   2.0.2
python:   3.7.5 (default, Oct 17 2019, 12:25:15)
[GCC 8.3.0]
Traceback (most recent call last):
  File "/usr/bin/s3cmd", line 3092, in <module>
    rc = main()
  File "/usr/bin/s3cmd", line 3001, in main
    rc = cmd_func(args)
  File "/usr/bin/s3cmd", line 145, in cmd_ls
    subcmd_bucket_list(s3, uri, cfg.limit)
  File "/usr/bin/s3cmd", line 181, in subcmd_bucket_list
    response = s3.bucket_list(bucket, prefix = prefix, limit = limit)
  File "/usr/lib/python3.7/site-packages/S3/S3.py", line 309, in bucket_list
    for truncated, dirs, objects in self.bucket_list_streaming(bucket, prefix, recursive, uri_params, limit):
  File "/usr/lib/python3.7/site-packages/S3/S3.py", line 343, in bucket_list_streaming
    response = self.bucket_list_noparse(bucket, prefix, recursive, uri_params, max_keys)
  File "/usr/lib/python3.7/site-packages/S3/S3.py", line 374, in bucket_list_noparse
    response = self.send_request(request)
  File "/usr/lib/python3.7/site-packages/S3/S3.py", line 1252, in send_request
    method_string, resource, headers = request.get_triplet()
  File "/usr/lib/python3.7/site-packages/S3/S3.py", line 201, in get_triplet
    self.sign()
  File "/usr/lib/python3.7/site-packages/S3/S3.py", line 197, in sign
    bucket_region, self.headers, self.body)
  File "/usr/lib/python3.7/site-packages/S3/Crypto.py", line 196, in sign_request_v4
    signing_key = getSignatureKey(secret_key, datestamp, region, service)
  File "/usr/lib/python3.7/site-packages/S3/Crypto.py", line 176, in getSignatureKey
    kDate = sign(encode_to_s3('AWS4' + key), dateStamp)
TypeError: can only concatenate str (not "bytes") to str

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/s3cmd", line 3172, in <module>
    report_exception(e)
  File "/usr/bin/s3cmd", line 3047, in report_exception
    sys.stderr.write(u"environment LANG=%s\n" % unicodise_s(os.getenv("LANG"), 'ascii'))
  File "/usr/lib/python3.7/site-packages/S3/Utils.py", line 321, in unicodise_s
    return unicodise(string, encoding, errors, True)
  File "/usr/lib/python3.7/site-packages/S3/Utils.py", line 312, in unicodise
    return unicode(string, encoding, errors)
TypeError: decoding to str: need a bytes-like object, NoneType found

aws s3 ls s3://XXX-bucket
2020-03-06 15:57:58        164 test
2020-03-06 15:58:03        164 test2

for reference in botocore: https://github.com/boto/botocore/blob/d01fba6ec1c6e183bb031a4e89b6d417213489f4/botocore/credentials.py#L1615

[fviard]

Thank you very much for the initial detailed issue report.

I have pushed some changes to MASTER to support your webidentity/oidc case.
With commits up to d761ead , this feature should now be available.

If you can give it a try and tell me if everything is working as expected, that would be highly appreciated.

s3cmd now supports AssumeRole and AssumeRoleWithWebIdentity.
If you only have the AWS_ROLE_ARN env variable set, "AssumeRole" will be used.
If you have AWS_WEB_IDENTITY_TOKEN_FILE set in addition, it is "AssumeRoleWithWebIdentity" that will be used.

[WarheadsSE]

Thanks to all for this work.

Can we get a release tagged that includes this? 🤞 🙇

[danielwegener]

Does this solution is intended to support sts regional endpoints?

[danielwegener]

Altermatively: Is there any way to let the sts client use the proxy configuration? It looks like as if httplib would not respect http(s)_proxy environment variables.

[jujugrrr]

Thank you!