s3cmd is not working with AWS web identity token

[snakebyte91]

Image: registry.gitlab.com/gitlab-org/build/cng/gitlab-toolbox-ee:v14.4.1
s3cmd version: 2.2.0

$ s3cmd ls s3://my-bucket-name
ERROR: /home/git/.s3cfg: None
ERROR: Configuration file not available.
ERROR: Consider using --configure parameter to create one.

Configuration file is available:

$ cat /home/git/.s3cfg
[default]
bucket_location = eu-central-1

Environment variables are set:

$ env | grep AWS
AWS_DEFAULT_REGION=eu-central-1
AWS_REGION=eu-central-1
AWS_ROLE_ARN=arn:aws:iam::123456789:role/my-iam-role
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/toke

aws sts get-caller-identity shows me the correct IAM role from the service account.

[jseiser]

Same issue using gitlab/EKS in govcloud.

sh-4.2$ kubectl exec -it pod/gitlab-task-runner-744d679d68-b2cs6 -n gitlab -- cat /home/git/.s3cfg
Defaulted container "task-runner" out of: task-runner, certificates (init), configure (init)
[default]
bucket_location = us-gov-west-1
host_base = s3-us-gov-west-1.amazonaws.com
host_bucket = %(bucket)s.s3-us-gov-west-1.amazonaws.com

git@gitlab-task-runner-744d679d68-b2cs6:/$ s3cmd ls --debug
DEBUG: s3cmd version 2.2.0
DEBUG: ConfigParser: Reading file '/home/git/.s3cfg'
DEBUG: ConfigParser: bucket_location->us-gov-west-1
DEBUG: ConfigParser: host_base->s3-us-gov-west-1.amazonaws.com
DEBUG: ConfigParser: host_bucket->%(bucket)s.s3-us-gov-west-1.amazonaws.com
ERROR: /home/git/.s3cfg: None
ERROR: Configuration file not available.
ERROR: Consider using --configure parameter to create one.

git@gitlab-task-runner-744d679d68-b2cs6:/$ printenv | grep AWS
AWS_DEFAULT_REGION=us-gov-west-1
AWS_REGION=us-gov-west-1
AWS_ROLE_ARN=arn:aws-us-gov:iam::removed:role/role-gitlab-eks-dev
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token

git@gitlab-task-runner-744d679d68-b2cs6:/$ aws sts get-caller-identity
{
    "UserId": "AROA3RPS4FUHJSY3A6W6H:botocore-session-1636641879",
    "Account": "removed",
    "Arn": "arn:aws-us-gov:sts::removed:assumed-role/role-gitlab-eks-dev/botocore-session-1636641879"
}
gi

git@gitlab-task-runner-744d679d68-b2cs6:/$ aws sts get-caller-identity
{
    "UserId": "AROA3RPS4FUHJSY3A6W6H:botocore-session-1636641879",
    "Account": "removed",
    "Arn": "arn:aws-us-gov:sts::removed:assumed-role/role-gitlab-eks-dev/botocore-session-1636641879"
}
git@gitlab-task-runner-744d679d68-b2cs6:/$ aws sts assume-role-with-web-identity --role-arn $AWS_ROLE_ARN  --role-session-name gitlab --web-identity-token file://$AWS_WEB_IDENTITY_TOKEN_FILE
{
    "Credentials": {
        "AccessKeyId": "ASIA3RPS4FUHBKH5Z55D",
        "SecretAccessKey": "CSaw1sqIgB87S0leaVfoSp7R3o6vz6ylQKB+9KvC",
        "SessionToken": "FwoDYXdzEBUaDBIQEHl5EVLTLm3ypSKYBAb7HvwlxRszT2vnSrvBfJF7PYwdB19BTXTHBvQpVvlZon8Yy6KfXXuvZB69fTakXTs97QytdUETf//ryWkthls5/foCXCodulBGP/+Y3Ch9pBv7UTgfQtRV6kLHI8DG8eGOc4X6Jo+PbfJL/G2kPamIuOs2AnRRyop0WOQ2edupKrj7opEG1zFVv71NmNc9LwBpqAUKHEzq+O6QS+tjlLQPt2jncPdSI2iIw0M7uw2jn4mCSqau4kSpp2Cp4avlBkRP6PgqsEIT5gafwvOGg2DcCciLUzDVndu0k9v3kHCIFIGnVTSAz5BHMQOcU5SoX1RRh3122o7EQRBm89H1vdQmzGRvVtN3zcKCzKj/aJutgl3gvrEmYp8m86sZ4uNG7ZBd1SF8R+BD2v0/RPopuaZee3W6xIdSNuxGLI7iDU5v+lSKgdr/ysuGTyLDHJongfRV9+ASSm3w3udhLkQ5LLWXEWQmZ5uwvqDcynuX2EAq/WASVvb3JxEHFPiUcCV5DX4naT66jZoy8AAC2+ygXRAj+tPQHHq/a1PEMQyI2IwYYt8lOiyu+KIvQG1KbESg8t/ktqs3ePDiS+SeaEo5reB1UyoJZE/2mXbr8FNGZiwAUVFC8bW68pF8opAntiseSyRCHI+09B0tVu0+LVSWSbLhjFCO2C62u2qQqR/tBRp0dXCJVkYx9dss099aQ5KQD6SjZICEaGuIKJbdtIwGMiovgA/+OX96/gybqJNvVOyRVACzyukTjPAoxmqoegoXlwpdU/85Mr5+JA8=",
        "Expiration": "2021-11-11T15:54:14Z"
    },
    "SubjectFromWebIdentityToken": "system:serviceaccount:gitlab:gitlab-task-runner",
    "AssumedRoleUser": {
        "AssumedRoleId": "AROA3RPS4FUHJSY3A6W6H:gitlab",
        "Arn": "arn:aws-us-gov:sts::removed:assumed-role/role-gitlab-eks-dev/gitlab"
    },
    "Provider": "arn:aws-us-gov:iam::REMOVED:oidc-provider/oidc.eks.us-gov-west-1.amazonaws.com/id/D69A961F87B66A406FF8DD64FE8CE1E8",
    "Audience": "sts.amazonaws.com"
}

[jseiser]

@fviard

Is there any additional information we should provide?

[signaleleven]

It seems to be available in the latest release
https://github.com/s3tools/s3cmd/releases/tag/v2.2.0
Added support for STS webidentity authentication (ie AssumeRole and AssumeRoleWithWebIdentity) (Samskeyti, Florent Viard)
after being announced here #1075

I haven't tried yet.

[izinovik]

IRSA support works for me with GitLab 14.5.1.

[jseiser]

IRSA support works for me with GitLab 14.5.1.

We are on Gitlab 14.5.0, let me update to 14.5.1 and confirm. Thanks.

[snakebyte91]

I still get the error:

$ s3cmd --version
s3cmd version 2.2.0
$ s3cmd ls s3://my-bucket-name
ERROR: /home/git/.s3cfg: None
ERROR: Configuration file not available.
ERROR: Consider using --configure parameter to create one.

Image: registry.gitlab.com/gitlab-org/build/cng/gitlab-toolbox-ee:v14.5.1

[izinovik]

I configure .s3cfg using Terraform. Here is what I have in it:

git@gitlab-toolbox-6586899c6d-859fm:~$ cat /home/git/.s3cfg
[default]
bucket_location = eu-west-1
multipart_chunk_size_mb = 128

[lanzrein]

Hello,

This error could be due to the fact that s3cmd does not take into account AWS_STS_REGIONAL_ENDPOINTS environment variable.

I have opened an MR for this #1227 and a specific issue #1228

[tculp]

Can we get a new s3cmd release for this? GitLab is using the latest release (2.2.0) and so doesn't have this change.

[fviard]

Should be fixed already, and the update 2.3.0 should be released with it today.

[VladoPortos]

Not fixed in 15.5.1 at least I get this during restore:

ERROR: File 's3://eu-central-1-gitlab-uploads/user/avatar/19/avatar.png' could not be copied: 403 (AccessDenied): Access Denied
ERROR: File 's3://eu-central-1-gitlab-uploads/user/avatar/23/alert-bot.png' could not be copied: 403 (AccessDenied): Access Denied
ERROR: File 's3://eu-central-1-gitlab-uploads/user/avatar/24/support-bot.png' could not be copied: 403 (AccessDenied): Access Denied

Despite backuping everything and storing to S3 works, all other services work with S3, just the restore.

git@gitlab-toolbox-6db6555db7-zcbml:/$ env | grep AWS
AWS_DEFAULT_REGION=eu-central-1
AWS_REGION=eu-central-1
AWS_ROLE_ARN= __redacted__
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
AWS_STS_REGIONAL_ENDPOINTS=regional

However, I get also this warning:

WARNING: Ignoring invalid line in '/home/git/.s3cfg': provider: AWS
WARNING: Ignoring invalid line in '/home/git/.s3cfg': region: eu-central-1
WARNING: Ignoring invalid line in '/home/git/.s3cfg': use_iam_profile: true

Maybe the /home/git/.s3cfg is not correct ? What should be in there ?

[VladoPortos]

OK here is what did it for me:

/home/git/.s3cfg contains only:

[default]
multipart_chunk_size_mb = 128

The IAM role to access the bucket got S3:* cause following permissions were not enough:

       Action = [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject",
                "s3:ListMultipartUploadParts",
                "s3:AbortMultipartUpload"
            ]

I would be very happy if somebody could point me to permission I was missing. Thanks.