Support sts and static credentials to work together

[rpudlowski93]

Expected behavior

I would like to set MC_HOST_s3 env using IAM role. I would like to generate temporary credentials for the AWS service.

Actual behavior

I can not install aws cli on minio in order to retrieve temporary credentials so I need to use AWS ACCESS AND SECRET KEY. I would like to use token.

Right now, there is just a feature to retrieve temporary credentials only for minio, but i want s3 bucket.

Steps to reproduce the behavior

I have minio with mc and i want to use my IAM role assigned to Pod in EKS in order to generate temporary credentials for s3 service

mc --version

RELEASE-2023-11-06T04-19-23Z

System information

[robert-pudlowski-mox]

@harshavardhana do You think that it is possible to implement this feature soon?

[harshavardhana]

Yeah it's minor stuff.

[robert-pudlowski-mox]

cool! That's awesome. So in Your opinion, You will be able to implement it and if yes, what is the approximately date of the feature? It is very urgent for our daily basis.

Just for confirmation, I would like to setup MC_HOST_s3 based on IAM roles. Usually I should be able to retrieve the temporary ACCESS/SECRET/TOKEN values using AWS CLI but it is not available on minio image or mc.

[harshavardhana]

fixed in #4763

[rpudlowski93]

@harshavardhana can You explain me how we can setup the MC_HOST_s3 now? Ho to retrieve the temp creds?

[harshavardhana]

@harshavardhana can You explain me how we can setup the MC_HOST_s3 now? Ho to retrieve the temp creds?

MC_HOST_s3=https://s3.amazonaws.com
MC_WEB_IDENTITY_TOKEN_FILE=
MC_ROLE_ARN=

Obtain web identity token file and role arn

https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-role.html#cli-configure-role-oidc

Just like how you would set

AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
AWS_ROLE_ARN=arn:aws:iam::xxxxxxxxxxxx:role/s3-access

You should set the above values for mc.

[robert-pudlowski-mox]

@harshavardhana I added MC_HOST_s3=https://s3.amazonaws.com and I have the envs AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
AWS_ROLE_ARN=arn:aws:iam::xxxxxxxxxxxx:role/s3-access

as well.

Looks it it trying to use them but I'm getting
mc: <ERROR> Failed to copy http://opencti-minio:9000/opencti-bucket/import/Artifact/xxxxxxx`. Insufficient permissions to access this path `https://s3.amazonaws.com/xxxxx/opencti-bucket/import/Artifact/xxxxxxxx``

Do You have any idea if something is still missing or maybe the role is not taken properly? I can see in AWS console that the role is not used in my last hours.

[robert-pudlowski-mox]

There is 403 error and Access Denied in debug mode to S3 bucket.

[robert-pudlowski-mox]

Looks like the MC_HOST_s3 is not taking the temporary Access Key, Secret Key and Session Token into account

We have MC_HOST_s3=https://s3.amazonaws.com/
but should be https:/$ACCESS_KEY:$SECRET_KEY:$TOKEN_SESSION@s3.ap-east-1.amazonaws.com

But how to retrieve these values?

[harshavardhana]

@harshavardhana I added MC_HOST_s3=https://s3.amazonaws.com and I have the envs AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
AWS_ROLE_ARN=arn:aws:iam::xxxxxxxxxxxx:role/s3-access

@robert-pudlowski-mox with mc you must MC_ envs not AWS_ - I had mentioned it clearly that that our values are equivalent of what AWS CLI provides.

MC_HOST_s3=https://s3.amazonaws.com
MC_WEB_IDENTITY_TOKEN_FILE=
MC_ROLE_ARN=

[robert-pudlowski-mox]

@harshavardhana sure! I already checked it as well but nothing has changed.

My minio deployment definition:

ENVS:

• name: MC_HOST_s3
  value: https://s3.ap-east-1.amazonaws.com
• name: MC_WEB_IDENTITY_TOKEN_FILE
  value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
• name: MC_ROLE_ARN
  value: arn:aws:iam::xxx:role/xxx-role

But the issue is still the same

[robert-pudlowski-mox]

I have the IAM role assigned to service account which is used by pod. So it should be fine. It works for ILM with the role and service account.

[harshavardhana]

@harshavardhana sure! I already checked it as well but nothing has changed.

My minio deployment definition:

ENVS:

• name: MC_HOST_s3
  value: https://s3.ap-east-1.amazonaws.com
• name: MC_WEB_IDENTITY_TOKEN_FILE
  value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
• name: MC_ROLE_ARN
  value: arn:aws:iam::xxx:role/xxx-role

But the issue is still the same

When you are doing this can you mc ls --debug and share the output?

[robert-pudlowski-mox]

@harshavardhana

`mc ls --debug s3/xxxx-bucket
mc: HEAD / HTTP/1.1
Host: xxxx-bucket.s3.dualstack.ap-east-1.amazonaws.com
User-Agent: MinIO (linux; amd64) minio-go/v7.0.63 mc/RELEASE.2023-11-15T22-45-58Z

mc: HTTP/1.1 403 Forbidden
Connection: close
Content-Type: application/xml
Date: Fri, 17 Nov 2023 13:07:27 GMT
Server: AmazonS3
X-Amz-Bucket-Region: ap-east-1
X-Amz-Id-2: P/VItbk7pmEJeRpQXkysYdd6enKcApaptywzGAKWJLn2mm3zFrFQg652CNCIvo0047t+Vp9lqls=
X-Amz-Request-Id: ZKBVWFC696YS785Y

mc: TLS Certificate found:
mc: >> Country: US
mc: >> Organization: Amazon
mc: >> Expires: 2024-03-03 23:59:59 +0000 UTC
mc: TLS Certificate found:
mc: >> Country: US
mc: >> Organization: Amazon
mc: >> Expires: 2030-08-23 22:21:28 +0000 UTC
mc: TLS Certificate found:
mc: >> Country: US
mc: >> Organization: Starfield Technologies, Inc.
mc: >> Expires: 2037-12-31 01:00:00 +0000 UTC
mc: TLS Certificate found:
mc: >> Country: US
mc: >> Organization: Starfield Technologies, Inc.
mc: >> Expires: 2034-06-28 17:39:16 +0000 UTC
mc: Response Time: 29.588739ms

mc: HEAD / HTTP/1.1
Host: xxxx-bucket.s3.dualstack.ap-east-1.amazonaws.com
User-Agent: MinIO (linux; amd64) minio-go/v7.0.63 mc/RELEASE.2023-11-15T22-45-58Z

mc: HTTP/1.1 403 Forbidden
Connection: close
Content-Type: application/xml
Date: Fri, 17 Nov 2023 13:07:27 GMT
Server: AmazonS3
X-Amz-Bucket-Region: ap-east-1
X-Amz-Id-2: gtwD1VlExVFnhd9pxeHKeWXhSf6EI7Qa46lr4xLPoJJN/VbKIe1L1VJkQDZr5OAyY9IWmxcY62U=
X-Amz-Request-Id: ZKBXK9CRAGV9R0NG

mc: TLS Certificate found:
mc: >> Country: US
mc: >> Organization: Amazon
mc: >> Expires: 2024-03-03 23:59:59 +0000 UTC
mc: TLS Certificate found:
mc: >> Country: US
mc: >> Organization: Amazon
mc: >> Expires: 2030-08-23 22:21:28 +0000 UTC
mc: TLS Certificate found:
mc: >> Country: US
mc: >> Organization: Starfield Technologies, Inc.
mc: >> Expires: 2037-12-31 01:00:00 +0000 UTC
mc: TLS Certificate found:
mc: >> Country: US
mc: >> Organization: Starfield Technologies, Inc.
mc: >> Expires: 2034-06-28 17:39:16 +0000 UTC
mc: Response Time: 4.413231ms

mc: Unable to list folder. Access Denied.
(2) ls.go:239 cmd.doList(..) Tags: [https://s3.ap-east-1.amazonaws.com/xxxx-bucket]
(1) client-s3.go:2364 cmd.(*S3Client).listInRoutine(..) Tags: [xxxx-bucket]
(0) client-s3.go:2330 cmd.(*S3Client).bucketStat(..)
Release-Tag:RELEASE.2023-11-15T22-45-58Z | Commit:4724c024c6de | Host:opencti-minio-8bd97bd7c-5lk65 | OS:linux | Arch:amd64 | Lang:go1.21.4 | Mem:3.0 MiB/15 MiB | Heap:3.0 MiB/7.4 MiB`

[rpudlowski93]

@harshavardhana ??

[harshavardhana]

And what are the environment variables set before using mc ?

[robert-pudlowski-mox]

@harshavardhana what do You mean?

All the envs which I adding to the deployment:

    env:
    - name: MINIO_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          key: MINIO_ACCESS_KEY
          name: opencti
    - name: MINIO_SECRET_KEY
      valueFrom:
        secretKeyRef:
          key: MINIO_SECRET_KEY
          name: opencti
    - name: S3_BUCKET_NAME
      valueFrom:
        secretKeyRef:
          key: S3_BUCKET_NAME
          name: opencti
    - name: MC_HOST_s3
      value: https://s3.ap-east-1.amazonaws.com
    - name: MC_WEB_IDENTITY_TOKEN_FILE
      value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    - name: MC_ROLE_ARN
      value: arn:aws:iam::xx:role/xxxx-role

Nothing more :) There are of course some default environments which comes from EKS. Any idea how to deal with the STS for S3?

[harshavardhana]

And what is the version of mc - can you provide mc --version output?

[robert-pudlowski-mox]

I use the minio image: minio/minio:RELEASE.2023-11-15T20-43-25Z

[harshavardhana]

I use the minio image: minio/minio:RELEASE.2023-11-15T20-43-25Z

Have you used latest mc? https://github.com/minio/mc/releases/tag/RELEASE.2023-11-15T22-45-58Z

The mc inside MinIO container is not the latest

[harshavardhana]

ah I see your problem you have not set the MC_STS_ENDPOINT @robert-pudlowski-mox

[harshavardhana]

In your ENV this must be set to perhaps https://sts.ap-east-1.amazonaws.com

[robert-pudlowski-mox]

@harshavardhana thanks man! it works. One more question. After adding the env MC_STS_ENDPOINT=https://sts.ap-east-1.amazonaws.com, now I can not add alias for local minio deployment which was working before:

mc alias set myminio http://opencti-minio:9000 $MINIO_ACCESS_KEY $MINIO_SECRET_KEY
mc: Configuration written to /tmp/.mc/config.json. Please update your access credentials.
mc: Successfully created /tmp/.mc/share.
mc: Initialized share uploads /tmp/.mc/share/uploads.json file.
mc: Initialized share downloads /tmp/.mc/share/downloads.json file.
mc: Unable to initialize new alias from the provided credentials. The Access Key Id you provided does not exist in our records.

Any idea how to make the S3 alias and minio local working?

[harshavardhana]

@robert-pudlowski-mox right now that is not possible - I have to think about adding that support.

[robert-pudlowski-mox]

@harshavardhana so how can I copy files from local minio to S3 bucket if I'm not able to connect in the same time to local minio and S3 bucket? :D

Now I can just send something to S3 bucket or I can only connect to local minio but not both.

[robert-pudlowski-mox]

@harshavardhana and one more question, if I have the configuration setup for S3 bucket (MC_HOST_s3, MC_STS_ENDPOINT etc..) does the ilm which is already in place and setup for local minio and ilm with S3 tier will work? Or it will stop working?

[harshavardhana]

@harshavardhana so how can I copy files from local minio to S3 bucket if I'm not able to connect in the same time to local minio and S3 bucket? :D

Now I can just send something to S3 bucket or I can only connect to local minio but not both.

Correct this is something that needs to be fixed.

[harshavardhana]

@harshavardhana and one more question, if I have the configuration setup for S3 bucket (MC_HOST_s3, MC_STS_ENDPOINT etc..) does the ilm which is already in place and setup for local minio and ilm with S3 tier will work? Or it will stop working?

mc changes won't affect ILM which is server side config.

[harshavardhana]

This is fully fixed here #4771