S3 bucket Access Denied when retrieving credentials using Amazon EKS Pod Identity #7157

jonsbun · 2024-02-22T07:13:22Z

Thanos version used:
v0.34.0

Object Storage Provider:
AWS S3

What happened:
Thanos Store gets "bucket store initial sync: sync block: BaseFetcher: iter bucket: Access Denied" error.

What you expected to happen:
Thanos Store can access AWS S3 bucket with credentials provided by Amazon EKS Pod Identity.

How to reproduce it (as minimally and precisely as possible):
Enable EKS Pod Identity for EKS cluster, and create Pod Identity associations for Thanos thanos-storegateway service account.

Helm Chart config override:

objstoreConfig:
  type: S3
  config:
    bucket: thanos-testing
    endpoint: s3.eu-west-2.amazonaws.com
    signature_version2: false
    aws_sdk_auth: true
    sse_config:
      type: SSE-S3
    trace:
      enable: true

storegateway:
  enabled: true
  logLevel: debug

Validate that required environment variables have been injected into Thanos Store pod:

kubectl get pods -n thanos thanos-storegateway-0 -o yaml | grep -A 10 env:
    env:
    - name: AWS_STS_REGIONAL_ENDPOINTS
      value: regional
    - name: AWS_DEFAULT_REGION
      value: eu-west-2
    - name: AWS_REGION
      value: eu-west-2
    - name: AWS_CONTAINER_CREDENTIALS_FULL_URI
      value: http://169.254.170.23/v1/credentials
    - name: AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE
      value: /var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-token

Full logs to relevant components:

ts=2024-02-22T06:37:53.436340159Z caller=main.go:67 level=debug msg="maxprocs: Leaving GOMAXPROCS=[2]: CPU quota undefined"
ts=2024-02-22T06:37:53.440305472Z caller=factory.go:53 level=info msg="loading bucket configuration"
ts=2024-02-22T06:37:53.444111533Z caller=inmemory.go:180 level=info msg="created in-memory index cache" maxItemSizeBytes=131072000 maxSizeBytes=262144000 maxItems=maxInt
ts=2024-02-22T06:37:53.445187821Z caller=options.go:26 level=info protocol=gRPC msg="disabled TLS, key and cert must be set to enable"
ts=2024-02-22T06:37:53.452140105Z caller=store.go:536 level=info msg="starting store node"
ts=2024-02-22T06:37:53.452491922Z caller=store.go:434 level=info msg="initializing bucket store"
ts=2024-02-22T06:37:53.453306038Z caller=fetcher.go:407 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=32
ts=2024-02-22T06:37:53.454546642Z caller=intrumentation.go:75 level=info msg="changing probe status" status=healthy
ts=2024-02-22T06:37:53.454633617Z caller=http.go:73 level=info service=http/server component=store msg="listening for requests and metrics" address=0.0.0.0:10902
ts=2024-02-22T06:37:53.455601899Z caller=tls_config.go:274 level=info service=http/server component=store msg="Listening on" address=[::]:10902
ts=2024-02-22T06:37:53.455627404Z caller=tls_config.go:277 level=info service=http/server component=store msg="TLS is disabled." http2=false address=[::]:10902
ts=2024-02-22T06:37:53.505091145Z caller=stdlib.go:105 level=debug s3TraceMsg=---------START-HTTP---------
ts=2024-02-22T06:37:53.505466076Z caller=stdlib.go:105 level=debug s3TraceMsg="GET /?delimiter=&encoding-type=url&fetch-owner=true&list-type=2&prefix= HTTP/1.1\r\nHost: thanos-testing.s3.dualstack.eu-west-2.amazonaws.com\r\nUser-Agent: MinIO (linux; amd64) minio-go/v7.0.61 thanos-store/0.34.0 (go1.21.7)\r\nAccept-Encoding: gzip\r\n\r"
ts=2024-02-22T06:37:53.505607441Z caller=stdlib.go:105 level=debug s3TraceMsg="HTTP/1.1 403 Forbidden\r\nTransfer-Encoding: chunked\r\nContent-Type: application/xml\r\nDate: Thu, 22 Feb 2024 06:37:53 GMT\r\nServer: AmazonS3\r\nX-Amz-Bucket-Region: eu-west-2\r\nX-Amz-Id-2: SjhXqkwh0JfwUuL9YU1Z9/oI2BX1gNMl5hsQMcKC54j4gbJ4fDreWZTUEH6ApEPAa2LmOsGB/Js=\r\nX-Amz-Request-Id: Z78BDFEX7BJ2C3D8\r\n\r\nf3\r\n<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>Z78BDFEX7BJ2C3D8</RequestId><HostId>SjhXqkwh0JfwUuL9YU1Z9/oI2BX1gNMl5hsQMcKC54j4gbJ4fDreWZTUEH6ApEPAa2LmOsGB/Js=</HostId></Error>\r\n0\r"
ts=2024-02-22T06:37:53.505713653Z caller=stdlib.go:105 level=debug s3TraceMsg=---------END-HTTP---------
ts=2024-02-22T06:38:03.45350041Z caller=fetcher.go:407 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=32
ts=2024-02-22T06:38:03.486260094Z caller=stdlib.go:105 level=debug s3TraceMsg=---------START-HTTP---------
ts=2024-02-22T06:38:03.486488347Z caller=stdlib.go:105 level=debug s3TraceMsg="GET /?delimiter=&encoding-type=url&fetch-owner=true&list-type=2&prefix= HTTP/1.1\r\nHost: thanos-testing.s3.dualstack.eu-west-2.amazonaws.com\r\nUser-Agent: MinIO (linux; amd64) minio-go/v7.0.61 thanos-store/0.34.0 (go1.21.7)\r\nAccept-Encoding: gzip\r\n\r"
ts=2024-02-22T06:38:03.486585737Z caller=stdlib.go:105 level=debug s3TraceMsg="HTTP/1.1 403 Forbidden\r\nTransfer-Encoding: chunked\r\nContent-Type: application/xml\r\nDate: Thu, 22 Feb 2024 06:38:03 GMT\r\nServer: AmazonS3\r\nX-Amz-Bucket-Region: eu-west-2\r\nX-Amz-Id-2: umOtTFxrcmnrS6IR4RtXg+0U7DxhngA6F4yFZcKokiXjMSYuTpDRBHl8UUc1jYbivN6TfxCNtQg=\r\nX-Amz-Request-Id: 07A9P8S3JVEN7FBY\r\n\r\nf3\r\n<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>07A9P8S3JVEN7FBY</RequestId><HostId>umOtTFxrcmnrS6IR4RtXg+0U7DxhngA6F4yFZcKokiXjMSYuTpDRBHl8UUc1jYbivN6TfxCNtQg=</HostId></Error>\r\n0\r"
ts=2024-02-22T06:38:03.486615125Z caller=stdlib.go:105 level=debug s3TraceMsg=---------END-HTTP---------
ts=2024-02-22T06:38:13.452870869Z caller=fetcher.go:407 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=32
ts=2024-02-22T06:38:13.475933686Z caller=stdlib.go:105 level=debug s3TraceMsg=---------START-HTTP---------
ts=2024-02-22T06:38:13.476296613Z caller=stdlib.go:105 level=debug s3TraceMsg="GET /?delimiter=&encoding-type=url&fetch-owner=true&list-type=2&prefix= HTTP/1.1\r\nHost: thanos-testing.s3.dualstack.eu-west-2.amazonaws.com\r\nUser-Agent: MinIO (linux; amd64) minio-go/v7.0.61 thanos-store/0.34.0 (go1.21.7)\r\nAccept-Encoding: gzip\r\n\r"
ts=2024-02-22T06:38:13.476531821Z caller=stdlib.go:105 level=debug s3TraceMsg="HTTP/1.1 403 Forbidden\r\nTransfer-Encoding: chunked\r\nContent-Type: application/xml\r\nDate: Thu, 22 Feb 2024 06:38:12 GMT\r\nServer: AmazonS3\r\nX-Amz-Bucket-Region: eu-west-2\r\nX-Amz-Id-2: wiI8VPr5jZ1agaocZ2FJOcJE7+YCGEmljemE8UsL7Ssm5PSf5FYJIiREV2PHel1lkw3k7oGqizQ=\r\nX-Amz-Request-Id: WYAY0DSNSE2SKNGG\r\n\r\nf3\r\n<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>WYAY0DSNSE2SKNGG</RequestId><HostId>wiI8VPr5jZ1agaocZ2FJOcJE7+YCGEmljemE8UsL7Ssm5PSf5FYJIiREV2PHel1lkw3k7oGqizQ=</HostId></Error>\r\n0\r"
ts=2024-02-22T06:38:13.476603236Z caller=stdlib.go:105 level=debug s3TraceMsg=---------END-HTTP---------
ts=2024-02-22T06:38:23.45535344Z caller=fetcher.go:407 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=32
ts=2024-02-22T06:38:23.506945138Z caller=stdlib.go:105 level=debug s3TraceMsg=---------START-HTTP---------
ts=2024-02-22T06:38:23.507202397Z caller=stdlib.go:105 level=debug s3TraceMsg="GET /?delimiter=&encoding-type=url&fetch-owner=true&list-type=2&prefix= HTTP/1.1\r\nHost: thanos-testing.s3.dualstack.eu-west-2.amazonaws.com\r\nUser-Agent: MinIO (linux; amd64) minio-go/v7.0.61 thanos-store/0.34.0 (go1.21.7)\r\nAccept-Encoding: gzip\r\n\r"
ts=2024-02-22T06:38:23.507355209Z caller=stdlib.go:105 level=debug s3TraceMsg="HTTP/1.1 403 Forbidden\r\nTransfer-Encoding: chunked\r\nContent-Type: application/xml\r\nDate: Thu, 22 Feb 2024 06:38:22 GMT\r\nServer: AmazonS3\r\nX-Amz-Bucket-Region: eu-west-2\r\nX-Amz-Id-2: 3r1NWkbr1iuLz/eenc0XsUgGXhmcHpwcJaDdx78lRT5VXfSfJUb0fUyw8cNJRMFrYIXGs/s+u+Q=\r\nX-Amz-Request-Id: ZWR94K981W7MGG6S\r\n\r\nf3\r\n<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>ZWR94K981W7MGG6S</RequestId><HostId>3r1NWkbr1iuLz/eenc0XsUgGXhmcHpwcJaDdx78lRT5VXfSfJUb0fUyw8cNJRMFrYIXGs/s+u+Q=</HostId></Error>\r\n0\r"
ts=2024-02-22T06:38:23.507383299Z caller=stdlib.go:105 level=debug s3TraceMsg=---------END-HTTP---------
ts=2024-02-22T06:38:23.508744419Z caller=intrumentation.go:67 level=warn msg="changing probe status" status=not-ready reason="bucket store initial sync: sync block: BaseFetcher: iter bucket: Access Denied"
ts=2024-02-22T06:38:23.50879216Z caller=http.go:91 level=info service=http/server component=store msg="internal server is shutting down" err="bucket store initial sync: sync block: BaseFetcher: iter bucket: Access Denied"
ts=2024-02-22T06:38:23.509022266Z caller=intrumentation.go:56 level=info msg="changing probe status" status=ready
ts=2024-02-22T06:38:23.509232581Z caller=grpc.go:131 level=info service=gRPC/server component=store msg="listening for serving gRPC" address=0.0.0.0:10901
ts=2024-02-22T06:38:23.509326963Z caller=http.go:110 level=info service=http/server component=store msg="internal server is shutdown gracefully" err="bucket store initial sync: sync block: BaseFetcher: iter bucket: Access Denied"
ts=2024-02-22T06:38:23.509349354Z caller=intrumentation.go:81 level=info msg="changing probe status" status=not-healthy reason="bucket store initial sync: sync block: BaseFetcher: iter bucket: Access Denied"
ts=2024-02-22T06:38:23.509911193Z caller=intrumentation.go:67 level=warn msg="changing probe status" status=not-ready reason="bucket store initial sync: sync block: BaseFetcher: iter bucket: Access Denied"
ts=2024-02-22T06:38:23.509947628Z caller=grpc.go:138 level=info service=gRPC/server component=store msg="internal server is shutting down" err="bucket store initial sync: sync block: BaseFetcher: iter bucket: Access Denied"
ts=2024-02-22T06:38:23.510006424Z caller=grpc.go:151 level=info service=gRPC/server component=store msg="gracefully stopping internal server"
ts=2024-02-22T06:38:23.510095322Z caller=grpc.go:164 level=info service=gRPC/server component=store msg="internal server is shutdown gracefully" err="bucket store initial sync: sync block: BaseFetcher: iter bucket: Access Denied"
ts=2024-02-22T06:38:23.510358692Z caller=main.go:161 level=error err="Access Denied\nBaseFetcher: iter bucket\ngithub.com/thanos-io/thanos/pkg/block.(*BaseFetcher).fetchMetadata\n\t/bitnami/blacksmith-sandox/thanos-0.34.0/src/github.com/thanos-io/thanos/pkg/block/fetcher.go:453\ngithub.com/thanos-io/thanos/pkg/block.(*BaseFetcher).fetch.func2\n\t/bitnami/blacksmith-sandox/thanos-0.34.0/src/github.com/thanos-io/thanos/pkg/block/fetcher.go:526\ngithub.com/golang/groupcache/singleflight.(*Group).Do\n\t/bitnami/blacksmith-sandox/thanos-0.34.0/pkg/mod/github.com/golang/groupcache@v0.0.0-20210331224755-41bb18bfe9da/singleflight/singleflight.go:56\ngithub.com/thanos-io/thanos/pkg/block.(*BaseFetcher).fetch\n\t/bitnami/blacksmith-sandox/thanos-0.34.0/src/github.com/thanos-io/thanos/pkg/block/fetcher.go:524\ngithub.com/thanos-io/thanos/pkg/block.(*MetaFetcher).Fetch\n\t/bitnami/blacksmith-sandox/thanos-0.34.0/src/github.com/thanos-io/thanos/pkg/block/fetcher.go:584\ngithub.com/thanos-io/thanos/pkg/store.(*BucketStore).SyncBlocks\n\t/bitnami/blacksmith-sandox/thanos-0.34.0/src/github.com/thanos-io/thanos/pkg/store/bucket.go:630\ngithub.com/thanos-io/thanos/pkg/store.(*BucketStore).InitialSync\n\t/bitnami/blacksmith-sandox/thanos-0.34.0/src/github.com/thanos-io/thanos/pkg/store/bucket.go:699\nmain.runStore.func5.1\n\t/bitnami/blacksmith-sandox/thanos-0.34.0/src/github.com/thanos-io/thanos/cmd/thanos/store.go:443\ngithub.com/thanos-io/thanos/pkg/runutil.RetryWithLog\n\t/bitnami/blacksmith-sandox/thanos-0.34.0/src/github.com/thanos-io/thanos/pkg/runutil/runutil.go:97\ngithub.com/thanos-io/thanos/pkg/runutil.Retry\n\t/bitnami/blacksmith-sandox/thanos-0.34.0/src/github.com/thanos-io/thanos/pkg/runutil/runutil.go:87\nmain.runStore.func5\n\t/bitnami/blacksmith-sandox/thanos-0.34.0/src/github.com/thanos-io/thanos/cmd/thanos/store.go:442\ngithub.com/oklog/run.(*Group).Run.func1\n\t/bitnami/blacksmith-sandox/thanos-0.34.0/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38\nruntime.goexit\n\t/opt/bitnami/go/src/runtime/asm_amd64.s:1650\nsync block\ngithub.com/thanos-io/thanos/pkg/store.(*BucketStore).InitialSync\n\t/bitnami/blacksmith-sandox/thanos-0.34.0/src/github.com/thanos-io/thanos/pkg/store/bucket.go:700\nmain.runStore.func5.1\n\t/bitnami/blacksmith-sandox/thanos-0.34.0/src/github.com/thanos-io/thanos/cmd/thanos/store.go:443\ngithub.com/thanos-io/thanos/pkg/runutil.RetryWithLog\n\t/bitnami/blacksmith-sandox/thanos-0.34.0/src/github.com/thanos-io/thanos/pkg/runutil/runutil.go:97\ngithub.com/thanos-io/thanos/pkg/runutil.Retry\n\t/bitnami/blacksmith-sandox/thanos-0.34.0/src/github.com/thanos-io/thanos/pkg/runutil/runutil.go:87\nmain.runStore.func5\n\t/bitnami/blacksmith-sandox/thanos-0.34.0/src/github.com/thanos-io/thanos/cmd/thanos/store.go:442\ngithub.com/oklog/run.(*Group).Run.func1\n\t/bitnami/blacksmith-sandox/thanos-0.34.0/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38\nruntime.goexit\n\t/opt/bitnami/go/src/runtime/asm_amd64.s:1650\nbucket store initial sync\nmain.runStore.func5\n\t/bitnami/blacksmith-sandox/thanos-0.34.0/src/github.com/thanos-io/thanos/cmd/thanos/store.go:448\ngithub.com/oklog/run.(*Group).Run.func1\n\t/bitnami/blacksmith-sandox/thanos-0.34.0/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38\nruntime.goexit\n\t/opt/bitnami/go/src/runtime/asm_amd64.s:1650\nstore command failed\nmain.main\n\t/bitnami/blacksmith-sandox/thanos-0.34.0/src/github.com/thanos-io/thanos/cmd/thanos/main.go:161\nruntime.main\n\t/opt/bitnami/go/src/runtime/proc.go:267\nruntime.goexit\n\t/opt/bitnami/go/src/runtime/asm_amd64.s:1650"

Anything else we need to know:
AWS released a new feature, EKS Pod Identity, that aims to simplify granting AWS access to pods running in an EKS cluster. This new feature is complementary to IAM roles for service accounts (IRSA), and provides a new alternative way to securely grant AWS permissions to pods.

Deep dive into the new Amazon EKS Pod Identity feature: https://securitylabs.datadoghq.com/articles/eks-pod-identity-deep-dive

The text was updated successfully, but these errors were encountered:

yeya24 · 2024-02-22T07:31:06Z

I think it is a valid request.
IIUC, Thanos doesn't support it right now. We rely on the underlying https://github.com/minio/minio-go to support this feature first.

From the current minio go SDK code, https://github.com/minio/minio-go/blob/de3d4928ef180eb8ca2eb7a15c0ced7a2f8ec667/pkg/credentials/iam_aws.go#L69 I don't see the new EKS pod identity is supported.

AI:

Open a feature request on minio-go Github repository

Test-008 · 2024-02-22T13:38:05Z

@here, I am also facing similar issues post upgrading thanos from 0.31.0 to 0.32.3.

takuan-osho · 2024-03-25T07:46:02Z

Now it seems that https://github.com/minio/minio-go started supporting EKS Pod Identity since it merged minio/minio-go#1944 into its master branch.

I hope the EKS Pod Identity-supported minio/minio-go version, will be released soon and that Thanos will use it and then release its new version.

yeya24 · 2024-03-25T08:51:21Z

Thanks for the update @takuan-osho. Let's wait for minio go to release a new version which includes that change first.
Then we can pull that change in ASAP at least in main branch.

Add support for EKS Pod Identity fix issue: thanos-io#7157 Signed-off-by: Hashem Taheri <iqfarhad@yahoo.com>

Add support for EKS Pod Identity fix issue: thanos-io#7157 Signed-off-by: farhad <eqfarhad@gmail.com>

* Update minio-go to v7.0.70 Add support for EKS Pod Identity fix issue: #7157 Signed-off-by: farhad <eqfarhad@gmail.com> * Changelog - support for EKS Pod Identity Updated changelog Signed-off-by: farhad <eqfarhad@gmail.com> --------- Signed-off-by: farhad <eqfarhad@gmail.com>

yeya24 · 2024-05-06T23:18:38Z

Resolved by #7335

* Update minio-go to v7.0.70 Add support for EKS Pod Identity fix issue: thanos-io#7157 Signed-off-by: farhad <eqfarhad@gmail.com> * Changelog - support for EKS Pod Identity Updated changelog Signed-off-by: farhad <eqfarhad@gmail.com> --------- Signed-off-by: farhad <eqfarhad@gmail.com> Signed-off-by: mluffman <nashluffman@gmail.com>

yeya24 added the help wanted label Feb 22, 2024

eqfarhad added a commit to eqfarhad/thanos that referenced this issue May 3, 2024

Update minio-go to v7.0.70

7abbcf1

Add support for EKS Pod Identity fix issue: thanos-io#7157 Signed-off-by: Hashem Taheri <iqfarhad@yahoo.com>

eqfarhad mentioned this issue May 3, 2024

Dependency - Update minio-go to v7.0.70 #7335

Merged

2 tasks

eqfarhad added a commit to eqfarhad/thanos that referenced this issue May 3, 2024

Update minio-go to v7.0.70

fe7e7bf

Add support for EKS Pod Identity fix issue: thanos-io#7157 Signed-off-by: farhad <eqfarhad@gmail.com>

eqfarhad added a commit to eqfarhad/thanos that referenced this issue May 3, 2024

Update minio-go to v7.0.70

e80544d

Add support for EKS Pod Identity fix issue: thanos-io#7157 Signed-off-by: farhad <eqfarhad@gmail.com>

eqfarhad added a commit to eqfarhad/thanos that referenced this issue May 6, 2024

Update minio-go to v7.0.70

4227dc8

Add support for EKS Pod Identity fix issue: thanos-io#7157 Signed-off-by: farhad <eqfarhad@gmail.com>

yeya24 closed this as completed May 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

S3 bucket Access Denied when retrieving credentials using Amazon EKS Pod Identity #7157

S3 bucket Access Denied when retrieving credentials using Amazon EKS Pod Identity #7157

jonsbun commented Feb 22, 2024

yeya24 commented Feb 22, 2024 •

edited

Loading

Test-008 commented Feb 22, 2024

takuan-osho commented Mar 25, 2024

yeya24 commented Mar 25, 2024

yeya24 commented May 6, 2024

S3 bucket Access Denied when retrieving credentials using Amazon EKS Pod Identity #7157

S3 bucket Access Denied when retrieving credentials using Amazon EKS Pod Identity #7157

Comments

jonsbun commented Feb 22, 2024

yeya24 commented Feb 22, 2024 • edited Loading

Test-008 commented Feb 22, 2024

takuan-osho commented Mar 25, 2024

yeya24 commented Mar 25, 2024

yeya24 commented May 6, 2024

yeya24 commented Feb 22, 2024 •

edited

Loading