Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for ap-south-2 by bumping minio-go #6541

Closed
aittam opened this issue Jul 20, 2023 · 0 comments · Fixed by #6548
Closed

Add support for ap-south-2 by bumping minio-go #6541

aittam opened this issue Jul 20, 2023 · 0 comments · Fixed by #6548

Comments

@aittam
Copy link

aittam commented Jul 20, 2023

Thanos, Prometheus and Golang version used:

thanos v0.31.0
prometheus v2.43.0

Object Storage Provider:
s3

What happened:

thanos-store-0 is crashlooping when running in AWS ap-south-2

What you expected to happen:

to work exactly as in any other region

How to reproduce it (as minimally and precisely as possible):

On the AWS portal, activate the region ap-south-2, the deploy Thanos on EKS with a conf like:

type: S3
config:
  bucket: myBucket1dh73j83jd8jd
  region: ap-south-2
  endpoint: s3.ap-south-2.amazonaws.com
  sts_endpoint: sts.ap-south-2.amazonaws.com

Full logs to relevant components:

Logs

level=info ts=2023-07-20T10:55:03.533309055Z caller=factory.go:52 msg="loading bucket configuration"
level=info ts=2023-07-20T10:55:03.533753386Z caller=inmemory.go:174 msg="created in-memory index cache" maxItemSizeBytes=131072000 maxSizeBytes=262144000 maxItems=maxInt
level=info ts=2023-07-20T10:55:03.534106312Z caller=options.go:26 protocol=gRPC msg="disabled TLS, key and cert must be set to enable"
level=info ts=2023-07-20T10:55:03.53507317Z caller=store.go:490 msg="starting store node"
level=info ts=2023-07-20T10:55:03.535299777Z caller=intrumentation.go:75 msg="changing probe status" status=healthy
level=info ts=2023-07-20T10:55:03.535323072Z caller=http.go:73 service=http/server component=store msg="listening for requests and metrics" address=0.0.0.0:10902
level=info ts=2023-07-20T10:55:03.535574622Z caller=tls_config.go:232 service=http/server component=store msg="Listening on" address=[::]:10902
level=info ts=2023-07-20T10:55:03.535604215Z caller=tls_config.go:235 service=http/server component=store msg="TLS is disabled." http2=false address=[::]:10902
level=info ts=2023-07-20T10:55:03.535645817Z caller=store.go:391 msg="initializing bucket store"
level=warn ts=2023-07-20T10:55:34.360025979Z caller=intrumentation.go:67 msg="changing probe status" status=not-ready reason="bucket store initial sync: sync block: BaseFetcher: iter bucket: The ap-south-2 location constraint is incompatible for the region specific endpoint this request was sent to."
level=info ts=2023-07-20T10:55:34.360069885Z caller=http.go:91 service=http/server component=store msg="internal server is shutting down" err="bucket store initial sync: sync block: BaseFetcher: iter bucket: The ap-south-2 location constraint is incompatible for the region specific endpoint this request was sent to."
level=info ts=2023-07-20T10:55:34.360246506Z caller=intrumentation.go:56 msg="changing probe status" status=ready
level=info ts=2023-07-20T10:55:34.360356813Z caller=grpc.go:131 service=gRPC/server component=store msg="listening for serving gRPC" address=0.0.0.0:10901
level=info ts=2023-07-20T10:55:34.360483043Z caller=http.go:110 service=http/server component=store msg="internal server is shutdown gracefully" err="bucket store initial sync: sync block: BaseFetcher: iter bucket: The ap-south-2 location constraint is incompatible for the region specific endpoint this request was sent to."
level=info ts=2023-07-20T10:55:34.360515124Z caller=intrumentation.go:81 msg="changing probe status" status=not-healthy reason="bucket store initial sync: sync block: BaseFetcher: iter bucket: The ap-south-2 location constraint is incompatible for the region specific endpoint this request was sent to."
level=warn ts=2023-07-20T10:55:34.360650575Z caller=intrumentation.go:67 msg="changing probe status" status=not-ready reason="bucket store initial sync: sync block: BaseFetcher: iter bucket: The ap-south-2 location constraint is incompatible for the region specific endpoint this request was sent to."
level=info ts=2023-07-20T10:55:34.360666419Z caller=grpc.go:138 service=gRPC/server component=store msg="internal server is shutting down" err="bucket store initial sync: sync block: BaseFetcher: iter bucket: The ap-south-2 location constraint is incompatible for the region specific endpoint this request was sent to."
level=info ts=2023-07-20T10:55:34.360725201Z caller=grpc.go:151 service=gRPC/server component=store msg="gracefully stopping internal server"
level=info ts=2023-07-20T10:55:34.360824267Z caller=grpc.go:164 service=gRPC/server component=store msg="internal server is shutdown gracefully" err="bucket store initial sync: sync block: BaseFetcher: iter bucket: The ap-south-2 location constraint is incompatible for the region specific endpoint this request was sent to."
level=error ts=2023-07-20T10:55:34.361008657Z caller=main.go:161 err="The ap-south-2 location constraint is incompatible for the region specific endpoint this request was sent to.\nBaseFetcher: iter bucket\ngithub.com/thanos-io/thanos/pkg/block.(*BaseFetcher).fetchMetadata\n\t/app/pkg/block/fetcher.go:383\ngithub.com/thanos-io/thanos/pkg/block.(*BaseFetcher).fetch.func2\n\t/app/pkg/block/fetcher.go:447\ngithub.com/golang/groupcache/singleflight.(*Group).Do\n\t/go/pkg/mod/github.com/golang/groupcache@v0.0.0-20210331224755-41bb18bfe9da/singleflight/singleflight.go:56\ngithub.com/thanos-io/thanos/pkg/block.(*BaseFetcher).fetch\n\t/app/pkg/block/fetcher.go:445\ngithub.com/thanos-io/thanos/pkg/block.(*MetaFetcher).Fetch\n\t/app/pkg/block/fetcher.go:505\ngithub.com/thanos-io/thanos/pkg/store.(*BucketStore).SyncBlocks\n\t/app/pkg/store/bucket.go:527\ngithub.com/thanos-io/thanos/pkg/store.(*BucketStore).InitialSync\n\t/app/pkg/store/bucket.go:596\nmain.runStore.func3.1\n\t/app/cmd/thanos/store.go:400\ngithub.com/thanos-io/thanos/pkg/runutil.RetryWithLog\n\t/app/pkg/runutil/runutil.go:97\ngithub.com/thanos-io/thanos/pkg/runutil.Retry\n\t/app/pkg/runutil/runutil.go:87\nmain.runStore.func3\n\t/app/cmd/thanos/store.go:399\ngithub.com/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594\nsync block\ngithub.com/thanos-io/thanos/pkg/store.(*BucketStore).InitialSync\n\t/app/pkg/store/bucket.go:597\nmain.runStore.func3.1\n\t/app/cmd/thanos/store.go:400\ngithub.com/thanos-io/thanos/pkg/runutil.RetryWithLog\n\t/app/pkg/runutil/runutil.go:97\ngithub.com/thanos-io/thanos/pkg/runutil.Retry\n\t/app/pkg/runutil/runutil.go:87\nmain.runStore.func3\n\t/app/cmd/thanos/store.go:399\ngithub.com/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594\nbucket store initial sync\nmain.runStore.func3\n\t/app/cmd/thanos/store.go:405\ngithub.com/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594\nstore command failed\nmain.main\n\t/app/cmd/thanos/main.go:161\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"

Anything else we need to know:
I see minio-go introduced the support for that region in version v7.0.61, see minio/minio-go@5d09d8f

@aittam aittam changed the title Add support for ap-south-2 Add support for ap-south-2 by bumping minio-go Jul 20, 2023
jacobbaungard added a commit to jacobbaungard/objstore that referenced this issue Jul 21, 2023
Bumps minio-go to latest version. This should fix:
thanos-io/thanos#6541

Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>
jacobbaungard added a commit to jacobbaungard/thanos that referenced this issue Jul 21, 2023
Upgrade objstore to include the dependency bump of minio-io. This should
fix: thanos-io#6541

Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>
jacobbaungard added a commit to jacobbaungard/thanos that referenced this issue Jul 21, 2023
Upgrade objstore to include the dependency bump of minio-io. This should
fix: thanos-io#6541

Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>
@jacobbaungard jacobbaungard mentioned this issue Jul 21, 2023
2 tasks
jacobbaungard added a commit to jacobbaungard/thanos that referenced this issue Jul 21, 2023
Upgrade objstore to include the dependency bump of minio-io. This should
fix: thanos-io#6541

Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>
jacobbaungard added a commit to jacobbaungard/thanos that referenced this issue Jul 21, 2023
Upgrade objstore to include the dependency bump of minio-io. This should
fix: thanos-io#6541

Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>
jacobbaungard added a commit to jacobbaungard/thanos that referenced this issue Jul 21, 2023
Upgrade objstore to include the dependency bump of minio-io. This should
fix: thanos-io#6541

Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>
yeya24 pushed a commit that referenced this issue Aug 1, 2023
Upgrade objstore to include the dependency bump of minio-io. This should
fix: #6541

Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant