RadosGW refusing requests by AccessDenied #13837
Unanswered
ausias-armesto
asked this question in
Q&A
Replies: 1 comment 6 replies
-
Does this only affect a single bucket? It seems there is an issue with the creds for that object user. Or do you see this across multiple buckets? If you create a new bucket, can it be accessed successfully? |
Beta Was this translation helpful? Give feedback.
6 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
My installation is the following:
OS: Debian GNU/Linux 11 (bullseye)
Kernel: Linux server-stage-fsn1-002 5.10.0-9-amd64 SMP Debian 5.10.70-1 (2021-09-30) x86_64 GNU/Linux
Cloud provider or hardware configuration: Hetzner Dedicated servers
Rook version: v1.10.13
Ceph version: v17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)
Kubernetes version: v1.24.6
Kubernetes cluster type: Kubespray
Healthy Cluster: OK
I'm facing an issue with clients trying to connect to perform operations in a bucket. All of a sudden it stopped working and now radosgw is rejecting all requests with an authentication error.
By executing this command I can see the
access_key
andsecret_key
for the bucket owner.Then in the toolbox pod, I configure those keys in ~/.aws/credentials and execute a simple list command:
I've increased the log level in radosgw and these are the logs there
RadosGW Logs
``` debug 2024-02-29T12:40:00.446+0000 7392ce94c700 1 ====== starting new request req=0x73927e76d730 ===== debug 2024-02-29T12:40:00.446+0000 7392ce94c700 2 req 8249676414073575284 0.000000000s initializing for trans_id = tx00000727cbcfd6cc61774-0065e07b20-18276c0-ceph-new-objectstore debug 2024-02-29T12:40:00.446+0000 7392ce94c700 10 req 8249676414073575284 0.000000000s rgw api priority: s3=8 s3website=7 debug 2024-02-29T12:40:00.446+0000 7392ce94c700 10 req 8249676414073575284 0.000000000s host=rook-ceph-rgw-ceph-new-objectstore.rook-ceph.svc debug 2024-02-29T12:40:00.446+0000 7392ce94c700 20 req 8249676414073575284 0.000000000s subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 debug 2024-02-29T12:40:00.446+0000 7392ce94c700 20 req 8249676414073575284 0.000000000s final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/bucket-loki/fake/4544f2114b8743e6/18de51eee34%3A18de5541c17%3A41ecc98b debug 2024-02-29T12:40:00.446+0000 7392ce94c700 10 req 8249676414073575284 0.000000000s meta>> HTTP_X_AMZ_CONTENT_SHA256 debug 2024-02-29T12:40:00.446+0000 7392ce94c700 10 req 8249676414073575284 0.000000000s meta>> HTTP_X_AMZ_DATE debug 2024-02-29T12:40:00.446+0000 7392ce94c700 10 req 8249676414073575284 0.000000000s meta>> HTTP_X_AMZ_STORAGE_CLASS debug 2024-02-29T12:40:00.446+0000 7392ce94c700 10 req 8249676414073575284 0.000000000s x>> x-amz-content-sha256:2aff068799ede5a30594bc5f2e49bfbf061867ecc9d85246406c3705739b7770 debug 2024-02-29T12:40:00.446+0000 7392ce94c700 10 req 8249676414073575284 0.000000000s x>> x-amz-date:20240229T123958Z debug 2024-02-29T12:40:00.446+0000 7392ce94c700 10 req 8249676414073575284 0.000000000s x>> x-amz-storage-class:STANDARD debug 2024-02-29T12:40:00.446+0000 7392ce94c700 20 req 8249676414073575284 0.000000000s get_handler handler=22RGWHandler_REST_Obj_S3 debug 2024-02-29T12:40:00.446+0000 7392ce94c700 10 req 8249676414073575284 0.000000000s handler=22RGWHandler_REST_Obj_S3 debug 2024-02-29T12:40:00.446+0000 7392ce94c700 2 req 8249676414073575284 0.000000000s getting op 1 debug 2024-02-29T12:40:00.446+0000 7392ce94c700 20 req 8249676414073575284 0.000000000s get_system_obj_state: rctx=0x73927e76c7c0 obj=ceph-new-objectstore.rgw.log:script.prerequest. state=0x567e336e5720 s->prefetch_data=0 debug 2024-02-29T12:40:00.446+0000 7392ce94c700 20 req 8249676414073575284 0.000000000s cache get: touching lru, lru_counter=2992647 promotion_ts=2992645 debug 2024-02-29T12:40:00.446+0000 7392ce94c700 10 req 8249676414073575284 0.000000000s moving ceph-new-objectstore.rgw.log++script.prerequest. to cache LRU end debug 2024-02-29T12:40:00.446+0000 7392ce94c700 10 req 8249676414073575284 0.000000000s cache get: name=ceph-new-objectstore.rgw.log++script.prerequest. : hit (negative entry) debug 2024-02-29T12:40:00.446+0000 7392ce94c700 10 req 8249676414073575284 0.000000000s s3:put_obj scheduling with throttler client=2 cost=1 debug 2024-02-29T12:40:00.446+0000 7392ce94c700 10 req 8249676414073575284 0.000000000s s3:put_obj op=21RGWPutObj_ObjStore_S3 debug 2024-02-29T12:40:00.446+0000 7392ce94c700 2 req 8249676414073575284 0.000000000s s3:put_obj verifying requester debug 2024-02-29T12:40:00.446+0000 7392ce94c700 20 req 8249676414073575284 0.000000000s s3:put_obj rgw::auth::StrategyRegistry::s3_main_strategy_t: trying rgw::auth::s3::AWSAuthStrategy debug 2024-02-29T12:40:00.446+0000 7392ce94c700 20 req 8249676414073575284 0.000000000s s3:put_obj rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::S3AnonymousEngine debug 2024-02-29T12:40:00.446+0000 7392ce94c700 20 req 8249676414073575284 0.000000000s s3:put_obj rgw::auth::s3::S3AnonymousEngine denied with reason=-1 debug 2024-02-29T12:40:00.446+0000 7392ce94c700 20 req 8249676414073575284 0.000000000s s3:put_obj rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::LocalEngine debug 2024-02-29T12:40:00.446+0000 7392ce94c700 10 req 8249676414073575284 0.000000000s v4 signature format = a43d83263aa32d3b64576022f6b872d8edbbed377c20bdd3df54f53b318d6168 debug 2024-02-29T12:40:00.446+0000 7392ce94c700 10 req 8249676414073575284 0.000000000s v4 credential format = 68T4YRAG6NJ6FFVYZUXS/20240229/dummy/s3/aws4_request debug 2024-02-29T12:40:00.446+0000 7392ce94c700 10 req 8249676414073575284 0.000000000s access key id = 68T4YRAG6NJ6FFVYZUXS debug 2024-02-29T12:40:00.446+0000 7392ce94c700 10 req 8249676414073575284 0.000000000s credential scope = 20240229/dummy/s3/aws4_request debug 2024-02-29T12:40:00.446+0000 7392ce94c700 10 req 8249676414073575284 0.000000000s canonical headers format = content-length:1293 content-md5:5zRdT/pPKDa6yi2Rhh3hdQ== host:rook-ceph-rgw-ceph-new-objectstore.rook-ceph.svc x-amz-content-sha256:2aff068799ede5a30594bc5f2e49bfbf061867ecc9d85246406c3705739b7770 x-amz-date:20240229T123958Z x-amz-storage-class:STANDARDdebug 2024-02-29T12:40:00.446+0000 7392ce94c700 10 req 8249676414073575284 0.000000000s payload request hash = 2aff068799ede5a30594bc5f2e49bfbf061867ecc9d85246406c3705739b7770
debug 2024-02-29T12:40:00.446+0000 7392ce94c700 10 req 8249676414073575284 0.000000000s canonical request = PUT
/bucket-loki/fake/4544f2114b8743e6/18de51eee34%3A18de5541c17%3A41ecc98b
content-length:1293
content-md5:5zRdT/pPKDa6yi2Rhh3hdQ==
host:rook-ceph-rgw-ceph-new-objectstore.rook-ceph.svc
x-amz-content-sha256:2aff068799ede5a30594bc5f2e49bfbf061867ecc9d85246406c3705739b7770
x-amz-date:20240229T123958Z
x-amz-storage-class:STANDARD
content-length;content-md5;host;x-amz-content-sha256;x-amz-date;x-amz-storage-class
2aff068799ede5a30594bc5f2e49bfbf061867ecc9d85246406c3705739b7770
debug 2024-02-29T12:40:00.446+0000 7392ce94c700 10 req 8249676414073575284 0.000000000s canonical request hash = 968c5bfc9731fd7cd43040741f0f37123a5d57ecdce31a348b6f2ef3747e7e1f
debug 2024-02-29T12:40:00.446+0000 7392ce94c700 10 req 8249676414073575284 0.000000000s string to sign = AWS4-HMAC-SHA256
20240229T123958Z
20240229/dummy/s3/aws4_request
968c5bfc9731fd7cd43040741f0f37123a5d57ecdce31a348b6f2ef3747e7e1f
debug 2024-02-29T12:40:00.446+0000 7392ce94c700 10 req 8249676414073575284 0.000000000s delaying v4 auth
debug 2024-02-29T12:40:00.446+0000 7392ce94c700 10 req 8249676414073575284 0.000000000s date_k = f10954408bf2f18454ead405e6e43089bb82cc8ea4815fb24953ea0e175becac
debug 2024-02-29T12:40:00.446+0000 7392ce94c700 10 req 8249676414073575284 0.000000000s region_k = 69c285ac28956dabdd8c2ecd177ebee3bc0c9f8cdddc94c7e99156b0f6254719
debug 2024-02-29T12:40:00.446+0000 7392ce94c700 10 req 8249676414073575284 0.000000000s service_k = a641aee4d278b4930f1d10cb8e37580005968b5d515b6aae01981a7f1be1668d
debug 2024-02-29T12:40:00.446+0000 7392ce94c700 10 req 8249676414073575284 0.000000000s signing_k = f8273cd3f1b3861b542104219c9fca91c9f08df27a8f782b7f29c144de5d3cd7
debug 2024-02-29T12:40:00.446+0000 7392ce94c700 10 req 8249676414073575284 0.000000000s generated signature = a43d83263aa32d3b64576022f6b872d8edbbed377c20bdd3df54f53b318d6168
debug 2024-02-29T12:40:00.446+0000 7392ce94c700 15 req 8249676414073575284 0.000000000s s3:put_obj string_to_sign=AWS4-HMAC-SHA256
20240229T123958Z
20240229/dummy/s3/aws4_request
968c5bfc9731fd7cd43040741f0f37123a5d57ecdce31a348b6f2ef3747e7e1f
debug 2024-02-29T12:40:00.446+0000 7392ce94c700 15 req 8249676414073575284 0.000000000s s3:put_obj server signature=a43d83263aa32d3b64576022f6b872d8edbbed377c20bdd3df54f53b318d6168
debug 2024-02-29T12:40:00.446+0000 7392ce94c700 15 req 8249676414073575284 0.000000000s s3:put_obj client signature=a43d83263aa32d3b64576022f6b872d8edbbed377c20bdd3df54f53b318d6168
debug 2024-02-29T12:40:00.446+0000 7392ce94c700 15 req 8249676414073575284 0.000000000s s3:put_obj compare=0
debug 2024-02-29T12:40:00.446+0000 7392ce94c700 20 req 8249676414073575284 0.000000000s s3:put_obj rgw::auth::s3::LocalEngine granted access
debug 2024-02-29T12:40:00.446+0000 7392ce94c700 20 req 8249676414073575284 0.000000000s s3:put_obj rgw::auth::s3::AWSAuthStrategy granted access
debug 2024-02-29T12:40:00.446+0000 7392ce94c700 2 req 8249676414073575284 0.000000000s s3:put_obj normalizing buckets and tenants
debug 2024-02-29T12:40:00.446+0000 7392ce94c700 10 req 8249676414073575284 0.000000000s s->object=fake/4544f2114b8743e6/18de51eee34:18de5541c17:41ecc98b s->bucket=bucket-loki
debug 2024-02-29T12:40:00.446+0000 7392ce94c700 2 req 8249676414073575284 0.000000000s s3:put_obj init permissions
debug 2024-02-29T12:40:00.446+0000 7392ce94c700 20 req 8249676414073575284 0.000000000s s3:put_obj get_system_obj_state: rctx=0x73927e76c200 obj=ceph-new-objectstore.rgw.meta:root:bucket-loki state=0x567e336e5720 s->prefetch_data=0
debug 2024-02-29T12:40:00.446+0000 7392ce94c700 20 req 8249676414073575284 0.000000000s s3:put_obj cache get: touching lru, lru_counter=2992648 promotion_ts=2992646
debug 2024-02-29T12:40:00.446+0000 7392ce94c700 10 req 8249676414073575284 0.000000000s s3:put_obj moving ceph-new-objectstore.rgw.meta+root+bucket-loki to cache LRU end
debug 2024-02-29T12:40:00.446+0000 7392ce94c700 10 req 8249676414073575284 0.000000000s s3:put_obj cache get: name=ceph-new-objectstore.rgw.meta+root+bucket-loki : hit (requested=0x16, cached=0x17)
debug 2024-02-29T12:40:00.446+0000 7392ce94c700 20 req 8249676414073575284 0.000000000s s3:put_obj get_system_obj_state: s->obj_tag was set empty
debug 2024-02-29T12:40:00.446+0000 7392ce94c700 10 req 8249676414073575284 0.000000000s s3:put_obj cache get: name=ceph-new-objectstore.rgw.meta+root+bucket-loki : hit (requested=0x11, cached=0x17)
debug 2024-02-29T12:40:00.446+0000 7392ce94c700 15 req 8249676414073575284 0.000000000s s3:put_obj decode_policy Read AccessControlPolicyobc-monitoring-bucket-loki-e996d5d2-9f2d-4443-aa28-7871f96e6fb1obc-monitoring-bucket-loki-e996d5d2-9f2d-4443-aa28-7871f96e6fb1obc-monitoring-bucket-loki-e996d5d2-9f2d-4443-aa28-7871f96e6fb1obc-monitoring-bucket-loki-e996d5d2-9f2d-4443-aa28-7871f96e6fb1FULL_CONTROL
debug 2024-02-29T12:40:00.446+0000 7392ce94c700 20 req 8249676414073575284 0.000000000s s3:put_obj get_system_obj_state: rctx=0x73927e76bcf8 obj=ceph-new-objectstore.rgw.meta:users.uid:obc-monitoring-bucket-loki-e996d5d2-9f2d-4443-aa28-7871f96e6fb1 state=0x567e336e5720 s->prefetch_data=0
debug 2024-02-29T12:40:00.446+0000 7392ce94c700 20 req 8249676414073575284 0.000000000s s3:put_obj cache get: touching lru, lru_counter=2992649 promotion_ts=2992647
debug 2024-02-29T12:40:00.446+0000 7392ce94c700 10 req 8249676414073575284 0.000000000s s3:put_obj moving ceph-new-objectstore.rgw.meta+users.uid+obc-monitoring-bucket-loki-e996d5d2-9f2d-4443-aa28-7871f96e6fb1 to cache LRU end
debug 2024-02-29T12:40:00.446+0000 7392ce94c700 10 req 8249676414073575284 0.000000000s s3:put_obj cache get: name=ceph-new-objectstore.rgw.meta+users.uid+obc-monitoring-bucket-loki-e996d5d2-9f2d-4443-aa28-7871f96e6fb1 : hit (requested=0x16, cached=0x17)
debug 2024-02-29T12:40:00.446+0000 7392ce94c700 20 req 8249676414073575284 0.000000000s s3:put_obj get_system_obj_state: s->obj_tag was set empty
debug 2024-02-29T12:40:00.446+0000 7392ce94c700 20 req 8249676414073575284 0.000000000s s3:put_obj Read xattr: user.rgw.idtag
debug 2024-02-29T12:40:00.446+0000 7392ce94c700 10 req 8249676414073575284 0.000000000s s3:put_obj cache get: name=ceph-new-objectstore.rgw.meta+users.uid+obc-monitoring-bucket-loki-e996d5d2-9f2d-4443-aa28-7871f96e6fb1 : hit (requested=0x13, cached=0x17)
debug 2024-02-29T12:40:00.446+0000 7392ce94c700 2 req 8249676414073575284 0.000000000s s3:put_obj recalculating target
debug 2024-02-29T12:40:00.446+0000 7392ce94c700 2 req 8249676414073575284 0.000000000s s3:put_obj reading permissions
debug 2024-02-29T12:40:00.446+0000 7392ce94c700 2 req 8249676414073575284 0.000000000s s3:put_obj init op
debug 2024-02-29T12:40:00.446+0000 7392ce94c700 2 req 8249676414073575284 0.000000000s s3:put_obj verifying op mask
debug 2024-02-29T12:40:00.446+0000 7392ce94c700 20 req 8249676414073575284 0.000000000s s3:put_obj required_mask= 2 user.op_mask=7
debug 2024-02-29T12:40:00.446+0000 7392ce94c700 2 req 8249676414073575284 0.000000000s s3:put_obj verifying op permissions
debug 2024-02-29T12:40:00.446+0000 7392ce94c700 20 req 8249676414073575284 0.000000000s get_encryption_defaults: kms_attr_seen is 0 and sse_algorithm is
debug 2024-02-29T12:40:00.446+0000 7392ce94c700 1 req 8249676414073575284 0.000000000s op->ERRORHANDLER: err_no=-13 new_err_no=-13
debug 2024-02-29T12:40:00.454+0000 7392f799e700 20 req 7884913922912177884 0.007999661s get_system_obj_state: rctx=0x73927ecf77c0 obj=ceph-new-objectstore.rgw.log:script.postrequest. state=0x567e2d6f3060 s->prefetch_data=0
debug 2024-02-29T12:40:00.454+0000 7392f799e700 20 req 7884913922912177884 0.007999661s cache get: touching lru, lru_counter=2992650 promotion_ts=2992644
debug 2024-02-29T12:40:00.454+0000 7392f799e700 10 req 7884913922912177884 0.007999661s moving ceph-new-objectstore.rgw.log++script.postrequest. to cache LRU end
debug 2024-02-29T12:40:00.454+0000 7392f799e700 10 req 7884913922912177884 0.007999661s cache get: name=ceph-new-objectstore.rgw.log++script.postrequest. : hit (negative entry)
debug 2024-02-29T12:40:00.454+0000 7392f799e700 2 req 7884913922912177884 0.007999661s s3:put_obj op status=0
debug 2024-02-29T12:40:00.454+0000 7392f799e700 2 req 7884913922912177884 0.007999661s s3:put_obj http status=403
debug 2024-02-29T12:40:00.454+0000 7392f799e700 1 ====== req done req=0x73927ecf8730 op status=0 http_status=403 latency=0.007999661s ======
debug 2024-02-29T12:40:00.454+0000 7392f799e700 1 beast: 0x73927ecf8730: 10.2.27.193 - obc-monitoring-bucket-loki-e996d5d2-9f2d-4443-aa28-7871f96e6fb1 [29/Feb/2024:12:40:00.446 +0000] "PUT /bucket-loki/fake/1153e02a58078dcc/18de55a36af%3A18de55a38d7%3A55c416ff HTTP/1.1" 403 266 - "aws-sdk-go/1.44.315 (go1.21.3; linux; amd64)" - latency=0.007999661s
Beta Was this translation helpful? Give feedback.
All reactions