Ceph object store dashboard cannot be enabled

[meirhazonAnyVision]

Is this a bug report or feature request? BUG

• Bug Report

Deviation from expected behavior:
No matter what I try I cannot make the object storage part in the dashboard work.
The Object Gateway seems to be working properly, buckets, files etc.
The other parts in the dashboard also work properly.

I am getting,
500 - Internal Server Error
RGW rest api failed request with status code 403

Expected behavior:
It should present object, buckets etc.

How to reproduce it (minimal and precise):

We are using Rook Ceph v0.9.3 (Ceph image: v13.2.4-20190109) on kubernetes v1.13.5 Ubuntu 18.4

Start the dashboard, follow dashboard conf http://docs.ceph.com/docs/mimic/mgr/dashboard/ and try it out

Environment:

• OS (e.g. from /etc/os-release):
  Ubuntu 18.4 LTS

• Kernel (e.g. uname -a):
  Linux master1 4.15.0-1029-gcp Move ceph manager from castled to cephmgr package #31-Ubuntu SMP Thu Mar 21 09:40:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

• Cloud provider or hardware configuration:
  GCP/ Local hardware - the same issue
  Local hardware, 30Cores+64GB RAM

• Rook version (use rook version inside of a Rook Pod):
  0.9.3
  [root@node1 /]# rook version
  rook: v0.9.3

• Kubernetes version (use kubectl version):
  13.05
  14.01

• Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift):
  Native

• Storage backend status (e.g. for Ceph use ceph health in the Rook Ceph toolbox):
  [root@node1 /]# ceph health
  HEALTH_OK

[travisn]

@mattbenjamin @mkogan1 What does it take to enable the object tab in the dashboard? I also tried recently without success so I'm wondering if we're missing something in the rook configuration that would allow it to be enabled. Could someone on rgw team try it out with rook and see what we're missing? thanks

[mkogan1]

@travisn , I follow the instruction at http://docs.ceph.com/docs/master/mgr/dashboard/#enabling-the-object-gateway-management-frontend to enable the object tab in the dashboard

[muhlba91]

I have a similar issue. First, I received 500 errors and after going through the instructions on the Ceph site (the ones @mkogan1 posted) I receive 404 errors.

[mkogan1]

@muhlba91, as far as I can remember for vstart cluster I just set the keys to the keys that vstart has assigned to the testsid user with the commands below and then logout and re-login to the dashboard

ceph dashboard set-rgw-api-access-key <access_key>
ceph dashboard set-rgw-api-secret-key <secret_key>

[travisn]

@mkogan1 In another attempt with the 14.2.0 client tools and configuring a 14.2.0 cluster with rgw, I still can't get it to load. I first get a 403 before I follow the guide. After setting up the rgw user, I get a 500.

Here are the settings I applied after configuring rgw through rook:

radosgw-admin user create --uid=system-user --display-name=“my system user” --system
ceph dashboard set-rgw-api-access-key ES53B0PXUADXZTVYWIGB
ceph dashboard set-rgw-api-secret-key pYS6l4ckfm8WhK2RucxBPSp30GNLvbSfjkpoaoCh
ceph dashboard set-rgw-api-scheme https
ceph dashboard set-rgw-api-ssl-verify False

Here is the full mgr log, which shows a crash when the 500 error is shown in the dashboard.
mgr.log

The mgr log shows the following stack:

[23/Apr/2019:21:55:30] HTTP 
Request Headers:
  AUTHORIZATION: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJjZXBoLWRhc2hib2FyZCIsImlhdCI6MTU1NjA1NjMxMiwidXNlcm5hbWUiOiJhZG1pbiIsImp0aSI6IjhiMjAwNTA5LTU5MzMtNDUxMC1hOTUzLTVmNWQxNjdmZDU3NCIsImV4cCI6MTU1NjA4NTExMn0.qnVJ7zVaYRF6VI_CMdcd1o2OX62-xII04SKEm8ZcJo8
  REFERER: https://192.168.99.101:32232/
  HOST: 192.168.99.101:32232
  CONNECTION: keep-alive
  Remote-Addr: 172.17.0.1
  ACCEPT: application/json, text/plain, */*
  USER-AGENT: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36
  COOKIE: session_id=4439a79211615bbb69bfbacb107e793e45d20b2d
  ACCEPT-LANGUAGE: en-US,en;q=0.9
  ACCEPT-ENCODING: gzip, deflate, br
[23/Apr/2019:21:55:30] HTTP Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cherrypy/_cprequest.py", line 656, in respond
    response.body = self.handler()
  File "/usr/lib/python2.7/site-packages/cherrypy/lib/encoding.py", line 188, in __call__
    self.body = self.oldhandler(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/cherrypy/_cptools.py", line 221, in wrap
    return self.newhandler(innerfunc, *args, **kwargs)
  File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 88, in dashboard_exception_handler
    return handler(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/cherrypy/_cpdispatch.py", line 34, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/usr/share/ceph/mgr/dashboard/controllers/__init__.py", line 649, in inner
    ret = func(*args, **kwargs)
  File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 26, in status
    instance = RgwClient.admin_instance()
  File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 258, in admin_instance
    return RgwClient.instance(RgwClient._SYSTEM_USERID)
  File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 236, in instance
    RgwClient._load_settings()
  File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 226, in _load_settings
    Settings.RGW_API_SECRET_KEY)
  File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 280, in __init__
    RgwClient._load_settings()
  File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 226, in _load_settings
    Settings.RGW_API_SECRET_KEY)
  File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 280, in __init__
...
RuntimeError: maximum recursion depth exceeded in cmp

[meirhazonAnyVision]

Hi. I did so too about 5 times and it didn’t work. Can you please share the commands that you used? Thanks so much. Meir

…

Sent from my iPhone

On 23 Apr 2019, at 18:49, Daniel Mühlbachler ***@***.***> wrote: I have a similar issue. First, I received 500 errors and after going through the instructions on the Ceph site (the ones @mkogan1 posted) I receive 404 errors. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[meirhazonAnyVision]

Hi. Please believe my when I say that I’ve done it. I will do it again next week and email you with screen shots. Done it more than once. Didn’t work. Thanks.

…

Sent from my iPhone

On 24 Apr 2019, at 1:03, Travis Nielsen ***@***.***> wrote: @mkogan1 In another attempt with the 14.2.0 client tools and configuring a 14.2.0 cluster with rgw, I still can't get it to load. I first get a 403 before I follow the guide. After setting up the rgw user, I get a 500. Here are the settings I applied after configuring rgw through rook: radosgw-admin user create --uid=system-user --display-name=“my system user” --system ceph dashboard set-rgw-api-access-key ES53B0PXUADXZTVYWIGB ceph dashboard set-rgw-api-secret-key pYS6l4ckfm8WhK2RucxBPSp30GNLvbSfjkpoaoCh ceph dashboard set-rgw-api-scheme https ceph dashboard set-rgw-api-ssl-verify False Here is the full mgr log, which shows a crash when the 500 error is shown in the dashboard. mgr.log The mgr log shows the following stack: [23/Apr/2019:21:55:30] HTTP Request Headers: AUTHORIZATION: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJjZXBoLWRhc2hib2FyZCIsImlhdCI6MTU1NjA1NjMxMiwidXNlcm5hbWUiOiJhZG1pbiIsImp0aSI6IjhiMjAwNTA5LTU5MzMtNDUxMC1hOTUzLTVmNWQxNjdmZDU3NCIsImV4cCI6MTU1NjA4NTExMn0.qnVJ7zVaYRF6VI_CMdcd1o2OX62-xII04SKEm8ZcJo8 REFERER: https://192.168.99.101:32232/ HOST: 192.168.99.101:32232 CONNECTION: keep-alive Remote-Addr: 172.17.0.1 ACCEPT: application/json, text/plain, */* USER-AGENT: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36 COOKIE: session_id=4439a79211615bbb69bfbacb107e793e45d20b2d ACCEPT-LANGUAGE: en-US,en;q=0.9 ACCEPT-ENCODING: gzip, deflate, br [23/Apr/2019:21:55:30] HTTP Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cherrypy/_cprequest.py", line 656, in respond response.body = self.handler() File "/usr/lib/python2.7/site-packages/cherrypy/lib/encoding.py", line 188, in __call__ self.body = self.oldhandler(*args, **kwargs) File "/usr/lib/python2.7/site-packages/cherrypy/_cptools.py", line 221, in wrap return self.newhandler(innerfunc, *args, **kwargs) File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 88, in dashboard_exception_handler return handler(*args, **kwargs) File "/usr/lib/python2.7/site-packages/cherrypy/_cpdispatch.py", line 34, in __call__ return self.callable(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/dashboard/controllers/__init__.py", line 649, in inner ret = func(*args, **kwargs) File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 26, in status instance = RgwClient.admin_instance() File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 258, in admin_instance return RgwClient.instance(RgwClient._SYSTEM_USERID) File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 236, in instance RgwClient._load_settings() File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 226, in _load_settings Settings.RGW_API_SECRET_KEY) File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 280, in __init__ RgwClient._load_settings() File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 226, in _load_settings Settings.RGW_API_SECRET_KEY) File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 280, in __init__ ... RuntimeError: maximum recursion depth exceeded in cmp — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[yanchicago]

Had similar issue and put my logs at #2722

[travisn]

@meirhazonAnyVision The steps in #3255 worked for me. Could you also verify?

I would suggest the following to automate this when rook configures an object store:

• Create a system object user (call it rook-system-<store>, for example rook-system-my-store)
• Configure the dashboard module with the needed variables:

ceph dashboard set-rgw-api-user-id rook-system-my-store
ceph dashboard set-rgw-api-access-key <access-key>
ceph dashboard set-rgw-api-secret-key <secret-key>

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[colilo]

[Resolved]
Kubernets:

NAME             STATUS   ROLES    AGE   VERSION
ceph-k8s-node1   Ready    master   18d   v1.15.3
ceph-k8s-node2   Ready    <none>   18d   v1.15.3
ceph-k8s-node3   Ready    <none>   18d   v1.15.3

Rook:

v1.1.0-beta.1
rook/ceph:v1.1.0-beta.1

Ceph Image:

ceph/ceph:v14.2.2-20190826

I changed the dashboard config ssl to false and port to 7000, and created the rook ceph cluster as order:

1. kubectl create -f common.yaml
2. kubectl create -f operator.yaml
3. kubectl create -f cluster.yaml
4. kubectl create -f dashboard-external-http.yaml
5. kubectl create -f object.yaml
6. kubectl create -f object-user.yaml
7. kubectl create -f rgw-external.yaml
8. kubectl create -f toolbox.yaml

And then I executed the commands

# Access toolbox CLI:
kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') bash

# Enable system flag on the user:
radosgw-admin user modify --uid=my-user --system

# Provide the user credentials:
ceph dashboard set-rgw-api-user-id my-user
ceph dashboard set-rgw-api-access-key <access-key>
ceph dashboard set-rgw-api-secret-key <secret-key>

The log of ceph dashboard set-rgw-api-*

debug 2019-09-09 06:31:46.350 7f0a8275e700 0 log_channel(audit) log [DBG] : from='client.7260 -' entity='client.admin' cmd=[{"prefix": "dashboard set-rgw-api-user-id", "target": ["mgr", ""], "value": "my-user"}]: dispatch

debug 2019-09-09 06:32:03.146 7f0a8275e700 0 log_channel(audit) log [DBG] : from='client.7269 -' entity='client.admin' cmd=[{"prefix": "dashboard set-rgw-api-access-key", "target": ["mgr", ""], "value": "178GT1BC5CP6FYKNM02K"}]: dispatch

debug 2019-09-09 06:32:24.710 7f0a8275e700 0 log_channel(audit) log [DBG] : from='client.7296 -' entity='client.admin' cmd=[{"prefix": "dashboard set-rgw-api-secret-key", "target": ["mgr", ""], "value": "hcFuuMBQnPSL2AAqClunQLzvSJY4COoYjJFA15Ed"}]: dispatch

re-login dashboard and click Object Gateway --> Daemons

The response was

Information
No RGW found
Please consult the documentation on how to configure and enable the Object Gateway management functionality.

Would you help to point where I was wrong?

After restart dashboard by command

$ ceph mgr module disable dashboard
$ ceph mgr module enable dashboard

The web page of Object Gateway can be accessed.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.

[ctrahey]

After futzing around for a while, I kept getting max-recursion errors when accessing the object gateway section of the dashboard. Fixing it for me was a matter of running every ceph dashboard reset-rgw-* command to clean up, then starting over. Here is a handy shortcut:

From my ceph-tools container:

ceph dashboard --help | grep reset-rgw | awk '{print $2}' | xargs -n 1 ceph dashboard
ceph dashboard set-rgw-api-user-id <your-user>
ceph dashboard set-rgw-api-access-key <your-access-key>
ceph dashboard set-rgw-api-secret-key <your-secret-key>

Logging out and back into the web UI seemed to be necessary.

At the time of this writing, the commands run by the first line:

ceph dashboard --help | grep reset-rgw | awk '{print $2}' | xargs -n 1 echo ceph dashboard 
ceph dashboard reset-rgw-api-access-key
ceph dashboard reset-rgw-api-admin-resource
ceph dashboard reset-rgw-api-host
ceph dashboard reset-rgw-api-port
ceph dashboard reset-rgw-api-scheme
ceph dashboard reset-rgw-api-secret-key
ceph dashboard reset-rgw-api-ssl-verify
ceph dashboard reset-rgw-api-user-id

[travisn]

@ctrahey Note that the object dashboard will be enabled in rook v1.4.1 after we get the fix in #4676 backported, then there will be no need to enable this manually.

[stepanovmm1992]

@ctrahey Thank you a lot!!! I setup Ceph and three days trying install RGW and have RGW REST API failed request with status code 403 (b'{"Code":"InvalidAccessKeyId","RequestId":"tx000000000000000000017-005ecac06c' b'-e349-eu-west-1","HostId":"e349-eu-west-1-default"}')! Tonight I reset all and run command again and it run! I want child from you!!!))))