Rook 1.0.3 throws 500 errors in the ceph dashboard #3424

mabushey · 2019-07-10T00:34:36Z

Is this a bug report or feature request?

Bug Report

Deviation from expected behavior:
In 0.9.3, the dashboard worked correctly. In 1.0.3, it comes up and somewhat works, but throws 500 errors in the application:

Expected behavior:
No red 500 error boxes

How to reproduce it (minimal and precise):
common.yaml and operator.yaml from the stock examples in 1.0.3:

kubectl apply -f rook/common.yaml
kubectl apply -f rook/operator.yaml
kubectl apply -f rook/cluster.yaml

cluster.yaml:

...
  dashboard:
    enabled: true
    ssl: false
  storage:
    useAllNodes: true
    useAllDevices: false
    # Important: Directories should only be used in pre-production environments
    directories:
    - path: /var/lib/rook

Istio ingress:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: ceph
  namespace: default
spec:
  hosts:
  - "ceph.example.com"
  - "ceph.green.example.com"
  - "ceph.blue.example.com"
  gateways:
  - elb-gateway.istio-system.svc.cluster.local
  http:
  - match:
    route:
    - destination:
        port:
          number: 8443
        host: rook-ceph-mgr-dashboard.rook-ceph.svc.cluster.local



**Environment**:
* OS (e.g. from /etc/os-release):
CoreOS
* Cloud provider or hardware configuration:
AWS/Kops
* Kubernetes version (use `kubectl version`):
kubectl version                                             
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.6", GitCommit:"ab91afd7062d4240e95e51ac00a18bd58fddd365", GitTreeState:"clean", BuildDate:"2019-02-26T12:59:46Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.8", GitCommit:"4e209c9383fa00631d124c8adcc011d617339b3c", GitTreeState:"clean", BuildDate:"2019-02-28T18:40:05Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}

* Storage backend status (e.g. for Ceph use `ceph health` in the [Rook Ceph toolbox]:

kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') bash
bash: warning: setlocale: LC_CTYPE: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_COLLATE: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_MESSAGES: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_NUMERIC: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_TIME: cannot change locale (en_US.UTF-8): No such file or directory
[root@ip-10-132-3-211 /]# ceph health
HEALTH_WARN mons a,b are low on available space

The text was updated successfully, but these errors were encountered:

jordanfowler · 2019-07-10T00:41:43Z

I'm also seeing this while not using istio.

dotnwat · 2019-07-10T01:04:26Z

Please provide the operator and mgr pod logs

mabushey · 2019-07-10T18:02:05Z

2019-07-10_rook-ceph-mgr-a.log
2019-07-10_rook-ceph-operator.log

dotnwat · 2019-07-10T18:27:35Z

@mabushey thanks. a couple things.

can you confirm that the manager pod is running the updated v14.2.1 version of ceph? you should be able to do this by either looking at the beginning of the mgr pod log (the one you posted didn't have the beginning of the log), or in the tool box you can run ceph versions to get the version of all the daemons in the cluster.
this looks like the mgr is in fact running v14.2.1 but the web app running in the browser is out-of-date. could you try reloading the app, maybe in an incognito window / clearing the cache and making sure there aren't intermediary caches serving an old version. if we can confirm that's the case, then I think we can create a permanent fix on the server side to invalidate caches on upgrade.

@epuertat
@LenzGr

[10/Jul/2019:17:36:41] HTTP
Request Headers:
  REFERER: https://ceph.example.com/
  Remote-Addr: ::ffff:100.111.118.143
  Content-Length: 67
  USER-AGENT: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:67.0) Gecko/20100101 Firefox/67.0
  X-ISTIO-ATTRIBUTES: Ck8KCnNvdXJjZS51aWQSQRI/a3ViZXJuZXRlczovL2lzdGlvLWluZ3Jlc3NnYXRld2F5LTdjZmNkYmM3NzctNnJuZ3AuaXN0aW8tc3lzdGVtCkwKE2Rlc3RpbmF0aW9uLnNlcnZpY2USNRIzcm9vay1jZXBoLW1nci1kYXNoYm9hcmQucm9vay1jZXBoLnN2Yy5jbHVzdGVyLmxvY2FsClEKGGRlc3RpbmF0aW9uLnNlcnZpY2UuaG9zdBI1EjNyb29rLWNlcGgtbWdyLWRhc2hib2FyZC5yb29rLWNlcGguc3ZjLmNsdXN0ZXIubG9jYWwKTwoXZGVzdGluYXRpb24uc2VydmljZS51aWQSNBIyaXN0aW86Ly9yb29rLWNlcGgvc2VydmljZXMvcm9vay1jZXBoLW1nci1kYXNoYm9hcmQKLAodZGVzdGluYXRpb24uc2VydmljZS5uYW1lc3BhY2USCxIJcm9vay1jZXBoCjUKGGRlc3RpbmF0aW9uLnNlcnZpY2UubmFtZRIZEhdyb29rLWNlcGgtbWdyLWRhc2hib2FyZA==
  X-B3-SPANID: 2f8c99f109af7485
  X-B3-SAMPLED: 1
  X-FORWARDED-PROTO: http
  HOST: ceph.example.com
  ACCEPT: application/json, text/plain, */*
  X-B3-TRACEID: 2f8c99f109af7485
  ACCEPT-LANGUAGE: en-US,en;q=0.5
  X-FORWARDED-FOR: 100.116.234.192
  X-ENVOY-DECORATOR-OPERATION: rook-ceph-mgr-dashboard.rook-ceph.svc.cluster.local:8443/*
  Content-Type: application/json
  X-ENVOY-EXTERNAL-ADDRESS: 100.116.234.192
  X-REQUEST-ID: 580e8184-eb8d-9c4a-8b96-957e08239a71
  ACCEPT-ENCODING: gzip, deflate, br
[10/Jul/2019:17:36:41] HTTP Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cherrypy/_cprequest.py", line 656, in respond
    response.body = self.handler()
  File "/usr/lib/python2.7/site-packages/cherrypy/lib/encoding.py", line 188, in __call__
    self.body = self.oldhandler(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/cherrypy/_cptools.py", line 221, in wrap
    return self.newhandler(innerfunc, *args, **kwargs)
  File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 88, in dashboard_exception_handler
    return handler(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/cherrypy/_cpdispatch.py", line 34, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/usr/share/ceph/mgr/dashboard/controllers/__init__.py", line 649, in inner
    ret = func(*args, **kwargs)
  File "/usr/share/ceph/mgr/dashboard/controllers/__init__.py", line 842, in wrapper
    return func(*vpath, **params)
TypeError: create() got an unexpected keyword argument 'stay_signed_in'

debug 2019-07-10 17:36:41.686 7f2c32e90700  0 mgr[dashboard] [10/Jul/2019:17:36:41] HTTP Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cherrypy/_cprequest.py", line 656, in respond
    response.body = self.handler()
  File "/usr/lib/python2.7/site-packages/cherrypy/lib/encoding.py", line 188, in __call__
    self.body = self.oldhandler(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/cherrypy/_cptools.py", line 221, in wrap
    return self.newhandler(innerfunc, *args, **kwargs)
  File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 88, in dashboard_exception_handler
    return handler(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/cherrypy/_cpdispatch.py", line 34, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/usr/share/ceph/mgr/dashboard/controllers/__init__.py", line 649, in inner
    ret = func(*args, **kwargs)
  File "/usr/share/ceph/mgr/dashboard/controllers/__init__.py", line 842, in wrapper
    return func(*vpath, **params)
TypeError: create() got an unexpected keyword argument 'stay_signed_in'

[nwatkins@smash ceph]$ git checkout v14.2.1
HEAD is now at d555a9489e 14.2.1
[nwatkins@smash ceph]$ git grep stay_signed_in

mabushey · 2019-07-10T19:02:30Z

# ceph versions
{
    "mon": {
        "ceph version 14.2.1 (d555a9489eb35f84f2e1ef49b77e19da9d113972) nautilus (stable)": 3
    },
    "mgr": {
        "ceph version 14.2.1 (d555a9489eb35f84f2e1ef49b77e19da9d113972) nautilus (stable)": 1
    },
    "osd": {
        "ceph version 14.2.1 (d555a9489eb35f84f2e1ef49b77e19da9d113972) nautilus (stable)": 9
    },
    "mds": {},
    "overall": {
        "ceph version 14.2.1 (d555a9489eb35f84f2e1ef49b77e19da9d113972) nautilus (stable)": 13
    }
}

I have the exact same 500 error issue with both Chromium and Firefox, and receive the same in a Chromium Incognito session. A coworker who has never had the panel on his computer before tried it and he also got 500's.

dotnwat · 2019-07-10T19:51:54Z

do you see multiple manager pods running?

it does appear that there is some sort of caching occurring. for example, in the v13 anv v14 containers that contain the code for the clients and for the server, we can see that the string 'stay_signed_in' doesn't actually appear in newer versions.

[nwatkins@smash rook]$ docker run -it ceph/ceph:v14 grep -r "stay_signed_in" /usr > /dev/null; echo $?
1
[nwatkins@smash rook]$ docker run -it ceph/ceph:v13 grep -r "stay_signed_in" /usr > /dev/null; echo $?
0

dotnwat · 2019-07-10T20:02:37Z

was this an upgrade, or a fresh cluster setup?

mabushey · 2019-07-10T22:03:15Z

kubectl -n rook-ceph get all | grep mgr
pod/rook-ceph-mgr-a-58df9d4978-5j74k                                  1/1     Running     0          22h

service/rook-ceph-mgr             ClusterIP   100.64.164.221   <none>        9283/TCP            22h
service/rook-ceph-mgr-dashboard   ClusterIP   100.64.139.181   <none>        8443/TCP            22h


deployment.apps/rook-ceph-mgr-a      1         1         1            1           22h

replicaset.apps/rook-ceph-mgr-a-58df9d4978     1         1         1       22h

I had 0.9.3 installed previously, however I blew it all away, including the namespace and all CRD's, then ran: ansible all -i "$(aws ec2 describe-instances --region us-west-2 --query "Reservations[*].Instances[*].PrivateIpAddress" --output=text --filters "Name=tag:Name,Values=nodes.${NAME}" | awk '{ printf "%s,", $1 }')" -u core -m raw -a 'sudo rm -rf /var/lib/rook/*' before creating the new 1.0.3 cluster.

epuertat · 2019-07-11T08:40:01Z

It seems that starting with 14.1.0 (with JWT authentication) that param was removed (see https://github.com/ceph/ceph/pull/22833/files#diff-9ab52cedf031887d0bae2595e5175486R18) from the back-end as well as from the front-end (https://github.com/ceph/ceph/pull/22833/files?file-filters%5B%5D=.html#diff-38f6be359ad26179bb4a6e823b9687fbL59).

So yes, it seems to be some kind of caching, the question is where from. I don't think it's in dashboard's, as I just checked and CherryPy server is handling If-Modified. Could it be some k8s proxy/caching?

HaveFun83 · 2019-07-11T09:31:22Z

maybe related to #3106

dotnwat · 2019-07-11T18:13:35Z

@mabushey would it be possible for you to load the dashboard through port forwarding directly into the mgr pod? rook should not be performing any caching, so searching for where this cached data is being served from is the most important next step.

dotnwat · 2019-07-11T18:15:28Z

@epuertat thanks for looking at this. does the dashboard server do any sort of cache invalidation, either by adding expirations to the headers or maybe even better serving the static assets with unique names specific to the version?

epuertat · 2019-07-11T18:41:43Z

@noahdesu currently ceph-dashboard doesn't have any HTTP server-side caching in place.

Nevertheless, I just checked with my Chrome and saw that for main JS files (where all the front-end logic lies) the Network Inspector (CTRL+SHIFT+i) shows "Status Code: 200 OK (from memory cache)". It seems that, as long as we are not specifying any Cache-Control/Expires/Pragma: no-cache HTTP Header, that might result in the browser performing client-side caching.

However it should be easily invalidated with a browser hard refresh.

dotnwat · 2019-07-11T18:47:40Z

@epuertat i didn't mean server side caching, i was referring to server side mechanisms for controlling caching at intermediate levels (e.g. proxies), which even browser cache invalidation won't fix. for example

index.html -> includes assets.git-sha1.js plus no-cache on the index. i guess there are several approaches to handling this.

epuertat · 2019-07-12T12:14:41Z

@epuertat i didn't mean server side caching, i was referring to server side mechanisms for controlling caching at intermediate levels (e.g. proxies), which even browser cache invalidation won't fix. for example

index.html -> includes assets.git-sha1.js plus no-cache on the index. i guess there are several approaches to handling this.

Yeah, those headers I mention (but not present in Dashboard static files) might influence intermediate HTTP caching. Cherrypy by default provides Last-Modified header (based on mtime). According to what I understand from RFC7234, default caching behaviour when no cache control headers are present is:

Although caching is an entirely OPTIONAL feature
of HTTP, it can be assumed that reusing a cached response is
desirable and that such reuse is the default behavior when no
requirement or local configuration prevents it

However RFC7234 states:

If the response has a Last-Modified header field (Section 2.2 of
[RFC7232]), caches are encouraged to use a heuristic expiration value
that is no more than some fraction of the interval since that time.
A typical setting of this fraction might be 10%.

So I'm afraid this is kind of undefined behaviour (or left to the implementer). I'll file an issue in ceph-dashboard for explicitly setting Cache-Control header.

epuertat · 2019-07-12T12:24:53Z

Ceph Tracker#40754: mgr/dashboard: configure HTTP caching for static files (JS/Angular)

allen13 · 2019-07-12T15:56:16Z

Also seeing this issue in Openshift 3.11. I guess the best way to handle this is to just use ceph:v13 until this issue is resolved?

dotnwat · 2019-07-12T15:58:53Z

If you are seeing this as well it'd be great to identify where the caching is occurring, whether it is in the browser or in some intermediate location like a proxy or cdn. clearing the cache at that level is the workaround. @allen13 are you able to verify this?

allen13 · 2019-07-12T16:03:49Z

I cleared browser cache so the only other place it could be is in the Openshift controlled Haproxy routes. We are running on baremetal so there isn't anything else that could interfere.

dotnwat · 2019-07-12T16:11:01Z

DId you try a different browser? Can you verify if it is being cached in the proxy? The other thing I've noticed is that if I view the source for the page, click on the javascript assets link, then refresh, it sometimes breaks loose something and starts using the latest version.

…

On Fri, Jul 12, 2019 at 9:03 AM allen13 ***@***.***> wrote: I cleared browser cache so the only other place it could be is in the Openshift controlled Haproxy routes. We are running on baremetal so there isn't anything else that could interfere. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3424?email_source=notifications&email_token=AAB3F4KQ5MF5N3C7A432IVTP7CTONA5CNFSM4H7KLLWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZ2FVRY#issuecomment-510941895>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAB3F4ODJYVW5VFIK265ULLP7CTONANCNFSM4H7KLLWA> .

dotnwat · 2019-07-12T16:12:31Z

The check I was hoping someone would do is to serve the dashboard directly via port forwarding into the mgr pod. One could also GET the pages using curl or something like that to rule out browser caching. On Fri, Jul 12, 2019 at 9:11 AM Noah Watkins <notifications@github.com> wrote:

…

DId you try a different browser? Can you verify if it is being cached in the proxy? The other thing I've noticed is that if I view the source for the page, click on the javascript assets link, then refresh, it sometimes breaks loose something and starts using the latest version. On Fri, Jul 12, 2019 at 9:03 AM allen13 ***@***.***> wrote: > I cleared browser cache so the only other place it could be is in the > Openshift controlled Haproxy routes. We are running on baremetal so there > isn't anything else that could interfere. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > < #3424?email_source=notifications&email_token=AAB3F4KQ5MF5N3C7A432IVTP7CTONA5CNFSM4H7KLLWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZ2FVRY#issuecomment-510941895 >, > or mute the thread > < https://github.com/notifications/unsubscribe-auth/AAB3F4ODJYVW5VFIK265ULLP7CTONANCNFSM4H7KLLWA > > . > — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3424?email_source=notifications&email_token=AAB3F4LKFVYGBFCUC3P776DP7CUJNA5CNFSM4H7KLLWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZ2GHVY#issuecomment-510944215>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAB3F4I5X22UJ226MBYTY2TP7CUJNANCNFSM4H7KLLWA> .

mabushey · 2019-07-15T19:05:52Z

One could also GET the pages using curl or something like that to rule out browser caching.

Doesn't using a browser on a different computer that has never pulled up the dashboard before meet this criteria?

dotnwat · 2019-07-15T20:30:43Z

yeh, it should. it won't rule out the proxy caching.

epuertat · 2019-07-16T11:03:18Z

After reviewing this with other ceph-dashboard folks (@ricardoasmarques), it seems that the JS files have per-build unique names (hence no way to be cached across versions). So only index.html could be causing this issue... but at least my browser (Chrome) is sending Cache-Control: max-age=0 on GET /, so that should force intermediate caches to revalidate their copies (via If-Modified, which Cherrypy properly deals with).

sebastian-philipp · 2019-07-16T15:52:58Z

One idea is that there are multiple unrelated issues causing those internal server errors. So, having a mgr log would make things more clear. there is a bug that is going to be fixed in 14.2.2

dotnwat · 2019-07-16T15:58:20Z

@sebastian-philipp did you see #3424 (comment) ? it looks as though the 500 error is caused by a browser client submitted requests from an older version than the server that is running.

dotnwat · 2019-07-16T16:00:44Z

thanks @epuertat. could be an intermediate layer just ignoring caching directives.

@mabushey @jordanfowler @allen13 if anyone is able to run a few more exploratory experiments that'd be great.

sebastian-philipp · 2019-07-16T16:03:03Z

@sebastian-philipp did you see #3424 (comment) ? it looks as though the 500 error is caused by a browser client submitted requests from an older version than the server that is running.

you're right, of course. But I've seen too many different bugs in this tracker merged together in different issues called "Dashboard throws 500 error"

mabushey · 2019-07-16T20:47:11Z

Exact same issue using port forwarding:
kubectl -n rook-ceph port-forward service/rook-ceph-mgr-dashboard 8443:8443 and
kubectl -n rook-ceph port-forward pod/rook-ceph-mgr-a-58df9d4978-5j74k 8443:8443

dotnwat · 2019-07-16T20:53:22Z

thanks @mabushey can you confirm that with port-forwarding the manager is logging the same instance of the error, and that it isn't a separate 500 issue?

mabushey · 2019-07-17T22:12:26Z

::ffff:127.0.0.1 - - [17/Jul/2019:22:06:28] "GET /api/prometheus/notifications?from=last HTTP/1.1" 200 22 "http://localhost:8443/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0"
debug 2019-07-17 22:06:29.002 7f918adda700  0 mgr[rook] Completion <read op> threw an exception:
Traceback (most recent call last):
  File "/usr/share/ceph/mgr/rook/module.py", line 191, in wait
    c.execute()
  File "/usr/share/ceph/mgr/rook/module.py", line 57, in execute
    self._result = self.cb()
  File "/usr/share/ceph/mgr/rook/module.py", line 132, in <lambda>
    return RookReadCompletion(lambda: f(*args, **kwargs))
  File "/usr/share/ceph/mgr/rook/module.py", line 348, in describe_service
    assert service_type in ("mds", "osd", "mgr", "mon", "nfs", None), service_type + " unsupported"
AssertionError: iscsi unsupported
[17/Jul/2019:22:06:29] HTTP
Request Headers:
  AUTHORIZATION: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJjZXBoLWRhc2hib2FyZCIsImlhdCI6MTU2MzQwMTE4OCwidXNlcm5hbWUiOiJhZG1pbiIsImp0aSI6IjVmM2QzNjVhLTk5OGYtNGUxNC1hNGRiLWQ5YmMxMjViNTg5YSIsImV4cCI6MTU2MzQyOTk4OH0.x_EHoJKh7Hr2z-MMsbO5MjflbIfr2Wmpwv6lehl1uiA
  REFERER: http://localhost:8443/
  HOST: localhost:8443
  CONNECTION: keep-alive
  Remote-Addr: ::ffff:127.0.0.1
  ACCEPT: application/json, text/plain, */*
  USER-AGENT: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0
  ACCEPT-LANGUAGE: en-US,en;q=0.5
  DNT: 1
  ACCEPT-ENCODING: gzip, deflate
[17/Jul/2019:22:06:29] HTTP Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cherrypy/_cprequest.py", line 656, in respond
    response.body = self.handler()
  File "/usr/lib/python2.7/site-packages/cherrypy/lib/encoding.py", line 188, in __call__
    self.body = self.oldhandler(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/cherrypy/_cptools.py", line 221, in wrap
    return self.newhandler(innerfunc, *args, **kwargs)
  File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 88, in dashboard_exception_handler
    return handler(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/cherrypy/_cpdispatch.py", line 34, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/usr/share/ceph/mgr/dashboard/controllers/__init__.py", line 649, in inner
    ret = func(*args, **kwargs)
  File "/usr/share/ceph/mgr/dashboard/controllers/health.py", line 197, in minimal
    return self.health_minimal.all_health()
  File "/usr/share/ceph/mgr/dashboard/controllers/health.py", line 62, in all_health
    result['iscsi_daemons'] = self.iscsi_daemons()
  File "/usr/share/ceph/mgr/dashboard/controllers/health.py", line 126, in iscsi_daemons
    gateways = IscsiGatewaysConfig.get_gateways_config()['gateways']
  File "/usr/share/ceph/mgr/dashboard/services/iscsi_config.py", line 93, in get_gateways_config
    for instance in instances:
TypeError: 'NoneType' object is not iterable

debug 2019-07-17 22:06:29.023 7f918adda700  0 mgr[dashboard] [17/Jul/2019:22:06:29] HTTP Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cherrypy/_cprequest.py", line 656, in respond
    response.body = self.handler()
  File "/usr/lib/python2.7/site-packages/cherrypy/lib/encoding.py", line 188, in __call__
    self.body = self.oldhandler(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/cherrypy/_cptools.py", line 221, in wrap
    return self.newhandler(innerfunc, *args, **kwargs)
  File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 88, in dashboard_exception_handler
    return handler(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/cherrypy/_cpdispatch.py", line 34, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/usr/share/ceph/mgr/dashboard/controllers/__init__.py", line 649, in inner
    ret = func(*args, **kwargs)
  File "/usr/share/ceph/mgr/dashboard/controllers/health.py", line 197, in minimal
    return self.health_minimal.all_health()
  File "/usr/share/ceph/mgr/dashboard/controllers/health.py", line 62, in all_health
    result['iscsi_daemons'] = self.iscsi_daemons()
  File "/usr/share/ceph/mgr/dashboard/controllers/health.py", line 126, in iscsi_daemons
    gateways = IscsiGatewaysConfig.get_gateways_config()['gateways']
  File "/usr/share/ceph/mgr/dashboard/services/iscsi_config.py", line 93, in get_gateways_config
    for instance in instances:
TypeError: 'NoneType' object is not iterable

debug 2019-07-17 22:06:29.023 7f918adda700  0 mgr[dashboard] [::ffff:127.0.0.1:56278] [GET] [500] [0.267s] [admin] [1.6K] /api/health/minimal
debug 2019-07-17 22:06:29.023 7f918adda700  0 mgr[dashboard] ['{"status": "500 Internal Server Error", "version": "3.2.2", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "traceback": "Traceback (most recent call last):\\n  File \\"/usr/lib/python2.7/site-packages/cherrypy/_cprequest.py\\", line 656, in respond\\n    response.body = self.handler()\\n  File \\"/usr/lib/python2.7/site-packages/cherrypy/lib/encoding.py\\", line 188, in __call__\\n    self.body = self.oldhandler(*args, **kwargs)\\n  File \\"/usr/lib/python2.7/site-packages/cherrypy/_cptools.py\\", line 221, in wrap\\n    return self.newhandler(innerfunc, *args, **kwargs)\\n  File \\"/usr/share/ceph/mgr/dashboard/services/exception.py\\", line 88, in dashboard_exception_handler\\n    return handler(*args, **kwargs)\\n  File \\"/usr/lib/python2.7/site-packages/cherrypy/_cpdispatch.py\\", line 34, in __call__\\n    return self.callable(*self.args, **self.kwargs)\\n  File \\"/usr/share/ceph/mgr/dashboard/controllers/__init__.py\\", line 649, in inner\\n    ret = func(*args, **kwargs)\\n  File \\"/usr/share/ceph/mgr/dashboard/controllers/health.py\\", line 197, in minimal\\n    return self.health_minimal.all_health()\\n  File \\"/usr/share/ceph/mgr/dashboard/controllers/health.py\\", line 62, in all_health\\n    result[\'iscsi_daemons\'] = self.iscsi_daemons()\\n  File \\"/usr/share/ceph/mgr/dashboard/controllers/health.py\\", line 126, in iscsi_daemons\\n    gateways = IscsiGatewaysConfig.get_gateways_config()[\'gateways\']\\n  File \\"/usr/share/ceph/mgr/dashboard/services/iscsi_config.py\\", line 93, in get_gateways_config\\n    for instance in instances:\\nTypeError: \'NoneType\' object is not iterable\\n"}']
::ffff:127.0.0.1 - - [17/Jul/2019:22:06:29] "GET /api/health/minimal HTTP/1.1" 500 1646 "http://localhost:8443/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0"
debug 2019-07-17 22:06:29.048 7f918b5db700  0 mgr[rook] Completion <read op> threw an exception:
Traceback (most recent call last):
  File "/usr/share/ceph/mgr/rook/module.py", line 191, in wait
    c.execute()
  File "/usr/share/ceph/mgr/rook/module.py", line 57, in execute
    self._result = self.cb()
  File "/usr/share/ceph/mgr/rook/module.py", line 132, in <lambda>
    return RookReadCompletion(lambda: f(*args, **kwargs))
  File "/usr/share/ceph/mgr/rook/module.py", line 348, in describe_service
    assert service_type in ("mds", "osd", "mgr", "mon", "nfs", None), service_type + " unsupported"
AssertionError: iscsi unsupported

dotnwat · 2019-07-17T22:26:20Z

@mabushey looks like your cache has cleared. this error is a known error with a fix in the pipeline: #3106. That issue has a workaround, but I think the fix should be landing very soon. More updates towards the end of the discussion.

mabushey · 2019-07-18T00:27:04Z

@noahdesu

Yes, that workaround worked. Thank you.

mabushey added the bug label Jul 10, 2019

sebastian-philipp added the ceph-dashboard label Jul 16, 2019

dotnwat closed this as completed Jul 18, 2019

epuertat mentioned this issue Sep 17, 2019

ceph-mgr errors with KeyError mimic in rook 1.1.0 and Nautilus 14.2.3 #3873

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rook 1.0.3 throws 500 errors in the ceph dashboard #3424

Rook 1.0.3 throws 500 errors in the ceph dashboard #3424

mabushey commented Jul 10, 2019

jordanfowler commented Jul 10, 2019

dotnwat commented Jul 10, 2019

mabushey commented Jul 10, 2019

dotnwat commented Jul 10, 2019 •

edited

mabushey commented Jul 10, 2019 •

edited

dotnwat commented Jul 10, 2019

dotnwat commented Jul 10, 2019

mabushey commented Jul 10, 2019 •

edited

epuertat commented Jul 11, 2019

HaveFun83 commented Jul 11, 2019

dotnwat commented Jul 11, 2019

dotnwat commented Jul 11, 2019

epuertat commented Jul 11, 2019

dotnwat commented Jul 11, 2019

epuertat commented Jul 12, 2019

epuertat commented Jul 12, 2019

allen13 commented Jul 12, 2019

dotnwat commented Jul 12, 2019

allen13 commented Jul 12, 2019

dotnwat commented Jul 12, 2019 via email

dotnwat commented Jul 12, 2019 via email

mabushey commented Jul 15, 2019

dotnwat commented Jul 15, 2019

epuertat commented Jul 16, 2019

sebastian-philipp commented Jul 16, 2019

dotnwat commented Jul 16, 2019

dotnwat commented Jul 16, 2019

sebastian-philipp commented Jul 16, 2019

mabushey commented Jul 16, 2019 •

edited

dotnwat commented Jul 16, 2019

mabushey commented Jul 17, 2019

dotnwat commented Jul 17, 2019 •

edited

mabushey commented Jul 18, 2019

Rook 1.0.3 throws 500 errors in the ceph dashboard #3424

Rook 1.0.3 throws 500 errors in the ceph dashboard #3424

Comments

mabushey commented Jul 10, 2019

jordanfowler commented Jul 10, 2019

dotnwat commented Jul 10, 2019

mabushey commented Jul 10, 2019

dotnwat commented Jul 10, 2019 • edited

mabushey commented Jul 10, 2019 • edited

dotnwat commented Jul 10, 2019

dotnwat commented Jul 10, 2019

mabushey commented Jul 10, 2019 • edited

epuertat commented Jul 11, 2019

HaveFun83 commented Jul 11, 2019

dotnwat commented Jul 11, 2019

dotnwat commented Jul 11, 2019

epuertat commented Jul 11, 2019

dotnwat commented Jul 11, 2019

epuertat commented Jul 12, 2019

epuertat commented Jul 12, 2019

allen13 commented Jul 12, 2019

dotnwat commented Jul 12, 2019

allen13 commented Jul 12, 2019

dotnwat commented Jul 12, 2019 via email

dotnwat commented Jul 12, 2019 via email

mabushey commented Jul 15, 2019

dotnwat commented Jul 15, 2019

epuertat commented Jul 16, 2019

sebastian-philipp commented Jul 16, 2019

dotnwat commented Jul 16, 2019

dotnwat commented Jul 16, 2019

sebastian-philipp commented Jul 16, 2019

mabushey commented Jul 16, 2019 • edited

dotnwat commented Jul 16, 2019

mabushey commented Jul 17, 2019

dotnwat commented Jul 17, 2019 • edited

mabushey commented Jul 18, 2019

dotnwat commented Jul 10, 2019 •

edited

mabushey commented Jul 10, 2019 •

edited

mabushey commented Jul 10, 2019 •

edited

mabushey commented Jul 16, 2019 •

edited

dotnwat commented Jul 17, 2019 •

edited