mgr/dashboard_v2: Initial submission of a web-based management UI (replacement for the existing dashboard) #20103

LenzGr · 2018-01-24T16:28:59Z

The original Ceph Manager Dashboard as shipped with Ceph "Luminous" started out as a simple read-only view into various run-time information and performance data of a Ceph cluster, without authentication or any administrative functionality.

However, there is a growing demand for adding more web-based management capabilities, to make it easier for administrators that prefer a WebUI over the command line.

This module aims at becoming a "plug-in-replacement" for the existing dashboard, to provide the foundation for creating a native, built-in web based monitoring and administration application for Ceph.

The code and architecture of this module is derived from and inspired by the openATTIC Ceph management and monitoring tool.

The WebUI implementation is built using Angular and TypeScript. See the screen shot gallery for some impressions of the look and feel of the UI.

A lof of the backend code (based on CherryPy has been ported from the original dashboard, and will be extended in the future to support more than just read-only functionality. See the HACKING.rst file for the developer documentation.

The development is actively driven by the team behind openATTIC at SUSE. The intention is to reuse existing openATTIC code where possible, while adapting it to the different environment.

This PR represents the first milestone (feature parity with the original dashboard, plus a simple authentication mechanism).

The porting and migration of the existing openATTIC and dashboard functionality will be done in future pull requests.

runsisi · 2018-01-25T01:22:26Z

why not merge those RESTful thing from dashboard[1] into restful[2], and then start a new React/Vue/AngularJS SPA, this way everybody could reuse the RESTful API...

i saw CherryPy[1]/Pecan[2]/Flask[3] were used by different modules, why not choose one and reduce our dependencies on different Python packages?

@liewegas @jcsp what's your opinion? thanks!

[1] https://github.com/ceph/ceph/tree/master/src/pybind/mgr/dashboard
[2] https://github.com/ceph/ceph/tree/master/src/pybind/mgr/restful
[3] https://github.com/ceph/ceph/blob/master/src/pybind/ceph_rest_api.py

LenzGr · 2018-01-25T10:15:17Z

@runsisi thank you for your feedback. As far as I understand it, the current dashboard does not use a REST API at all, the Python backend code performs the data collection and most of rendering here.

The original ceph_rest_api.py is primarily a REST wrapper around the librados Python bindings, so it provides less functionality than what is available to a module running in the Mgr context (and what the dashboard_v2 will require).

In our case, the dashboard_v2 REST API is not intended to become an "official/stable" REST API endpoint - it's primary purpose is to serve the needs of the WebUI. Right now, we try to create a REST API that closely resembles the one used by openATTIC, in order to minimize the porting effort on our side.

The restful manager module might be an option and a future convergence point. However, as we understand it, this module is meant to be used by external consumers that need a stable API. Using restful would make it much more difficult to change our API in the future.

Regarding Flask vs CherryPy vs Pecan: AFAIR, @jcsp already commented on the subject of using different Python modules on the ceph-devel mailing list. We chose CherryPy primarily because it is used by the existing dashboard and it fits our needs.

However, we're still quite early in the process of the dashboard_v2 backend. If there are strong arguments for it, we'd be fine with switching to another framework at some point. For now and the sake of making progress, we'd rather stick to using CherryPy, though.

liewegas · 2018-02-28T18:26:25Z

jenkins render docs

ceph-jenkins · 2018-02-28T18:37:10Z

Doc render available at http://docs.ceph.com/ceph-prs/20103/

LenzGr · 2018-03-01T12:09:18Z

OK, we've now concluded our first milestone (feature parity with the existing dashboard) and cleaned up the commit history by rebasing/squashing many commits. We'd be grateful for reviews/comments!

rjfd · 2018-03-01T14:18:20Z

dashboard_v2 teuthology suite results:

http://pulpito.ceph.com/rdias-2018-03-01_13:56:36-rados:mgr-wip-mgr-dashboard_v2-distro-basic-smithi/

smithfarm · 2018-03-01T14:46:36Z

@rdias While you're at it, can you schedule a /2000 rados run?

alfredodeza · 2018-03-01T14:55:50Z

I agree with @runsisi about the multiple different frameworks used, and would love to see reducing them to just 1. Most everything in the Ceph ecosystem (build services, ceph-installer, shaman, etc...) is Pecan based, which means there is a lot of know-how in our team, and I am strongly in favor of using it.

Having browsed through the initial PR, it seems unreasonable to ask for that transition from the get-go. I would eventually like to have an open discussion to understand what the framework of choice should be moving forward, and what that means for other frameworks being used in the source tree.

As an in-tree project which is choosing Angular, how would it look if another dashboard/app chooses React? There is value in uniformity of frameworks.

Again, not meaning to derail the PR moving forward. Hope the framework discussion can be picked up later!

Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

Also: * Disabled API tests in tox. * No longer start a vstart.sh cluster in tox. Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

* Added new CMake flag `WITH_MGR_DASHBOARD_V2_FRONTEND`: Build the mgr/dashboard_v2 frontend using `npm install && npm run build` * Set this flag to `OFF` when building packages. * Removed creation of the frontend from `vstart.sh` Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

With this change, we avoid the disabling/enabling of the ceph-mgr module being tested for each test function declared in each test case. Now the ceph-mgr module being tested is disabled/enabled only once for each test case. Signed-off-by: Ricardo Dias <rdias@suse.com>

We moved the dashboard_v2 test cases that required a real ceph cluster to run into qa/tasks/mgr with the necessary adaptations. These tests can now be run in teuthology using the cephfs_test_runner task. Signed-off-by: Ricardo Dias <rdias@suse.com>

Signed-off-by: Ricardo Dias <rdias@suse.com>

smithfarm

As far as I can tell, this passed a rados suite and works as advertised.

smithfarm · 2018-03-05T20:38:44Z

ceph.spec.in

@@ -888,6 +906,7 @@ cmake .. \
    -DWITH_EMBEDDED=OFF \
    -DWITH_MANPAGE=ON \
    -DWITH_PYTHON3=ON \
+    -DWITH_MGR_DASHBOARD_V2_FRONTEND=OFF \


why OFF, though?

The frontend build requires internet access which is not available in packaging build services, such as OBS. Because of this reason we build the frontend when generating the tarball and include the result files in the tarball.

@rjfd yeah, i have the same question. since make-dist is updated to include the generated files under src/pybind/mgr/dashboard_v2/frontend/dist , we can safely enable this option in debian/rules and ceph.spec.in, i think?

the WITH_MGR_DASHBOARD_V2_FRONTEND enables/disables the frontend compilation, if we enable it during the packaging build (inside OBS, or other packaging service) it will require nodejs and npm as system dependencies, and will try to download all nodejs packages frontend dependencies when running npm install (https://github.com/ceph/ceph/pull/20103/files#diff-bb5b00ad365f9278caa7bfa071fc7ce3R15). This will fail for two reasons:

a) usually the nodejs and npm system dependencies are too old, that's why we have to upgrade node and npm in install-deps.sh
b) packaging services don't allow internet access during build and npm install will fail

This is the reason why we include the frontend compiled files in the tarball to avoid the above problems.

It's counter-intuitive for the dashboard to be set to "OFF" in the packaging. There is an analogous situation with Boost, which is downloaded by make-dist. During the build cmake checks for the existence of a Boost-specific file, and only attempts the download if that file doesn't exist. See https://github.com/ceph/ceph/blob/master/cmake/modules/BuildBoost.cmake#L132-L159 for the implementation.

If doing that is too much trouble, could we at least rename the option to something more descriptive (and non-negative) like "-DWITH_MGR_DASHBOARD_V2_NODEJS_DEPS_IN_TARBALL=ON"

If doing that is too much trouble, could we at least rename the option to something more descriptive (and non-negative) like "-
DWITH_MGR_DASHBOARD_V2_NODEJS_DEPS_IN_TARBALL=ON"

From a POV of cmake, this flag disables or enables the build of the frontend. Changing this cmake flag has no effect on the tarball creation.

@smithfarm I agree with your concern, and I think we can even drop the cmake option, but that will require another round of build testing that takes some time. Can I open a PR after this one is merged dropping the cmake option?

@rjfd Yes, I was going to say that this concern of mine should not block merge of this PR.

smithfarm · 2018-03-05T20:38:59Z

debian/rules

@@ -5,7 +5,7 @@ export DESTDIR=$(CURDIR)/debian/tmp

 export DEB_HOST_ARCH      ?= $(shell dpkg-architecture -qDEB_HOST_ARCH)

-extraopts += -DUSE_CRYPTOPP=OFF -DWITH_OCF=ON -DWITH_LTTNG=ON -DWITH_PYTHON3=ON -DWITH_EMBEDDED=OFF
+extraopts += -DUSE_CRYPTOPP=OFF -DWITH_OCF=ON -DWITH_LTTNG=ON -DWITH_PYTHON3=ON -DWITH_EMBEDDED=OFF -DWITH_MGR_DASHBOARD_V2_FRONTEND=OFF


same reason as above

jcsp

I'm happy with the vstart+qa changes + the UI code itself has already been peer reviewed.

jcsp · 2018-03-05T22:25:19Z

Installed latest centos RPMs in my test environment and it all looks good! 👍 👍 👍

tchaikov

just skimmed through the change, lgtm in general.

mgr/dashboard_v2: Initial submission of a web-based management UI (replacement for the existing dashboard) Reviewed-by: Nathan Cutler <ncutler@suse.com> Reviewed-by: John Spray <john.spray@redhat.com> Reviewed-by: Kefu Chai <kchai@redhat.com> (cherry picked from commit 23eb14d) Signed-off-by: Ernesto Puerta <epuertat@redhat.com> Conflicts: ceph.spec.in install-deps.sh qa/tasks/mgr/mgr_test_case.py qa/tasks/mgr/test_prometheus.py src/pybind/CMakeLists.txt src/vstart.sh Conflict-solving summary: ceph.spec.in: - Add make-check distro-conditional Python BuildRequires: CherryPy, tox, coverage and bcrypt. - Add ceph-mgr package Python Requires: bcrypt install-deps.sh - Add ensure_min_npm_version qa/tasks/mgr/mgr_test_case.py - Pick setUpClass method from remote qa/tasks/mgr/test_prometheus.py - Delete src/pybind/CMakeLists.txt - Add add_subdirectory(mgr) src/vstart.sh - Bring DASH_V2_URLS and dashboard_v2 config setting.

Merge pull request ceph#20103 from openattic/wip-mgr-dashboard_v2 Reviewed-by: Nathan Cutler <ncutler@suse.com> Reviewed-by: John Spray <john.spray@redhat.com> Reviewed-by: Kefu Chai <kchai@redhat.com> (cherry picked from commit 23eb14d) Signed-off-by: Ernesto Puerta <epuertat@redhat.com> Conflicts: ceph.spec.in install-deps.sh qa/tasks/mgr/mgr_test_case.py qa/tasks/mgr/test_prometheus.py src/pybind/CMakeLists.txt src/vstart.sh Conflict-solving summary: ceph.spec.in: - Add make-check distro-conditional Python BuildRequires: CherryPy, tox, coverage and bcrypt. - Add ceph-mgr package Python Requires: bcrypt install-deps.sh - Add ensure_min_npm_version qa/tasks/mgr/mgr_test_case.py - Pick setUpClass method from remote qa/tasks/mgr/test_prometheus.py - Delete src/pybind/CMakeLists.txt - Add add_subdirectory(mgr) src/vstart.sh - Bring DASH_V2_URLS and dashboard_v2 config setting.

jcsp changed the title ~~[WIP] mgr/dashboard_v2: Initial submission of a web-based management/monitoring UI~~ [RFC, DNM] mgr/dashboard_v2: Initial submission of a web-based management/monitoring UI Jan 24, 2018

jcsp added mgr feature labels Jan 24, 2018

tchaikov added the DNM label Jan 25, 2018

rjfd force-pushed the wip-mgr-dashboard_v2 branch 2 times, most recently from c77ba1a to 2ebb335 Compare February 5, 2018 13:04

tspmelo force-pushed the wip-mgr-dashboard_v2 branch from 294e973 to 0d984a4 Compare February 12, 2018 14:35

rjfd force-pushed the wip-mgr-dashboard_v2 branch from 6ef48b7 to da2caea Compare February 19, 2018 09:54

LenzGr added the dashboard label Feb 27, 2018

rjfd force-pushed the wip-mgr-dashboard_v2 branch from cafd5bb to 1efdf5c Compare March 1, 2018 12:00

LenzGr changed the title ~~[RFC, DNM] mgr/dashboard_v2: Initial submission of a web-based management/monitoring UI~~ mgr/dashboard_v2: Initial submission of a web-based management UI (replacement for the existing dashboard) Mar 1, 2018

LenzGr added needs-review and removed DNM labels Mar 1, 2018

rjfd added needs-qa DNM needs-review and removed needs-review DNM labels Mar 1, 2018

rjfd requested review from jcsp, tchaikov, b-ranto and smithfarm March 1, 2018 12:09

p-se and others added 9 commits March 5, 2018 13:07

mgr/dashboard_v2: Remove Service and ServiceMeta class

12f3c7a

Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

mgr/dashboard_v2: Integrate dashboard_v2 into make check

d7ead63

Also: * Disabled API tests in tox. * No longer start a vstart.sh cluster in tox. Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard_v2: Add mgr-dashboard_v2 frontend unittests to ctest

97606b4

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard_v2: Added smoke test.

19bb5db

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

qa/suites/mgr/dashboard_v2: added suite yaml for testing dashboard_v2

7eba318

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard_v2: added script to run backend API tests locally

5449857

Signed-off-by: Ricardo Dias <rdias@suse.com>

smithfarm approved these changes Mar 5, 2018

View reviewed changes

smithfarm reviewed Mar 5, 2018

View reviewed changes

rjfd force-pushed the wip-mgr-dashboard_v2 branch from 8eab5a2 to 5449857 Compare March 5, 2018 21:58

jcsp mentioned this pull request Mar 5, 2018

mgr/dashboard v2: implement can_run method #20728

Merged

jcsp approved these changes Mar 5, 2018

View reviewed changes

tchaikov approved these changes Mar 6, 2018

View reviewed changes

votdev mentioned this pull request Mar 6, 2018

mgr/dashboard v2: Add units to performance counters #20742

Merged

smithfarm removed needs-qa needs-review labels Mar 6, 2018

tchaikov merged commit 23eb14d into ceph:master Mar 6, 2018

epuertat mentioned this pull request Jul 27, 2018

luminous: mgr/dashboard_v2: #1 - Initial submission of a web-based management UI (replacement for the existing dashboard) #23271

Closed

mgr/dashboard_v2: Initial submission of a web-based management UI (replacement for the existing dashboard) #20103

mgr/dashboard_v2: Initial submission of a web-based management UI (replacement for the existing dashboard) #20103

LenzGr commented Jan 24, 2018 •

edited

runsisi commented Jan 25, 2018 •

edited

LenzGr commented Jan 25, 2018

liewegas commented Feb 28, 2018

ceph-jenkins commented Feb 28, 2018

LenzGr commented Mar 1, 2018

rjfd commented Mar 1, 2018

smithfarm commented Mar 1, 2018

alfredodeza commented Mar 1, 2018

smithfarm left a comment

smithfarm Mar 5, 2018

rjfd Mar 5, 2018

tchaikov Mar 6, 2018

rjfd Mar 6, 2018

smithfarm Mar 6, 2018

sebastian-philipp Mar 6, 2018

rjfd Mar 6, 2018

smithfarm Mar 6, 2018

smithfarm Mar 5, 2018

rjfd Mar 5, 2018

jcsp left a comment

jcsp commented Mar 5, 2018

tchaikov left a comment

mgr/dashboard_v2: Initial submission of a web-based management UI (replacement for the existing dashboard) #20103

mgr/dashboard_v2: Initial submission of a web-based management UI (replacement for the existing dashboard) #20103

Conversation

LenzGr commented Jan 24, 2018 • edited

runsisi commented Jan 25, 2018 • edited

LenzGr commented Jan 25, 2018

liewegas commented Feb 28, 2018

ceph-jenkins commented Feb 28, 2018

LenzGr commented Mar 1, 2018

rjfd commented Mar 1, 2018

smithfarm commented Mar 1, 2018

alfredodeza commented Mar 1, 2018

smithfarm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jcsp left a comment

Choose a reason for hiding this comment

jcsp commented Mar 5, 2018

tchaikov left a comment

Choose a reason for hiding this comment

LenzGr commented Jan 24, 2018 •

edited

runsisi commented Jan 25, 2018 •

edited