ceph: add missing python library

[dermetfan]

This is a regression in 22.11. After updating from 22.05 the dashboard no longer works and ceph status reports a health check error.

The routes python library required by the mgr dashboard module is missing.

ceph -s reports:

  cluster:
    id:     …
    health: HEALTH_ERR
            Module 'dashboard' has failed: No module named 'routes'

ceph -w reports:

mgr.b [ERR] Unhandled exception from module 'dashboard' while running on mgr.b: No module named 'routes'
mon.b [ERR] Health check failed: Module 'dashboard' has failed: No module named 'routes' (MGR_MODULE_ERROR)

Description of changes

This simply adds the routes library to the python environment.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 23.05 Release Notes (or backporting 22.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
  • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
• Fits CONTRIBUTING.md.

[mweinelt]

Can you add the error you saw in ceph -s as well as how you set up the dashboard? It would be good to know, so we can add a test to keep this working in the future.

I would also be interested to know where ceph defines those python dependencies. Ideally we'd be looking them up, but I can't find routes anywhere in the 16.2.10 tarball.

[dermetfan]

Can you add the error you saw in ceph -s […]?

Sure, updated the description.

Can you add […] how you set up the dashboard?

I really only ran these commands via this NixOS module:

ceph mgr module enable dashboard
ceph config set mgr mgr/dashboard/b/server_port 6780

[dermetfan]

Looks like the python dependencies are only listed in the debian package.

[mweinelt]

Tried to make a test out of this, but the cluster remains healthy and the dashboard does not start listening on the given port.

diff --git a/nixos/tests/ceph-multi-node.nix b/nixos/tests/ceph-multi-node.nix
index 556546beee7..b8e17bee211 100644
--- a/nixos/tests/ceph-multi-node.nix
+++ b/nixos/tests/ceph-multi-node.nix
@@ -215,6 +215,17 @@ let
     monA.wait_until_succeeds("ceph osd stat | grep -e '3 osds: 3 up[^,]*, 3 in'")
     monA.wait_until_succeeds("ceph -s | grep 'mgr: ${cfg.monA.name}(active,'")
     monA.wait_until_succeeds("ceph -s | grep 'HEALTH_OK'")
+
+    # Enable the dashboard and recheck health
+    monA.succeed("ceph mgr module enable dashboard")
+    monA.succeed("ceph config set mgr mgr/dashboard/${cfg.monA.name}/server_port 8080")
+    monA.log(monA.execute("ceph -s")[1])
+    monA.sleep(10)
+    monA.log(monA.execute("ceph -s")[1])
+    monA.wait_for_open_port(8080)
+    monA.log(monA.execute("ss -lpn | grep 8080")[1])
+    monA.wait_until_succeeds("curl -q -fail http://localhost:8080")
+    monA.wait_until_succeeds("ceph -s | grep 'HEALTH_OK'")
   '';
 in {
   name = "basic-multi-node-ceph-cluster";

[dermetfan]

Turns out SSL is enabled by default and it waits for a key to be configured. Disabled that and it starts right up.

[mweinelt]

Giving it a quick spin w/o the fix to test the negative case.

[mweinelt]

@ofborg test ceph-single-node

[mweinelt]

Fails successfully due to the missing library. 🥳

monA # [  198.672953] ceph-mgr[5931]: 2023-01-20T23:35:50.854+0000 7f0567fff640 -1 dashboard.serve:                                               
monA # [  198.680830] ceph-mgr[5931]: 2023-01-20T23:35:50.854+0000 7f0567fff640 -1 Traceback (most recent call last):
monA # [  198.682508] ceph-mgr[5931]:   File "/nix/store/9dqwapw4b9r4vxgh2dccmc4x255wxhr7-ceph-16.2.10-lib/lib/ceph/mgr/dashboard/module.py", line 335, in serve
monA # [  198.684791] ceph-mgr[5931]:     mapper, parent_urls = Router.generate_routes(self.url_prefix)                                           
monA # [  198.686271] ceph-mgr[5931]:   File "/nix/store/9dqwapw4b9r4vxgh2dccmc4x255wxhr7-ceph-16.2.10-lib/lib/ceph/mgr/dashboard/controllers/_router.py", line 52, in generate_routes
monA # [  198.688732] ceph-mgr[5931]:     mapper = cherrypy.dispatch.RoutesDispatcher()                                                           
monA # [  198.690018] ceph-mgr[5931]:   File "/nix/store/079sjiv8nbahmhlfsw3l1ly9grfi0x6s-python3-3.9.16-env/lib/python3.9/site-packages/cherrypy/_cpdispatch.py", line 506, in __init__
monA # [  198.692549] ceph-mgr[5931]:     import routes                                                                                           
monA # [  198.693451] ceph-mgr[5931]: ModuleNotFoundError: No module named 'routes'                                                               
monA # [  200.339276] ceph-mon[873]: 2023-01-20T23:35:52.518+0000 7f27fd7fb640 -1 log_channel(cluster) log [ERR] : Health check failed: Module 'dashboard' has failed: No module named 'routes' (MGR_MODULE_ERROR)

[mweinelt]

Ideally at some point we'd split up the tests based on feature set. But that day is not today.

[github-actions]

Successfully created backport PR #211849 for release-22.11.