ceph-osd-device-activation: a systemd service to trigger underlying LVM device activation

[deepfire]

Description of changes

This addresses point #2 of #242152, by creating a service to activate the LVM devices underlying the OSDs.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[SuperSandro2000]

other than the formatting cleanup I can't contribute much here.

@astro can you maybe?

[flokli]

Before going into detail about formatting, I'd first like to understand why this is necessary, and whether that approach is the right one. We call a low-level plumbing command, in a systemd unit, with a --no-systemd cmdline argument. This sounds wrong.

I wonder if LVM or ceph are supposed to provide some udev rules that normally take care of that?

[nh2]

I cannot review this PR, as I'm not using the NixOS Ceph service; we use our own NixOS service which does not use ceph-volume but writes systemd units explicitly (I dumped that as open source somewhere but want to make it available to NixOS in a more cleaned-up way eventually).

[nh2]

@deepfire Our custom NixOS Ceph service modules: https://gist.github.com/nh2/13425a1f18b4c1ce82edb63c10b163c9

[johanot]

@flokli

From: https://docs.ceph.com/en/mimic/ceph-volume/lvm/activate/#summary

tldr; ceph-volume tries to activate the LVMs themselves if they are not activated already, and... if it was only that, you're right, we wouldn't need it. But, activation step 3 and 4 prepares "stuff" in /var which needs to be present before that start of the OSD's will succeed.

--no-systemd just tells ceph-volume to skip step 2, which would fail hard on NixOS anyway.

[deepfire]

@deepfire Our custom NixOS Ceph service modules: https://gist.github.com/nh2/13425a1f18b4c1ce82edb63c10b163c9

Looks very interesting, thank you @nh2!

I'd need to get my long-planned playground Ceph cluster with nixops-libvirtd, before I get to hack on that, though..

[deepfire]

@SuperSandro2000, done, thanks!

@flokli, what are your considerations now, after @johanot 's clarification?

[flokli]

I tried to dig deeper into this, but apparently ceph is broken on nixos-unstable and master.

The first point in #242152 also mentions:

Use template service files, particularly for the OSD daemons.

I'm wondering how this plays together with this approach, just want to make sure we don't put ourselves into one corner where it's harder to navigate out of.

Spelunking around in (a successful) build, there's ceph-volume-systemd, which normally seems to get be called by a ceph-volume@… template unit (which we don't ship in NixOS yet).

It seems it takes care of all the necessary activation bits mentioned under the hood as well, so moving to template units should would the necessity for the unit file introduced here alltogether.

[deepfire]

@flokli, oh, so we have a better startup point -- so that's great news!

I'd like to play with that, once I get a staging setup with nixops-libvirtd.

So, I guess, it'd be wise to deprecate this PR in favor of the better future implementation.

[flokli]

Yeah, I don't think there's a certain urgency to land this right now, especially considering even ceph itself doesn't build on the current channel ;-) I'd prefer to stay more in line with how it looks like on other distros, so yeah, let's try that approach first.