mgr/cephadm: adding support for nvmeof

[rkachach]

To test the changes on this branch:

1. Use the image from registry.hub.docker.com/rkachach/ceph-nvmeof:v1.0 to bootstrap your cluster
2. One the cluster is up, healthy and running create a pool named rbd by ceph osd pool create rbd
3. from a cephadm shell, use the following spec to create the nvmeof service as following (saved in nvmeof.yaml i.e):

service_type: nvmeof
service_id: nvmeof.test
placement:
  hosts:
    - ceph-node-0
spec:
  pool: rbd
  name: myname
  group: mygroup

4. ceph orch apply -i nvmeof.yaml

To test the daemon you we can use the quay.io/ceph/nvmeof-cli:0.0.1, i.e:

podman run --rm -it "quay.io/ceph/nvmeof-cli:0.0.1" --server-address 192.168.100.100 --server-port 5500 get_subsystems

Output:

INFO:__main__:Get subsystems:
[
    {
        "nqn": "nqn.2014-08.org.nvmexpress.discovery",
        "subtype": "Discovery",
        "listen_addresses": [],
        "allow_any_host": true,
        "hosts": []
    }
]

https://tracker.ceph.com/issues/61929

Contribution Guidelines

• To sign and title your commits, please refer to Submitting Patches to Ceph.

• If you are submitting a fix for a stable branch (e.g. "pacific"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.

Checklist

• Tracker (select at least one)
  • References tracker ticket
  • Very recent bug; references commit where it was introduced
  • New feature (ticket optional)
  • Doc update (no ticket needed)
  • Code cleanup (no ticket needed)
• Component impact
  • Affects Dashboard, opened tracker ticket
  • Affects Orchestrator, opened tracker ticket
  • No impact that needs to be tracked
• Documentation (select at least one)
  • Updates relevant documentation
  • No doc update is appropriate
• Tests (select at least one)
  • Includes unit test(s)
  • Includes integration test(s)
  • Includes bug reproducer
  • No tests

Show available Jenkins commands

• jenkins retest this please
• jenkins test classic perf
• jenkins test crimson perf
• jenkins test signed
• jenkins test make check
• jenkins test make check arm64
• jenkins test submodules
• jenkins test dashboard
• jenkins test dashboard cephadm
• jenkins test api
• jenkins test docs
• jenkins render docs
• jenkins test ceph-volume all
• jenkins test ceph-volume tox
• jenkins test windows

[github-actions]

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

[rkachach]

jenkins retest this please

[rkachach]

jenkins test make check

[rkachach]

jenkins test dashboard cephadm

[rkachach]

jenkins test make check

[rkachach]

jenkins test api

[rkachach]

jenkins test make check

[epuertat]

Good progress. I think it's almost there! Just a few comments.

[phlogistonjohn]

I just skimmed thru this PR and nothing major jumps out at me. I would recommend squashing your patches together since the 3 later patches just fix things in your new code. But it's not a major requirement.

Much of the patch is just plumbing the new types through. The interesting bits are the extra parameters to make the container function and while I don't love seeing extra caps added, if they're needed they're needed.

[adk3798]

Looks like it's mostly there. Still unsure about how SSL is working with this (I see there are options to pass in certs in the spec, just not sure how they're used) and it would be nice to clean up the keyring stuff, but the overall structure seems pretty good.

[rkachach]

I just skimmed thru this PR and nothing major jumps out at me. I would recommend squashing your patches together since the 3 later patches just fix things in your new code. But it's not a major requirement.

Much of the patch is just plumbing the new types through. The interesting bits are the extra parameters to make the container function and while I don't love seeing extra caps added, if they're needed they're needed.

Thank you for reviewing the code @phlogistonjohn, I'll be squashing the changes are in fact most of the commits are small changes to fix UT and other changes from the reviews. The caps are needed by the nvmeof daemon to function so at this moment we have to add them.

[adk3798]

https://pulpito.ceph.com/adking-2023-07-12_03:23:58-orch:cephadm-wip-adk-cephadm-nvmeof-distro-default-smithi/

Obviously, we don't have a test for deploying nvme-of itself yet, but did a run to make sure this wasn't breaking anything. Only failure was a known issue (https://tracker.ceph.com/issues/59704) so the run was clean.

[rkachach]

https://pulpito.ceph.com/adking-2023-07-12_03:23:58-orch:cephadm-wip-adk-cephadm-nvmeof-distro-default-smithi/

Obviously, we don't have a test for deploying nvme-of itself yet, but did a run to make sure this wasn't breaking anything. Only failure was a known issue (https://tracker.ceph.com/issues/59704) so the run was clean.

Awesome, thank you very much @adk3798 for taking care of this.

[rkachach]

jenkins test make check

[adk3798]

https://pulpito.ceph.com/adking-2023-07-19_13:28:53-orch:cephadm-wip-adk4-testing-2023-07-18-1038-distro-default-smithi/

3 Failures, 5 Dead jobs

• 2 Failures caused by https://tracker.ceph.com/issues/62084, unrelated to PRs in the run
• 1 failure caused by https://tracker.ceph.com/issues/59704, known issue, unrelated to PRs in the run
• 5 dead jobs were infra/teuthology issues, falling under some variation of Error reimaging machines. Didn't bother with reruns for this particular run

Overall, nothing to block merging

[adk3798]

jenkins retest this please

[adk3798]

This still needs follow up work, but as @rkachach is going to be out for some time, going to merge this to main and fix the keyring issue and add proper testing in a follow up