mgr/cephadm: nvme-of follow up work #52691
Conversation
| - cephadm.shell: | ||
| host.a: | ||
| - ceph osd pool create foo | ||
| - rbd pool init foo |
There was a problem hiding this comment.
Just curious, what is the difference between the above and ceph osd pool application enable foo rbd?
There was a problem hiding this comment.
Very little, if anything at all. One "initializes" a pool for rbd use and the other lets you "enable" the use of rbd on the pool (from their descriptions). I think we can use them interchangeably in this context.
adk3798
force-pushed
the
nvme-of-cleanup
branch
from
c08dff0
to
b07e62f
Compare
|
The current changes LGTM, modulo the temporary container image hack. Once there's a usable upstream build with the change and you drop that temporary commit please ping me for final approval! |
This is going to be used as the rados_id to be set when connecting to the cluster using the keyring we generate for the nvmeof daemon. The python librados library defaults the name to "client.admin" and so if we don't provide a name or rados_id, we'll only be able to use nvmeof with the "client.admin" keyring Signed-off-by: Adam King <adking@redhat.com>
Before, we were just using the client.admin keyring as a temporary workaround while we figured out how to get the keyring to work. We should swap over to using the keyring we actually generated for the nvmeof daemon. Signed-off-by: Adam King <adking@redhat.com>
The ok-to-stop function works for certain daemons by checking if there are at least a certain number (typically 1) daemon(s) that are actually running and saying it's not ok-to-stop if if that won't be true after the removals. This case breaks down when all the daemons are in error state, making it so cephadm will refuse to remove a set of daemons that aren't even working because they're not "ok to stop". Since ok-to-stop works in a yes or no fashion, something like this where we want to be willing to remove a certain subset (or potentially all currently deployed) daemons it's easier to keep this logic as part of applying the service Signed-off-by: Adam King <adking@redhat.com>
Similar to what is done for iscsi, basic deployment test to make sure we can deploy the daemon and it comes up in running state with no issue Signed-off-by: Adam King <adking@redhat.com>
Rather than giving full admin privileges, try to be a bit more strict by limiting it to profile rbd mon caps and full OSD privileges for rbd tagged pools. I also wanted to include an OSD cap like allow all pool="*" object_prefix "nvmeof.state" but this caused a failure in the nvme-of daemon RADOS permission error (Failed to operate write op for oid nvmeof.None.state) Signed-off-by: Adam King <adking@redhat.com>
This is the IP the nvmeof daemon will bind to, so it should be the IP of the host we're deploying the nvmeof daemon on, not the IP of the active mgr Signed-off-by: Adam King <adking@redhat.com>
"igw_id" was leftover from the nvmeof implementation being taken heavily from the iscsi implementation. "igw" means nothing in this context, so we can change the name. Signed-off-by: Adam King <adking@redhat.com>
0.0.2 includes a patch that allows the nvmeof daemon to use non-admin keyrings, so we should use it over 0.0.1 Signed-off-by: Adam King <adking@redhat.com>
adk3798
force-pushed
the
nvme-of-cleanup
branch
from
b07e62f
to
3f0bece
Compare
this is no longer pointing towards a temp image. An official nvmeof image 0.0.2 was made with the necessary changes (ceph/ceph-nvmeof#172) and we now point to that. |
|
jenkins retest this please |
|
7 failures, 3 dead (as of writing this 1 dead and 2 running but the 2 running will time out) jobs
Overall, nothing related to the mds upgrade sequence or jaeger-tracing can be merged, but nothing here to block merging most other things in the run. |
The first piece of the nvme-of deployment work in cephadm was already merged in main. This PR is a follow up that fixes a handful of issues found in the original implementation.
This depends on having an nvme-of container with changes being proposed in ceph/ceph-nvmeof#166. For now, I have a temporary commit that points the default nvme-of image to an image on my personal quay with the changes present. That should be removed before merging (signed-off-by will fail as long as that commit is present.
Contribution Guidelines
To sign and title your commits, please refer to Submitting Patches to Ceph.
If you are submitting a fix for a stable branch (e.g. "pacific"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.
Checklist
Show available Jenkins commands
jenkins retest this pleasejenkins test classic perfjenkins test crimson perfjenkins test signedjenkins test make checkjenkins test make check arm64jenkins test submodulesjenkins test dashboardjenkins test dashboard cephadmjenkins test apijenkins test docsjenkins render docsjenkins test ceph-volume alljenkins test ceph-volume toxjenkins test windows