Moving drive between nodes

[panteparak]

I have 2 NVME SSD installed in each of my two out of three nodes. And I would like to move it to a different node. Is there a way to do this?

Helm Chart Version:
Operator: v1.7.3
RookCephCluster: v1.7.3

Goal: Move two nvme0n1 currently presented on k8s-node1, k8s-node2 to k8s-node0

Helm Values Files for RookCephCluster

configOverride: |
 [global]
 osd_pool_default_min_size = 1
 osd_pool_default_size = 1
 mon_warn_on_pool_no_redundancy = false

cephClusterSpec:
  cephVersion:
    image: ceph/ceph:v15.2.7
  dataDirHostPath: /var/lib/rook
  skipUpgradeChecks: true
  continueUpgradeAfterChecksEvenIfNotHealthy: true
  
  monitoring:
    enabled: false
    rulesNamespace: rook-ceph
  mon:
    count: 3
    allowMultiplePerNode: false
  mgr:
    modules:
    - name: pg_autoscaler
      enabled: true

  dashboard:
    enabled: true

  storage: # cluster level storage configuration and selection
    useAllNodes: false
    useAllDevices: false
    config:
      osdsPerDevice: "3"
    nodes:
      - name: "k8s-node1"
        devices:
          - name: "sda3"
          - name: "nvme0n1"
            config:
              osdsPerDevice: "4"
      - name: "k8s-node2"
        devices:
          - name: "sda3"
          - name: "nvme0n1"
            config:
              osdsPerDevice: "4"
      - name: "k8s-node0"
        devices:
          - name: "sda1"
            config:
              osdsPerDevice: "1"

[travisn]

@panteparak A couple points that should help you:

1. The osd prepare job always looks for existing OSDs configured on any device, whether or not it's specified in the cluster CR
2. The check for an OSD should be isolated completely to the disk

So if you move a device to another node and restart the operator to trigger the reconcile, the osd prepare job is expected to detect the OSD on the disk that was just moved from the other node.

I don't expect the CRUSH location will be updated automatically after the move. You will want to update the CRUSH map with the OSD's new host location such as:

ceph osd crush move osd.<ID> host=<newHost>

To see the CRUSH hierarchy: ceph osd tree

[panteparak]

works flawlessly, Thanks.

[blackliner]

@travisn did the automated relocation of the OSD change since you answered? See https://docs.ceph.com/en/pacific/rados/operations/crush-map/#crush-location

The CRUSH location for an OSD can be defined by adding the crush location option in ceph.conf. Each time the OSD starts, it verifies it is in the correct location in the CRUSH map and, if it is not, it moves itself. To disable this automatic CRUSH map management, add the following to your configuration file in the [osd] section:
osd crush update on start = false

[travisn]

It's quite possible. Are you seeing the crush location automatically updated? I haven't tested it lately.