rolling_update: migrate ceph-disk osds to ceph-volume

[andrewschoen]

When upgrading to nautlius run ceph-volume simple scan and
ceph-volume simple activate --all to migrate any running
ceph-disk osds to ceph-volume.

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1656460

Signed-off-by: Andrew Schoen aschoen@redhat.com

[dsavineau]

I think you need to add a condition because this will only work for non-containerized deployment.

[andrewschoen]

I think you need to add a condition because this will only work for non-containerized deployment.

You're right, I've got that added now.

[andrewschoen]

The dev-centos-non_container-update test is passing now, but it's deploying mimic initially with ceph-volume created osds. It's good to test that the updates here will noop when ran against a ceph-volume created osd, but I'd like to add another testing scenario that upgrades from luminous with ceph-disk created osds to nautilus.

[andrewschoen]

I manually ran the new testing sceneario added here and it was successful. https://2.jenkins.ceph.com/job/ceph-ansible-scenario/548/consoleFull

You can see the ceph-volume simple commands in the log here:

TASK [scan ceph-disk osds with ceph-volume if deploying nautilus] **************
task path: /home/jenkins-build/build/workspace/ceph-ansible-scenario/rolling_update.yml:422
Monday 01 April 2019  15:17:30 +0000 (0:00:00.054)       0:14:48.096 ********** 
changed: [osd0] => changed=true 
  cmd:
  - ceph-volume
  - --cluster=test
  - simple
  - scan
  delta: '0:00:01.706350'
  end: '2019-04-01 15:17:34.692910'
  rc: 0
  start: '2019-04-01 15:17:32.986560'
  stderr: ''
  stderr_lines: []
  stdout: |2-
     stderr: lsblk: /var/lib/ceph/osd/test-0: not a block device
     stderr: Unknown device, --name=, --path=, or absolute path in /dev/ or /sys expected.
    Running command: /usr/sbin/cryptsetup status /dev/sdb1
    --> OSD 0 got scanned and metadata persisted to file: /etc/ceph/osd/0-c93e4979-6ff6-4d7e-aba3-34c4c3bb7184.json
    --> To take over management of this scanned OSD, and disable ceph-disk and udev, run:
    -->     ceph-volume simple activate 0 c93e4979-6ff6-4d7e-aba3-34c4c3bb7184
     stderr: lsblk: /var/lib/ceph/osd/test-1: not a block device
     stderr: Unknown device, --name=, --path=, or absolute path in /dev/ or /sys expected.
    Running command: /usr/sbin/cryptsetup status /dev/sdc1
    --> OSD 1 got scanned and metadata persisted to file: /etc/ceph/osd/1-1e090b86-0ce4-4db1-90e9-071873b17d2d.json
    --> To take over management of this scanned OSD, and disable ceph-disk and udev, run:
    -->     ceph-volume simple activate 1 1e090b86-0ce4-4db1-90e9-071873b17d2d
  stdout_lines: <omitted>

TASK [activate scanned ceph-disk osds and migrate to ceph-volume if deploying nautilus] ***
task path: /home/jenkins-build/build/workspace/ceph-ansible-scenario/rolling_update.yml:430
Monday 01 April 2019  15:17:35 +0000 (0:00:04.542)       0:14:52.639 ********** 
changed: [osd0] => changed=true 
  cmd:
  - ceph-volume
  - --cluster=test
  - simple
  - activate
  - --all
  delta: '0:00:01.209180'
  end: '2019-04-01 15:17:38.771386'
  rc: 0
  start: '2019-04-01 15:17:37.562206'
  stderr: ''
  stderr_lines: []
  stdout: |-
    --> activating OSD specified in /etc/ceph/osd/0-c93e4979-6ff6-4d7e-aba3-34c4c3bb7184.json
    Running command: /bin/ln -snf /dev/sdb2 /var/lib/ceph/osd/test-0/block
    Running command: /bin/chown -R ceph:ceph /dev/sdb2
    Running command: /bin/systemctl enable ceph-volume@simple-0-c93e4979-6ff6-4d7e-aba3-34c4c3bb7184
     stderr: Created symlink from /etc/systemd/system/multi-user.target.wants/ceph-volume@simple-0-c93e4979-6ff6-4d7e-aba3-34c4c3bb7184.service to /usr/lib/systemd/system/ceph-volume@.service.
    Running command: /bin/ln -sf /dev/null /etc/systemd/system/ceph-disk@.service
    --> All ceph-disk systemd units have been disabled to prevent OSDs getting triggered by UDEV events
    Running command: /bin/systemctl enable --runtime ceph-osd@0
    Running command: /bin/systemctl start ceph-osd@0
    --> Successfully activated OSD 0 with FSID c93e4979-6ff6-4d7e-aba3-34c4c3bb7184
    --> activating OSD specified in /etc/ceph/osd/1-1e090b86-0ce4-4db1-90e9-071873b17d2d.json
    Running command: /bin/ln -snf /dev/sdc2 /var/lib/ceph/osd/test-1/block
    Running command: /bin/chown -R ceph:ceph /dev/sdc2
    Running command: /bin/systemctl enable ceph-volume@simple-1-1e090b86-0ce4-4db1-90e9-071873b17d2d
     stderr: Created symlink from /etc/systemd/system/multi-user.target.wants/ceph-volume@simple-1-1e090b86-0ce4-4db1-90e9-071873b17d2d.service to /usr/lib/systemd/system/ceph-volume@.service.
    Running command: /bin/ln -sf /dev/null /etc/systemd/system/ceph-disk@.service
    --> All ceph-disk systemd units have been disabled to prevent OSDs getting triggered by UDEV events
    Running command: /bin/systemctl enable --runtime ceph-osd@1
    Running command: /bin/systemctl start ceph-osd@1
    --> Successfully activated OSD 1 with FSID 1e090b86-0ce4-4db1-90e9-071873b17d2d
  stdout_lines: <omitted>

There is a caveat to upgrading to nautilus. For the upgrade to complete, the user must switch the osd_scenario to lvm. If the devices config option was previously used to deploy ceph-disk osds then those can not be used with ceph-volume. To get around this I've made it so no new osds will be created during an update. Afterwards the user will need to remove any devices listed in devices that were previous used by ceph-disk. These devices can not be used by ceph-volume because 1) they are already running OSDs and 2) they are rejected immediately because they have GPT headers.

[guits]

The idea in ceph-ansible@master is to test an upgrade from ceph@nautilus to ceph@dev

The same scenario to upgrade ceph@luminous to ceph@nautilus will be tested in ceph-ansible@stable-4.0

[andrewschoen]

The idea in ceph-ansible@master is to test an upgrade from ceph@nautilus to ceph@dev

The same scenario to upgrade ceph@luminous to ceph@nautilus will be tested in ceph-ansible@stable-4.0

What value are we really gaining by running tests that upgrade from nautilus to ceph@master? The point is that I must upgrade from luminous (or mimic) to nautilus so that I can verify that the new ceph-volume simple additions are working correctly. Deploying nautilus initially doesn't give me ceph-disk created OSDs to test with.

[guits]

so it should be done in stable-4.0 branch, you upgrade from luminous to nautilus

[andrewschoen]

so it should be done in stable-4.0 branch, you upgrade from luminous to nautilus

I'm not sure I understand. Are you saying that I should deploy luminous with stable-3.2 and then deploy nautilus with stable-4.0?

[guits]

jenkins test pipeline

[andrewschoen]

@guits I think I've addressed everything here, would you mind taking another look? I reran my test manually and it still succeeds as well. https://2.jenkins.ceph.com/job/ceph-ansible-scenario/572/consoleFull

[guits]

failures are unrelated, merging anyway