Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

need workaround for ceph bug, for OSD nodes and ceph-volume batch mode #4855

Closed
ppbrown opened this issue Dec 11, 2019 · 9 comments
Closed

need workaround for ceph bug, for OSD nodes and ceph-volume batch mode #4855

ppbrown opened this issue Dec 11, 2019 · 9 comments
Labels
needs info This issue requires further information (logs, inventory, group_vars/*.yml) wontfix

Comments

@ppbrown
Copy link
Contributor

ppbrown commented Dec 11, 2019 •
edited

ansible 2.8
ceph nautilus

# ceph-volume lvm --report --bluestore --yes batch /dev/sda /dev/sdb /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdl
--> Aborting because strategy changed from bluestore.MixedType to bluestore.SingleType after filtering

This is similar to issue #4748
In THAT issue, someone commented along the lines of,
"well yes its expected to fail, if you initially create a bunch of OSDs, then call ceph-volume with a different set of devices".

Except that I'm NOT using a different set of devices. its the exact same set that was first set up on the machine.

So apparently, theres some kind of bug in ceph, where it happily sets up a set of mixed ssd/hdd devices the first time in batch mode... but then --report will be broken ever after ?

Unless I'm misunderstanding, and the previous bug report is saying that ALL machines in the cluster must have the EXACT SAME device paths as each other.
Which would be insane, given that linux can randomly rewrite device paths, and /dev/sda can become /dev/sdl after a reboot.

Right now, the only workaround I would seem to have, is:
Add a machine one time to the [osds] ansible host group. get it configured... then take it OUT of the group.

Would be nice to have a better alternative.

(or is that kind of expected ceph-ansible behaviour? to create a dynamic inventory file, and only put in it, what needs to be (re)configured this time ? )
@dsavineau
Copy link
Contributor

Except that I'm NOT using a different set of devices. its the exact same set that was first set up on the machine.

Did you zap/purge/destroy (whatever you use) the devices before the initial deployment ?

So apparently, theres some kind of bug in ceph, where it happily sets up a set of mixed ssd/hdd devices the first time in batch mode... but then --report will be broken ever after ?

Can you share the /var/log/ceph/ceph-volume.log output when running the command ?

On my side I can rerun the ceph-volume batch command without any issue either manually or via new ceph-ansible execution.

# ceph-volume lvm --report --bluestore --yes batch /dev/sdb /dev/sdc
--> All devices are already used by ceph. No OSDs will be created.

Unless I'm misunderstanding, and the previous bug report is saying that ALL machines in the cluster must have the EXACT SAME device paths as each other.

That's not true. If you don't have the same devices configuration on all OSD nodes then instead of using the group_vars you can use the host_vars.

Right now, the only workaround I would seem to have, is:
Add a machine one time to the [osds] ansible host group. get it configured... then take it OUT of the group.

You don't need to do that.

Again, share the logs, configuration or anything that could help.

BTW before using stuff like need workaround for ceph bug in the issue title let's investigate to find the root cause.

@dsavineau dsavineau added the needs info This issue requires further information (logs, inventory, group_vars/*.yml) label Dec 11, 2019
@ppbrown
Copy link
Contributor Author

ppbrown commented Dec 11, 2019

For your ceph-volume test above, was one of those devicdes SSD and the other one HDD ?
Thats specifically what needs to be tested.

on the brighter side, looks like I can share the ceph-volume log, after I filtered out the keys.
Im just attaching fot the last 24 hours. which is large enough.

ceph-volume.log

@ppbrown
Copy link
Contributor Author

ppbrown commented Dec 12, 2019 •
edited by guits

Okay. I happen to have had to recreate my test cluster from scratch. so here is explicit demo output
demonstrating the bug in ceph, for ceph-volume, batch mode,and mixed SSD/HDD reporting.
batch --report works fine... until after the devices are actually deployed in ceph.. then it blows up.

[root@XHOSTNAMEX deploy]#
[root@XHOSTNAMEX deploy]# ceph -s
  cluster:
    id:     xxxxxxxx-6d9f-4fbc-9c22-6c189efa1b3f
    health: HEALTH_OK

  services:
    mon: 1 daemons, quorum XHOSTNAMEX (age 8m)
    mgr: XHOSTNAMEX(active, since 8m)
    osd: 3 osds: 3 up (since 17s), 3 in (since 17s)

  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0 B
    usage:   3.0 GiB used, 5.5 TiB / 5.5 TiB avail
    pgs:

[root@XHOSTNAMEX deploy]# cat /tmp/devs
   /dev/sda /dev/sdb /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh


[root@XHOSTNAMEX deploy]# ceph-volume lvm batch --report `cat /tmp/devs`

Total OSDs: 3

Solid State VG:
  Targets:   block.db                  Total size: 220.00 GB
  Total LVs: 3                         Size per LV: 73.33 GB
  Devices:   /dev/sda, /dev/sdb

  Type            Path                                                    LV Size         % of device
----------------------------------------------------------------------------------------------------
  [data]          /dev/sdd                                                1.82 TB         100%
  [block.db]      vg: vg/lv                                               73.33 GB        33%
----------------------------------------------------------------------------------------------------
  [data]          /dev/sde                                                1.82 TB         100%
  [block.db]      vg: vg/lv                                               73.33 GB        33%
----------------------------------------------------------------------------------------------------
  [data]          /dev/sdf                                                1.82 TB         100%
  [block.db]      vg: vg/lv                                               73.33 GB        33%
[root@XHOSTNAMEX deploy]# ceph-volume lvm batch `cat /tmp/devs`

Total OSDs: 3

Solid State VG:
  Targets:   block.db                  Total size: 220.00 GB
  Total LVs: 3                         Size per LV: 73.33 GB
  Devices:   /dev/sda, /dev/sdb

  Type            Path                                                    LV Size         % of device
----------------------------------------------------------------------------------------------------
  [data]          /dev/sdd                                                1.82 TB         100%
  [block.db]      vg: vg/lv                                               73.33 GB        33%
----------------------------------------------------------------------------------------------------
  [data]          /dev/sde                                                1.82 TB         100%
  [block.db]      vg: vg/lv                                               73.33 GB        33%
----------------------------------------------------------------------------------------------------
  [data]          /dev/sdf                                                1.82 TB         100%
  [block.db]      vg: vg/lv                                               73.33 GB        33%
--> The above OSDs would be created if the operation continues
--> do you want to proceed? (yes/no) yes
Running command: /usr/sbin/vgcreate -s 1G --force --yes ceph-block-499b12ab-639c-4428-b833-186f346c1fef /dev/sdd
 stdout: Physical volume "/dev/sdd" successfully created.
 stdout: Volume group "ceph-block-499b12ab-639c-4428-b833-186f346c1fef" successfully created
Running command: /usr/sbin/vgcreate -s 1G --force --yes ceph-block-073e90c8-9440-4e97-8b5e-121ea710219c /dev/sde
 stdout: Physical volume "/dev/sde" successfully created.
 stdout: Volume group "ceph-block-073e90c8-9440-4e97-8b5e-121ea710219c" successfully created
Running command: /usr/sbin/vgcreate -s 1G --force --yes ceph-block-042c6671-6205-40d0-962c-aec63721b29c /dev/sdf
 stdout: Physical volume "/dev/sdf" successfully created.
 stdout: Volume group "ceph-block-042c6671-6205-40d0-962c-aec63721b29c" successfully created
Running command: /usr/sbin/vgcreate -s 1G --force --yes ceph-block-dbs-34fb0c98-5da3-4c2c-bd90-56cc74741465 /dev/sdb /dev/sda
 stdout: Physical volume "/dev/sdb" successfully created.
 stdout: Physical volume "/dev/sda" successfully created.
 stdout: Volume group "ceph-block-dbs-34fb0c98-5da3-4c2c-bd90-56cc74741465" successfully created
Running command: /usr/sbin/lvcreate --yes -l 1863 -n osd-block-3ea03e63-1477-4361-8772-b4cf2b6ec68b ceph-block-499b12ab-639c-4428-b833-186f346c1fef
 stdout: Logical volume "osd-block-3ea03e63-1477-4361-8772-b4cf2b6ec68b" created.
Running command: /usr/sbin/lvcreate --yes -l 74 -n osd-block-db-87563fce-3548-480f-940f-51975026030d ceph-block-dbs-34fb0c98-5da3-4c2c-bd90-56cc74741465
 stdout: Logical volume "osd-block-db-87563fce-3548-480f-940f-51975026030d" created.
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 826d858d-b5d0-43f7-869e-b446125513e3
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-3
Running command: /usr/sbin/restorecon /var/lib/ceph/osd/ceph-3
Running command: /bin/chown -h ceph:ceph /dev/ceph-block-499b12ab-639c-4428-b833-186f346c1fef/osd-block-3ea03e63-1477-4361-8772-b4cf2b6ec68b
Running command: /bin/chown -R ceph:ceph /dev/dm-5
Running command: /bin/ln -s /dev/ceph-block-499b12ab-639c-4428-b833-186f346c1fef/osd-block-3ea03e63-1477-4361-8772-b4cf2b6ec68b /var/lib/ceph/osd/ceph-3/block
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-3/activate.monmap
 stderr: 2019-12-11 23:34:34.221 7f5b6c1c2700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-12-11 23:34:34.221 7f5b6c1c2700 -1 AuthRegistry(0x7f5b64064e08) no keyring found at /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx
 stderr: got monmap epoch 1
Running command: /bin/ceph-authtool /var/lib/ceph/osd/ceph-3/keyring --create-keyring --name osd.3 --add-key AQBWw/Fd0o72BRAAtQx8BhxPHMS9Msi4LsMFLA==
 stdout: creating /var/lib/ceph/osd/ceph-3/keyring
added entity osd.3 auth(key=AQBWw/Fd0o72BRAAtQx8BhxPHMS9Msi4LsMFLA==)
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3/keyring
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3/
Running command: /bin/chown -h ceph:ceph /dev/ceph-block-dbs-34fb0c98-5da3-4c2c-bd90-56cc74741465/osd-block-db-87563fce-3548-480f-940f-51975026030d
Running command: /bin/chown -R ceph:ceph /dev/dm-6
Running command: /bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 3 --monmap /var/lib/ceph/osd/ceph-3/activate.monmap --keyfile - --bluestore-block-db-path /dev/ceph-block-dbs-34fb0c98-5da3-4c2c-bd90-56cc74741465/osd-block-db-87563fce-3548-480f-940f-51975026030d --osd-data /var/lib/ceph/osd/ceph-3/ --osd-uuid 826d858d-b5d0-43f7-869e-b446125513e3 --setuser ceph --setgroup ceph
--> ceph-volume lvm prepare successful for: ceph-block-499b12ab-639c-4428-b833-186f346c1fef/osd-block-3ea03e63-1477-4361-8772-b4cf2b6ec68b
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3
Running command: /bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-block-499b12ab-639c-4428-b833-186f346c1fef/osd-block-3ea03e63-1477-4361-8772-b4cf2b6ec68b --path /var/lib/ceph/osd/ceph-3 --no-mon-config
Running command: /bin/ln -snf /dev/ceph-block-499b12ab-639c-4428-b833-186f346c1fef/osd-block-3ea03e63-1477-4361-8772-b4cf2b6ec68b /var/lib/ceph/osd/ceph-3/block
Running command: /bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-3/block
Running command: /bin/chown -R ceph:ceph /dev/dm-5
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3
Running command: /bin/ln -snf /dev/ceph-block-dbs-34fb0c98-5da3-4c2c-bd90-56cc74741465/osd-block-db-87563fce-3548-480f-940f-51975026030d /var/lib/ceph/osd/ceph-3/block.db
Running command: /bin/chown -h ceph:ceph /dev/ceph-block-dbs-34fb0c98-5da3-4c2c-bd90-56cc74741465/osd-block-db-87563fce-3548-480f-940f-51975026030d
Running command: /bin/chown -R ceph:ceph /dev/dm-6
Running command: /bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-3/block.db
Running command: /bin/chown -R ceph:ceph /dev/dm-6
Running command: /bin/systemctl enable ceph-volume@lvm-3-826d858d-b5d0-43f7-869e-b446125513e3
 stderr: Created symlink from /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-3-826d858d-b5d0-43f7-869e-b446125513e3.service to /usr/lib/systemd/system/ceph-volume@.service.
Running command: /bin/systemctl enable --runtime ceph-osd@3
 stderr: Created symlink from /run/systemd/system/ceph-osd.target.wants/ceph-osd@3.service to /usr/lib/systemd/system/ceph-osd@.service.
Running command: /bin/systemctl start ceph-osd@3
--> ceph-volume lvm activate successful for osd ID: 3
--> ceph-volume lvm create successful for: ceph-block-499b12ab-639c-4428-b833-186f346c1fef/osd-block-3ea03e63-1477-4361-8772-b4cf2b6ec68b
Running command: /usr/sbin/lvcreate --yes -l 1863 -n osd-block-7001cf27-4173-4251-931d-beaf6505a9c2 ceph-block-073e90c8-9440-4e97-8b5e-121ea710219c
 stdout: Logical volume "osd-block-7001cf27-4173-4251-931d-beaf6505a9c2" created.
Running command: /usr/sbin/lvcreate --yes -l 74 -n osd-block-db-7ad12425-bba7-4b64-a207-48c602534a6d ceph-block-dbs-34fb0c98-5da3-4c2c-bd90-56cc74741465
 stdout: Wiping ext3 signature on /dev/ceph-block-dbs-34fb0c98-5da3-4c2c-bd90-56cc74741465/osd-block-db-7ad12425-bba7-4b64-a207-48c602534a6d.
 stdout: Logical volume "osd-block-db-7ad12425-bba7-4b64-a207-48c602534a6d" created.
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new ad832476-effc-4788-bb6b-4db933117622
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-4
Running command: /usr/sbin/restorecon /var/lib/ceph/osd/ceph-4
Running command: /bin/chown -h ceph:ceph /dev/ceph-block-073e90c8-9440-4e97-8b5e-121ea710219c/osd-block-7001cf27-4173-4251-931d-beaf6505a9c2
Running command: /bin/chown -R ceph:ceph /dev/dm-7
Running command: /bin/ln -s /dev/ceph-block-073e90c8-9440-4e97-8b5e-121ea710219c/osd-block-7001cf27-4173-4251-931d-beaf6505a9c2 /var/lib/ceph/osd/ceph-4/block
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-4/activate.monmap
 stderr: 2019-12-11 23:34:44.287 7f6ed473c700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-12-11 23:34:44.287 7f6ed473c700 -1 AuthRegistry(0x7f6ecc064e08) no keyring found at /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx
 stderr: got monmap epoch 1
Running command: /bin/ceph-authtool /var/lib/ceph/osd/ceph-4/keyring --create-keyring --name osd.4 --add-key AQBgw/FdO6pbCRAA5PGM/qJdxqmM4RwFIkm3Ww==
 stdout: creating /var/lib/ceph/osd/ceph-4/keyring
added entity osd.4 auth(key=AQBgw/FdO6pbCRAA5PGM/qJdxqmM4RwFIkm3Ww==)
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4/keyring
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4/
Running command: /bin/chown -h ceph:ceph /dev/ceph-block-dbs-34fb0c98-5da3-4c2c-bd90-56cc74741465/osd-block-db-7ad12425-bba7-4b64-a207-48c602534a6d
Running command: /bin/chown -R ceph:ceph /dev/dm-8
Running command: /bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 4 --monmap /var/lib/ceph/osd/ceph-4/activate.monmap --keyfile - --bluestore-block-db-path /dev/ceph-block-dbs-34fb0c98-5da3-4c2c-bd90-56cc74741465/osd-block-db-7ad12425-bba7-4b64-a207-48c602534a6d --osd-data /var/lib/ceph/osd/ceph-4/ --osd-uuid ad832476-effc-4788-bb6b-4db933117622 --setuser ceph --setgroup ceph
--> ceph-volume lvm prepare successful for: ceph-block-073e90c8-9440-4e97-8b5e-121ea710219c/osd-block-7001cf27-4173-4251-931d-beaf6505a9c2
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4
Running command: /bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-block-073e90c8-9440-4e97-8b5e-121ea710219c/osd-block-7001cf27-4173-4251-931d-beaf6505a9c2 --path /var/lib/ceph/osd/ceph-4 --no-mon-config
Running command: /bin/ln -snf /dev/ceph-block-073e90c8-9440-4e97-8b5e-121ea710219c/osd-block-7001cf27-4173-4251-931d-beaf6505a9c2 /var/lib/ceph/osd/ceph-4/block
Running command: /bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-4/block
Running command: /bin/chown -R ceph:ceph /dev/dm-7
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4
Running command: /bin/ln -snf /dev/ceph-block-dbs-34fb0c98-5da3-4c2c-bd90-56cc74741465/osd-block-db-7ad12425-bba7-4b64-a207-48c602534a6d /var/lib/ceph/osd/ceph-4/block.db
Running command: /bin/chown -h ceph:ceph /dev/ceph-block-dbs-34fb0c98-5da3-4c2c-bd90-56cc74741465/osd-block-db-7ad12425-bba7-4b64-a207-48c602534a6d
Running command: /bin/chown -R ceph:ceph /dev/dm-8
Running command: /bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-4/block.db
Running command: /bin/chown -R ceph:ceph /dev/dm-8
Running command: /bin/systemctl enable ceph-volume@lvm-4-ad832476-effc-4788-bb6b-4db933117622
 stderr: Created symlink from /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-4-ad832476-effc-4788-bb6b-4db933117622.service to /usr/lib/systemd/system/ceph-volume@.service.
Running command: /bin/systemctl enable --runtime ceph-osd@4
 stderr: Created symlink from /run/systemd/system/ceph-osd.target.wants/ceph-osd@4.service to /usr/lib/systemd/system/ceph-osd@.service.
Running command: /bin/systemctl start ceph-osd@4
--> ceph-volume lvm activate successful for osd ID: 4
--> ceph-volume lvm create successful for: ceph-block-073e90c8-9440-4e97-8b5e-121ea710219c/osd-block-7001cf27-4173-4251-931d-beaf6505a9c2
Running command: /usr/sbin/lvcreate --yes -l 1863 -n osd-block-03f9931a-493f-4501-a7f0-8df5177c1335 ceph-block-042c6671-6205-40d0-962c-aec63721b29c
 stdout: Logical volume "osd-block-03f9931a-493f-4501-a7f0-8df5177c1335" created.
Running command: /usr/sbin/lvcreate --yes -l 74 -n osd-block-db-af156afc-c3a1-4cc5-ac1f-7fdd19446b77 ceph-block-dbs-34fb0c98-5da3-4c2c-bd90-56cc74741465
 stdout: Logical volume "osd-block-db-af156afc-c3a1-4cc5-ac1f-7fdd19446b77" created.
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 7adf16c5-efae-4a13-81fe-e3827e48265e
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-5
Running command: /usr/sbin/restorecon /var/lib/ceph/osd/ceph-5
Running command: /bin/chown -h ceph:ceph /dev/ceph-block-042c6671-6205-40d0-962c-aec63721b29c/osd-block-03f9931a-493f-4501-a7f0-8df5177c1335
Running command: /bin/chown -R ceph:ceph /dev/dm-9
Running command: /bin/ln -s /dev/ceph-block-042c6671-6205-40d0-962c-aec63721b29c/osd-block-03f9931a-493f-4501-a7f0-8df5177c1335 /var/lib/ceph/osd/ceph-5/block
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-5/activate.monmap
 stderr: 2019-12-11 23:34:54.467 7f76d569e700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-12-11 23:34:54.467 7f76d569e700 -1 AuthRegistry(0x7f76d0064e08) no keyring found at /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx
 stderr: got monmap epoch 1
Running command: /bin/ceph-authtool /var/lib/ceph/osd/ceph-5/keyring --create-keyring --name osd.5 --add-key AQBqw/Fdk5ZbBxAA6zW9RYSExF2OFWYtl0j6XA==
 stdout: creating /var/lib/ceph/osd/ceph-5/keyring
added entity osd.5 auth(key=AQBqw/Fdk5ZbBxAA6zW9RYSExF2OFWYtl0j6XA==)
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5/keyring
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5/
Running command: /bin/chown -h ceph:ceph /dev/ceph-block-dbs-34fb0c98-5da3-4c2c-bd90-56cc74741465/osd-block-db-af156afc-c3a1-4cc5-ac1f-7fdd19446b77
Running command: /bin/chown -R ceph:ceph /dev/dm-10
Running command: /bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 5 --monmap /var/lib/ceph/osd/ceph-5/activate.monmap --keyfile - --bluestore-block-db-path /dev/ceph-block-dbs-34fb0c98-5da3-4c2c-bd90-56cc74741465/osd-block-db-af156afc-c3a1-4cc5-ac1f-7fdd19446b77 --osd-data /var/lib/ceph/osd/ceph-5/ --osd-uuid 7adf16c5-efae-4a13-81fe-e3827e48265e --setuser ceph --setgroup ceph
--> ceph-volume lvm prepare successful for: ceph-block-042c6671-6205-40d0-962c-aec63721b29c/osd-block-03f9931a-493f-4501-a7f0-8df5177c1335
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5
Running command: /bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-block-042c6671-6205-40d0-962c-aec63721b29c/osd-block-03f9931a-493f-4501-a7f0-8df5177c1335 --path /var/lib/ceph/osd/ceph-5 --no-mon-config
Running command: /bin/ln -snf /dev/ceph-block-042c6671-6205-40d0-962c-aec63721b29c/osd-block-03f9931a-493f-4501-a7f0-8df5177c1335 /var/lib/ceph/osd/ceph-5/block
Running command: /bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-5/block
Running command: /bin/chown -R ceph:ceph /dev/dm-9
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5
Running command: /bin/ln -snf /dev/ceph-block-dbs-34fb0c98-5da3-4c2c-bd90-56cc74741465/osd-block-db-af156afc-c3a1-4cc5-ac1f-7fdd19446b77 /var/lib/ceph/osd/ceph-5/block.db
Running command: /bin/chown -h ceph:ceph /dev/ceph-block-dbs-34fb0c98-5da3-4c2c-bd90-56cc74741465/osd-block-db-af156afc-c3a1-4cc5-ac1f-7fdd19446b77
Running command: /bin/chown -R ceph:ceph /dev/dm-10
Running command: /bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-5/block.db
Running command: /bin/chown -R ceph:ceph /dev/dm-10
Running command: /bin/systemctl enable ceph-volume@lvm-5-7adf16c5-efae-4a13-81fe-e3827e48265e
 stderr: Created symlink from /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-5-7adf16c5-efae-4a13-81fe-e3827e48265e.service to /usr/lib/systemd/system/ceph-volume@.service.
Running command: /bin/systemctl enable --runtime ceph-osd@5
Running command: /bin/systemctl start ceph-osd@5
--> ceph-volume lvm activate successful for osd ID: 5
--> ceph-volume lvm create successful for: ceph-block-042c6671-6205-40d0-962c-aec63721b29c/osd-block-03f9931a-493f-4501-a7f0-8df5177c1335
[root@XHOSTNAMEX deploy]#
[root@XHOSTNAMEX deploy]#
[root@XHOSTNAMEX deploy]#
[root@XHOSTNAMEX deploy]#
[root@XHOSTNAMEX deploy]# ceph-volume lvm batch --report `cat /tmp/devs`
--> Aborting because strategy changed from bluestore.MixedType to bluestore.SingleType after filtering

$

@stale
Copy link

stale bot commented Jan 11, 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the wontfix label Jan 11, 2020
@dimm0
Copy link

dimm0 commented Jan 12, 2020

Still an issue

@stale stale bot removed the wontfix label Jan 12, 2020
@stale
Copy link

stale bot commented Feb 11, 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the wontfix label Feb 11, 2020
@stale stale bot closed this as completed Feb 18, 2020
@madkiss
Copy link

madkiss commented May 25, 2020

This is still an issue. I just ran into this on a RHOP 16 cluster.

@dsavineau
Copy link
Contributor

This has been fixed by e4c24f3

@foysalkayum
Copy link

ansible 2.8
ceph nautilus

# ceph-volume lvm --report --bluestore --yes batch /dev/sda /dev/sdb /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdl
--> Aborting because strategy changed from bluestore.MixedType to bluestore.SingleType after filtering

This is similar to issue #4748
In THAT issue, someone commented along the lines of,
"well yes its expected to fail, if you initially create a bunch of OSDs, then call ceph-volume with a different set of devices".

Except that I'm NOT using a different set of devices. its the exact same set that was first set up on the machine.

So apparently, theres some kind of bug in ceph, where it happily sets up a set of mixed ssd/hdd devices the first time in batch mode... but then --report will be broken ever after ?

Unless I'm misunderstanding, and the previous bug report is saying that ALL machines in the cluster must have the EXACT SAME device paths as each other.
Which would be insane, given that linux can randomly rewrite device paths, and /dev/sda can become /dev/sdl after a reboot.

Right now, the only workaround I would seem to have, is:
Add a machine one time to the [osds] ansible host group. get it configured... then take it OUT of the group.

You mean just take out all the OSD nodes from host group and only running ansible script adding the new node?
We have 9 active node on the cluster, Is there have any disastrous issue doing so?

Would be nice to have a better alternative.

(or is that kind of expected ceph-ansible behaviour? to create a dynamic inventory file, and only put in it, what needs to be (re)configured this time ? )

@micmoyles micmoyles mentioned this issue Dec 10, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
needs info This issue requires further information (logs, inventory, group_vars/*.yml) wontfix
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@dimm0 @madkiss @ppbrown @dsavineau @foysalkayum