Certain disks skipped for being too small, when they are large enough. #11474
Comments
|
"rock5b-13" seems not to specify "/dev/sda". Could you add "/dev/sda" to be picked by Rook (e.g. If osd prepare pod won't pick up sda, please show me the log of parepare pod again.
This log was shown while processing "deviceFilter: (sd[a-z])" filter and doesn't directly correspond to the detection of "/dev/sda." |
|
According to the documentation, Here are the logs from setting rook-ceph-operator-5bc5659499-bw4lp.log Additionally, I had attempted this previously with
This sounds promising! Would you please direct me to the details of this bug and perhaps suggest a version not affected by it? |
|
I read the new logs and found that this behavior is actually odd. sda is one of the a candidate of the osd creation, has enough space, hasn't any fs, and has a valid device type ("disk"). I'll try to reproduce this problem later (might be the next year.)
I'll search the ceph issue tracker and will tell you the issue. |
|
We're also running into this. Our full configuration: https://github.com/ocf/kubernetes/blob/main/apps/rook.py Our drives that are 3.5T aren't affected. Also on ceph Did a little bit of debugging, it looks like the LHS of this is returning
and I think Funnily enough, this code was actually touched very recently (should be in Ceph 18) but the change won't fix this: ceph/ceph@4b3d174 since it happens before the point where it checks removable. I tried working around this with a udev rule but it did not work :( |
|
I ended up doing a kernel patch for fun: ocf/nix@bac6ec2, but this is clearly a hack 🙃 It works great though, my OSDs are all up now 🤣🤣🤣 |
|
I read Ceph's code. There are two problems.
I'll submit an Ceph issue about (b) because it's apparently a bug. On the other hand, whether (a) is correct or not depends on Ceph's design. This behavior was introduced in this commit. Since this is not a Rook's matter but a Ceph's matter, @MrDrMcCoy @nikhiljha Could you open a new issue to support removable devices? |
|
I have the same issue when using 32G USB devices in my Raspberry Pi playground. Is it not possible to use USB devices for a ceph cluster? |
|
It's impossible. Please open a new issue(feature request) to Ceph issue tracker if necessary. |
|
Good to know, thank you. I understand the limitations after reading some more discussion threads but it would be good to have something like this in the documentation if possible? Or is this something that really I should have already had the knowledge for? |
|
Awesome, thank you! |
|
FYI, there is a rejected feature request to support removble devices. ceph-volume does not allow the use of removable disks |
done. report "Insufficient space (<5GB)" even when disk size is sufficient |
|
This problem is not Rook's problem. It will be fixed in this Ceph'S PR. |
|
Weird, these drives use to be detected on this same motherboard/disk combo. Now only the nvmes are. |
|
Heads up for anyone in the same boat: Create a lvm volume spanning the entire disk and pass that to ceph. NOTE: For some bizarre reason Ceph doesn't take into account vg names so it needs different lv names for all disks on the same node, otherwise it won't create any OSDs because of name conflicts. DISK="/dev/sda"
DISK_NO="1"
VG_NAME="hdd$DISK_NO"
LV_NAME="lv$DISK_NO"
vgcreate $VG_NAME $DISK
lvcreate -n $LV_NAME -l 100%FREE $VG_NAMEcluster.yaml: nodes:
- name: node-1
devices:
- name: /dev/mapper/hdd1-lv1
config:
deviceClass: hdd
- name: /dev/mapper/hdd2-lv2
config:
deviceClass: hdd |
|
For information, I had the same problem and could relate it to the fact that the drives where flagged as removable. Changing device settings from AHCI to IDE in the BIOS fixed the issue. Ceph now uses the drives. Writing this in case it can be a solution for someone else. |
Deviation from expected behavior:
Rook / Ceph believes that a certain disk type used on all my nodes is smaller than 5GB and refuses to use it, despite its actual size being 256GB.
Prepare pod log message:
cephosd: skipping device "sda": ["Insufficient space (<5GB)"].Actual size of device:
Expected behavior:
Rook / Ceph should use this disk.
How to reproduce it (minimal and precise):
/var/lib/rookFile(s) to submit:
Helm values:
rook-ceph-cluster-v1.10.7-values.yml.txt
rook-ceph-v1.10.7-values.yml.txt
Logs to submit:
rook-ceph-osd-prepare-rock5b-13-dz2fd.log
rook-ceph-operator-5bc5659499-ndq4z.log
rook-discover-xr4tw.log
Cluster Status to submit:
Environment:
Debian GNU/Linux Bookworm (Sid, custom built Armbian image, needed for certain kernel modules and btrfs rootfs)uname -a):Linux rock5b-13 5.10.110-rockchip-rk3588-gadfc1747e7fe-dirty #trunk SMP Tue Dec 20 11:19:50 UTC 2022 aarch64 GNU/LinuxRadxa Rock5 Model Brook versioninside of a Rook Pod):v1.10.7ceph -v):ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)kubectl version):v1.26.0Microk8sceph healthin the Rook Ceph toolbox):HEALTH_OKThe text was updated successfully, but these errors were encountered: