Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support other options besides device names since those change across reboots #1228

Closed
lorenz opened this issue Nov 14, 2017 · 24 comments · Fixed by #4489
Closed

support other options besides device names since those change across reboots #1228

lorenz opened this issue Nov 14, 2017 · 24 comments · Fixed by #4489
Labels

Comments

@lorenz
Copy link
Contributor

lorenz commented Nov 14, 2017

Currently Rook does not work with symlinked devices (/dev/zvol, /dev/disk/by-id, LVM, ...), it requires a real device inode. This makes it hard to use on block device providers (I'm personally interested in LVM and ZFS zvol) for which these inodes are not consistent across reboots. Rook code extensively works with raw inode names, so I don't really know where support for that should be implemented.

@costrouc
Copy link

Bump. I have had the same issue and would really like to use /dev/disk/by-id/* symlinked devices. There is no way to guarantee consistent naming to /dev/sd*

@mhulscher
Copy link

I too would like to be able to use /dev/disk/by-id/* instead of the /dev/xxx ...

@costrouc
Copy link

@mhulsher I forgot to update my response in this issue. I talked with the devs on slack and I looks like rook uses the /dev/disk/by-partuuid/ once it has created the osds. This guarantees that once the osds are prepared the correct osd is always selected even if /dev/sd* changes. But I do agree that by-id would be nice to use.

@jbw976 jbw976 changed the title Bluestore on symlinked devices support other options besides device names since those change across reboots Mar 1, 2018
@jbw976
Copy link
Member

jbw976 commented Mar 1, 2018

The issue of device names changing across reboots was addressed in the cluster update design doc at https://github.com/rook/rook/blob/master/design/cluster-update.md#device-name-changes

However, also adding support for /dev/disk/by-id/ and others would still be quite useful.

@sjdevries
Copy link

I'd love to see /dev/disk/by-path/

IIRC, on my server the SAS expander backplane or the HBA has limits on mixing SAS / SATA, ssd/hdd drives. Specifically bays 0-1 are supposed to be the two used for cache drives when mixing types.

It would be great to be able to set metadata to the /dev/disk/by-path/ for those 2 bays and storage to the paths for the other 10.

Not sure if newer hardware has similar issues with NVMe / SAS / SATA / etc.

@pdf
Copy link

pdf commented Mar 11, 2018

Even ignoring reboots, being able to reference backing disks by stable device names would be very useful, particularly for environments where the disks may be attached dynamically, and (along with rook) programmatically.

@jbw976
Copy link
Member

jbw976 commented May 25, 2018

As described in https://github.com/rook/rook/blob/master/design/multiple-storage-types-support.md#storagescopespec, a user should be able to use the StorageScopeSpec to select devices not just by name, but other more persistent properties such as by-id or by-path.

@alfredodeza
Copy link

Once rook stats using ceph-volume, this will not be a problem since LVM takes care of matching a device to its logical volume name

@lorenz
Copy link
Contributor Author

lorenz commented Aug 15, 2018

Honestly an even better idea would be to rely on Kubernetes volumes and block devices which would make the whole detection and identity thing just go away. Security-wise it's also pretty nice since it woud forego the requirement to have any sort of privileged access.

I'm working on a CSI ZFS provisioner which can provision local volumes and block devices which could then be used to run Rook on top of it. This would allow a nicely layered setup, some sort of local storage system for things that need local storage or can live with downtime when moving storage and distributed storage systems layered on top of that.

@stale
Copy link

stale bot commented Nov 13, 2018

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the wontfix label Nov 13, 2018
@voron
Copy link

voron commented Nov 18, 2018

/removestale

@stale
Copy link

stale bot commented Feb 16, 2019

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the wontfix label Feb 16, 2019
@voron
Copy link

voron commented Feb 17, 2019

/removestale

@stale
Copy link

stale bot commented May 18, 2019

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the wontfix label May 18, 2019
@stale
Copy link

stale bot commented May 25, 2019

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.

@stale stale bot closed this as completed May 25, 2019
@travisn travisn reopened this May 29, 2019
@stale stale bot removed the wontfix label May 29, 2019
@stale
Copy link

stale bot commented Aug 27, 2019

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the wontfix label Aug 27, 2019
@Bessonov
Copy link

Activity!

@stale stale bot removed the wontfix label Aug 28, 2019
@travisn
Copy link
Member

travisn commented Oct 9, 2019

@mvollman Weren't you looking at configuring OSDs based on the full path for the device? I can't seem to find a related PR, but I was sure one was at least in progress.

@mvollman
Copy link
Contributor

@travisn I am not currently looking into it. It did come up in the scope of another PR a while back, but I was unable to add the capability at that time. I will try to find time to take another look at it.

@pdf
Copy link

pdf commented Nov 27, 2019

Looks like this is released with v1.1?

@tarjintor
Copy link

hello,it seems I met the problem too.
My collegue add some new disks to my cluster,and restart the node
Then the disk name changed.
I used to use device: sdb
but now sdb is the new disk,and osd find the disk is empty and recovery now
Can we use uuid to select the osd device now?

@BlaineEXE
Copy link
Member

Is this resolved by #4285 ?

leseb added a commit to leseb/rook that referenced this issue Dec 12, 2019
Rook-Ceph now supports specify device by their fullpath instead of links
created by the kernel like /dev/sdb.

Closes: rook#1228
Co-authored-by: Sean Micklethwaite <sean@wayve.ai>
Signed-off-by: Sébastien Han <seb@redhat.com>
leseb added a commit to leseb/rook that referenced this issue Dec 12, 2019
Rook-Ceph now supports specify device by their fullpath instead of links
created by the kernel like /dev/sdb.

Closes: rook#1228
Co-authored-by: Sean Micklethwaite <sean@wayve.ai>
Signed-off-by: Sébastien Han <seb@redhat.com>
leseb added a commit to leseb/rook that referenced this issue Feb 13, 2020
Rook-Ceph now supports specify device by their fullpath instead of links
created by the kernel like /dev/sdb.

Closes: rook#1228
Co-authored-by: Sean Micklethwaite <sean@wayve.ai>
Signed-off-by: Sébastien Han <seb@redhat.com>
leseb added a commit to leseb/rook that referenced this issue Feb 13, 2020
Rook-Ceph now supports specify device by their fullpath instead of links
created by the kernel like /dev/sdb.

Closes: rook#1228
Co-authored-by: Sean Micklethwaite <sean@wayve.ai>
Signed-off-by: Sébastien Han <seb@redhat.com>
leseb added a commit to leseb/rook that referenced this issue Feb 13, 2020
Rook-Ceph now supports specify device by their fullpath instead of links
created by the kernel like /dev/sdb.

Closes: rook#1228
Co-authored-by: Sean Micklethwaite <sean@wayve.ai>
Signed-off-by: Sébastien Han <seb@redhat.com>
@muhlba91
Copy link

muhlba91 commented Apr 1, 2020

Since we would need this change: is a backport to 1.2 planned or will the 1.3 release be available rather soon-ish? :)

@travisn
Copy link
Member

travisn commented Apr 1, 2020

It isn't planned for backport to 1.2. The target for 1.3 is early next week.

binoue pushed a commit to binoue/rook that referenced this issue Apr 10, 2020
Rook-Ceph now supports specify device by their fullpath instead of links
created by the kernel like /dev/sdb.

Closes: rook#1228
Co-authored-by: Sean Micklethwaite <sean@wayve.ai>
Signed-off-by: Sébastien Han <seb@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet