Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OSD on PVC: Support LVM's logical volume #4185

Closed
satoru-takeuchi opened this issue Oct 25, 2019 · 4 comments · Fixed by #4219
Closed

OSD on PVC: Support LVM's logical volume #4185

satoru-takeuchi opened this issue Oct 25, 2019 · 4 comments · Fixed by #4219
Labels
Milestone

Comments

@satoru-takeuchi
Copy link
Contributor

@satoru-takeuchi satoru-takeuchi commented Oct 25, 2019

Is this a bug report or feature request?

  • Feature Request

What should the feature do:

Support LV with OSD on PVC.

What is use case behind this feature:

PV would be not only physical devices but also logical devices. For example,
there is a CSI plugin using LVM.

https://github.com/cybozu-go/topolvm

There is a PR to support LV with OSD. However, it's not for OSD on PVC,
but on normal block devices:
#2100

@travisn

This comment has been minimized.

Copy link
Member

@travisn travisn commented Oct 25, 2019

@satoru-takeuchi Can you add more details for this scenario? The CSI driver is independent from the OSDs, so I'm not sure what feature you're requesting. If this is for a feature request for OSDs on PVCs, can you give an example for how it is related to the StorageClassDeviceSet in this design doc? Thanks!

@satoru-takeuchi

This comment has been minimized.

Copy link
Contributor Author

@satoru-takeuchi satoru-takeuchi commented Oct 29, 2019

@travisn OK, I should explain this issue in detail.

There is no problem with StorageClassDeviceSet level. However, there are several problems
to use LV-based PVC as OSD.

First, LV is not regarded as valid block device in PopulateDeviceInfo.

https://github.com/rook/rook/blob/master/pkg/clusterd/disk.go#L84

So rook-ceph-osd-prepare pod fails with the following log.

2019-10-17 02:59:50.640651 I | exec: Running command: lsblk /mnt/set1-0-data-kvd8t --bytes --nodeps --pairs --output SIZE,ROTA,RO,TYPE,PKNAME
failed to get device info for /mnt/set1-0-data-kvd8t. unsupported diskType lvm

Second, when executing ceph-volume lvm prepare --data , we should pass LV not as
<path/to/device>, but <vg_name/pv_name> way. However, there is no such logic.

I'm preparing a PR that solves these problems.

morimoto-cybozu added a commit to cybozu-go/rook that referenced this issue Oct 29, 2019
In "OSD on PVC", the prepared PV can be an LVM logical volume.
This commit accepts the lvm-type volume in disk type checking,
and invokes "ceph-volume lvm prepare" with "--data vg/lv"
instead of "--data /path/to/device".
This also suppresses activation/deactivation of the parent VG
because this VG is not dedicated for the LV.

Fixes: rook#4185
satoru-takeuchi added a commit to cybozu-go/rook that referenced this issue Oct 30, 2019
"OSD on PVC" doesn't work for PV backed by LV. Fixing this problem
by the following changes.

- Rook accepts LVM disk type.
- If a LV-backed device is passed, Rook/Ceph invokes
  "ceph-volume lvm prepare" with "--data vg/lv"
  instead of "--data /path/to/device".
- If a LV-backed device is passed, Rook/Ceph suppresses
  activation/deactivation of VG that owns this LV.

Fixes: rook#4185
Signed-off-by: Satoru Takeuchi <sat@cybozu.co.jp>
@satoru-takeuchi satoru-takeuchi mentioned this issue Oct 30, 2019
9 of 9 tasks complete
satoru-takeuchi added a commit to cybozu-go/rook that referenced this issue Oct 30, 2019
"OSD on PVC" doesn't work for PV backed by LV. Fixing this problem
by the following changes.

- Rook accepts LVM disk type.
- If a LV-backed device is passed, Rook/Ceph invokes
  "ceph-volume lvm prepare" with "--data vg/lv"
  instead of "--data /path/to/device".
- If a LV-backed device is passed, Rook/Ceph suppresses
  activation/deactivation of VG that owns this LV.

Fixes: rook#4185
Signed-off-by: Satoru Takeuchi <sat@cybozu.co.jp>
satoru-takeuchi added a commit to cybozu-go/rook that referenced this issue Nov 1, 2019
"OSD on PVC" doesn't work for PV backed by LV. Fixing this problem
by the following changes.

- Rook accepts LVM disk type.
- If a LV-backed device is passed, Rook/Ceph invokes
  "ceph-volume lvm prepare" with "--data vg/lv"
  instead of "--data /path/to/device".
- If a LV-backed device is passed, Rook/Ceph suppresses
  activation/deactivation of VG that owns this LV.

Fixes: rook#4185
Signed-off-by: Satoru Takeuchi <sat@cybozu.co.jp>
satoru-takeuchi added a commit to cybozu-go/rook that referenced this issue Nov 5, 2019
"OSD on PVC" doesn't work for PV backed by LV. Fixing this problem
by the following changes.

- Rook accepts LVM disk type.
- If a LV-backed device is passed, Rook/Ceph invokes
  "ceph-volume lvm prepare" with "--data vg/lv"
  instead of "--data /path/to/device".
- If a LV-backed device is passed, Rook/Ceph suppresses
  activation/deactivation of VG that owns this LV.

Fixes: rook#4185
Signed-off-by: Satoru Takeuchi <sat@cybozu.co.jp>
@jbw976 jbw976 added this to the 1.2 milestone Nov 5, 2019
satoru-takeuchi added a commit to cybozu-go/rook that referenced this issue Nov 7, 2019
"OSD on PVC" doesn't work for PV backed by LV. Fixing this problem
by the following changes.

- Rook accepts LVM disk type.
- If a LV-backed device is passed, Rook/Ceph invokes
  "ceph-volume lvm prepare" with "--data vg/lv"
  instead of "--data /path/to/device".
- If a LV-backed device is passed, Rook/Ceph suppresses
  activation/deactivation of VG that owns this LV.

Fixes: rook#4185
Signed-off-by: Satoru Takeuchi <sat@cybozu.co.jp>
satoru-takeuchi added a commit to cybozu-go/rook that referenced this issue Nov 7, 2019
"OSD on PVC" doesn't work for PV backed by LV. Fixing this problem
by the following changes.

- Rook accepts LVM disk type.
- If a LV-backed device is passed, Rook/Ceph invokes
  "ceph-volume lvm prepare" with "--data vg/lv"
  instead of "--data /path/to/device".
- If a LV-backed device is passed, Rook/Ceph suppresses
  activation/deactivation of VG that owns this LV.

Fixes: rook#4185
Signed-off-by: Satoru Takeuchi <sat@cybozu.co.jp>
satoru-takeuchi added a commit to cybozu-go/rook that referenced this issue Nov 7, 2019
"OSD on PVC" doesn't work for PV backed by LV. Fixing this problem
by the following changes.

- Rook accepts LVM disk type.
- If a LV-backed device is passed, Rook/Ceph invokes
  "ceph-volume lvm prepare" with "--data vg/lv"
  instead of "--data /path/to/device".
- If a LV-backed device is passed, Rook/Ceph suppresses
  activation/deactivation of VG that owns this LV.

Fixes: rook#4185
Signed-off-by: Satoru Takeuchi <sat@cybozu.co.jp>
Signed-off-by: Takahiro Tsuruda <isrgnoe@gmail.com>
satoru-takeuchi added a commit to cybozu-go/rook that referenced this issue Nov 7, 2019
"OSD on PVC" doesn't work for PV backed by LV. Fixing this problem
by the following changes.

- Rook accepts LVM disk type.
- If a LV-backed device is passed, Rook/Ceph invokes
  "ceph-volume lvm prepare" with "--data vg/lv"
  instead of "--data /path/to/device".
- If a LV-backed device is passed, Rook/Ceph suppresses
  activation/deactivation of VG that owns this LV.

Fixes: rook#4185
Signed-off-by: Takahiro Tsuruda <isrgnoe@gmail.com>
satoru-takeuchi added a commit to cybozu-go/rook that referenced this issue Nov 7, 2019
"OSD on PVC" doesn't work for PV backed by LV. Fixing this problem
by the following changes.

- Rook accepts LVM disk type.
- If a LV-backed device is passed, Rook/Ceph invokes
  "ceph-volume lvm prepare" with "--data vg/lv"
  instead of "--data /path/to/device".
- If a LV-backed device is passed, Rook/Ceph suppresses
  activation/deactivation of VG that owns this LV.

Fixes: rook#4185
Signed-off-by: dulltz <isrgnoe@gmail.com>
satoru-takeuchi added a commit to cybozu-go/rook that referenced this issue Nov 12, 2019
"OSD on PVC" doesn't work for PV backed by LV. Fixing this problem
by the following changes.

- Rook accepts LVM disk type.
- If a LV-backed device is passed, Rook/Ceph invokes
  "ceph-volume lvm prepare" with "--data vg/lv"
  instead of "--data /path/to/device".
- If a LV-backed device is passed, Rook/Ceph suppresses
  activation/deactivation of VG that owns this LV.

Fixes: rook#4185
Signed-off-by: dulltz <isrgnoe@gmail.com>
satoru-takeuchi added a commit to cybozu-go/rook that referenced this issue Nov 13, 2019
"OSD on PVC" doesn't work for PV backed by LV. Fixing this problem
by the following changes.

- Rook accepts LVM disk type.
- If a LV-backed device is passed, Rook/Ceph invokes
  "ceph-volume lvm prepare" with "--data vg/lv"
  instead of "--data /path/to/device".
- If a LV-backed device is passed, Rook/Ceph suppresses
  activation/deactivation of VG that owns this LV.

Fixes: rook#4185
Signed-off-by: dulltz <isrgnoe@gmail.com>
satoru-takeuchi added a commit to cybozu-go/rook that referenced this issue Nov 13, 2019
"OSD on PVC" doesn't work for PV backed by LV. Fixing this problem
by the following changes.

- Rook accepts LVM disk type.
- If a LV-backed device is passed, Rook/Ceph invokes
  "ceph-volume lvm prepare" with "--data vg/lv"
  instead of "--data /path/to/device".
- If a LV-backed device is passed, Rook/Ceph suppresses
  activation/deactivation of VG that owns this LV.

Fixes: rook#4185
Signed-off-by: dulltz <isrgnoe@gmail.com>
satoru-takeuchi added a commit to cybozu-go/rook that referenced this issue Nov 28, 2019
"OSD on PVC" doesn't work for PV backed by LV. Fixing this problem
by the following changes.

- Rook accepts LVM disk type.
- If a LV-backed device is passed, Rook/Ceph invokes
  "ceph-volume lvm prepare" with "--data vg/lv"
  instead of "--data /path/to/device".
- If a LV-backed device is passed, Rook/Ceph suppresses
  activation/deactivation of VG that owns this LV.

Fixes: rook#4185
Signed-off-by: dulltz <isrgnoe@gmail.com>
Co-authored-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
satoru-takeuchi added a commit to cybozu-go/rook that referenced this issue Dec 4, 2019
"OSD on PVC" doesn't work for PV backed by LV. Fixing this problem
by the following changes.

- Rook accepts LVM disk type.
- If a LV-backed device is passed, Rook/Ceph invokes
  "ceph-volume lvm prepare" with "--data vg/lv"
  instead of "--data /path/to/device".
- If a LV-backed device is passed, Rook/Ceph suppresses
  activation/deactivation of VG that owns this LV.

Fixes: rook#4185
Signed-off-by: dulltz <isrgnoe@gmail.com>
Co-authored-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
tapih added a commit to cybozu-go/rook that referenced this issue Dec 5, 2019
"OSD on PVC" doesn't work for PV backed by LV. Fixing this problem
by the following changes.

- Rook accepts LVM disk type.
- If a LV-backed device is passed, Rook/Ceph invokes
  "ceph-volume lvm prepare" with "--data vg/lv"
  instead of "--data /path/to/device".
- If a LV-backed device is passed, Rook/Ceph suppresses
  activation/deactivation of VG that owns this LV.

Fixes: rook#4185
Signed-off-by: dulltz <isrgnoe@gmail.com>
Co-authored-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
satoru-takeuchi added a commit to cybozu-go/rook that referenced this issue Dec 5, 2019
"OSD on PVC" doesn't work for PV backed by LV. Fixing this problem
by the following changes.

- Rook accepts LVM disk type.
- If a LV-backed device is passed, Rook/Ceph invokes
  "ceph-volume lvm prepare" with "--data vg/lv"
  instead of "--data /path/to/device".
- If a LV-backed device is passed, Rook/Ceph suppresses
  activation/deactivation of VG that owns this LV.

Fixes: rook#4185
Signed-off-by: dulltz <isrgnoe@gmail.com>
Co-authored-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
satoru-takeuchi added a commit to cybozu-go/rook that referenced this issue Dec 5, 2019
"OSD on PVC" doesn't work for PV backed by LV. Fixing this problem
by the following changes.

- Rook accepts LVM disk type.
- If a LV-backed device is passed, Rook/Ceph invokes
  "ceph-volume lvm prepare" with "--data vg/lv"
  instead of "--data /path/to/device".
- If a LV-backed device is passed, Rook/Ceph suppresses
  activation/deactivation of VG that owns this LV.

Fixes: rook#4185
Signed-off-by: dulltz <isrgnoe@gmail.com>
Co-authored-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
@travisn travisn closed this in #4219 Dec 5, 2019
tapih added a commit to cybozu-go/rook that referenced this issue Dec 10, 2019
"OSD on PVC" doesn't work for PV backed by LV. Fixing this problem
by the following changes.

- Rook accepts LVM disk type.
- If a LV-backed device is passed, Rook/Ceph invokes
  "ceph-volume lvm prepare" with "--data vg/lv"
  instead of "--data /path/to/device".
- If a LV-backed device is passed, Rook/Ceph suppresses
  activation/deactivation of VG that owns this LV.

Fixes: rook#4185
Signed-off-by: dulltz <isrgnoe@gmail.com>
Co-authored-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
rajatsingh25aug pushed a commit to rajatsingh25aug/rook that referenced this issue Dec 17, 2019
"OSD on PVC" doesn't work for PV backed by LV. Fixing this problem
by the following changes.

- Rook accepts LVM disk type.
- If a LV-backed device is passed, Rook/Ceph invokes
  "ceph-volume lvm prepare" with "--data vg/lv"
  instead of "--data /path/to/device".
- If a LV-backed device is passed, Rook/Ceph suppresses
  activation/deactivation of VG that owns this LV.

Fixes: rook#4185
Signed-off-by: dulltz <isrgnoe@gmail.com>
Co-authored-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
@R-omk

This comment has been minimized.

Copy link

@R-omk R-omk commented Dec 19, 2019

@travisn, Please explain me.

Suppose I have two lvm groups vg-nvme and vg-hdd.

I can create PVs with different fast-block and slow-block classes and associate them with different lvs respectively vg-nvme / ceph-metadata and vg-hdd / ceph-data.

How can I configure CephCluster so that metadata (wal, db) is placed on the fast drive, and the data on the slow one.

Can I even use the existing lvm instead of the raw device for metadata?

@travisn

This comment has been minimized.

Copy link
Member

@travisn travisn commented Dec 19, 2019

@R-omk I believe what you're looking for isn't implemented yet with OSDs on PVCs. See "Example 5: Multiple devices per OSD" in the design doc for storageClassDeviceSets. This work item is tracked by #3852. Until it is implemented, the separate of WAL/DB to a higher performance device is only available with raw devices via the metadataDevice property as documented in the Cluster CR.

zoetrope added a commit to cybozu-go/rook that referenced this issue Dec 26, 2019
"OSD on PVC" doesn't work for PV backed by LV. Fixing this problem
by the following changes.

- Rook accepts LVM disk type.
- If a LV-backed device is passed, Rook/Ceph invokes
  "ceph-volume lvm prepare" with "--data vg/lv"
  instead of "--data /path/to/device".
- If a LV-backed device is passed, Rook/Ceph suppresses
  activation/deactivation of VG that owns this LV.

Fixes: rook#4185
Signed-off-by: dulltz <isrgnoe@gmail.com>
Co-authored-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.