Skip to content

Ceph on XCP ng 7.5 or later

Samuel VERSCHELDE edited this page May 22, 2019 · 12 revisions

Warning: in XCP-ng 7.6, no modified version of qemu-dp is provided yet, due to the discussion at https://github.com/xcp-ng-rpms/qemu-dp/issues/5 not being resolved in the RBD on QEMU connection. However work is in progress to understand and build on the storage stack. Help welcome.


For the full discussion about Ceph in XCP-ng, see this forum thread: https://xcp-ng.org/forum/topic/4/ceph-on-xcp-ng


Following some working setup and installation guide.

Install EPEL repo

In order to install the needed package we'll need to add extra repos. The first one is EPEL:

yum install epel-release -y --enablerepo=extras

WARNING: Remember to disable EPEL (which is automatically turned on after installation). Because EPEL will conflict with XCP upgrade scripts.

Install Ceph Release Repo

We need to had the repository of the Ceph Version we decided to use. This can be done by installing the release package centos-release-ceph-{version}. Of course you must install only one version of these repository:

# yum install centos-release-ceph-nautilus --enablerepo=extras
# yum install centos-release-ceph-mimic --enablerepo=extras
# yum install centos-release-ceph-luminous --enablerepo=extras

Install the Ceph Packages

Install the yum-plugin-priority:

yum install yum-plugin-priorities --enablerepo='base'

Note from the release manager (stormi): I do not advise installing yum-plugin-priorities in dom0, due to possible upgrade issues in the future. It would be good to test whether the procedure really needs this step.

Install the Ceph packages needed:

yum install ceph-common --enablerepo='base,extras,epel'

Note from stormi, still: make sure this doesn't overwrite existing libs or packages from original dom0.

Choose your connection mode Ceph

Warning: Ceph is not yet supported at it's best. Currently there are many concurrent solution in order to plugin. No one of theme are fully tested.

CephFS with Kernel

CephFS is not the best way to connect but it's probably the most easy way to setup. You can read about the patch to apply here: https://xcp-ng.org/forum/topic/1006/ceph-fs-storage-driver/2 Please be aware that on XCP 7.6 with kernel 4.4.0+10 you'll need to downgrade the tunables due to lack features on this kernel. You can downgrade your Ceph Feature Profile by running this command on a ceph-mon:

ceph osd crush tunables hammer

CephFS with Fuse

If you don't want to downgrade your Ceph Cluster Feature than you'll need to connect avoiding Kernel support. However this will decrease your performance: you can do by installing FUSE and Ceph-FUSE support.

yum install ceph-fuse --enablerepo='base,extras,epel,centos-release-ceph-nautilus'

RBD on QEMU

This implementation is currently working on XCP7.5, but broken on XCP7.6 due to the lack of a driver which is not opensource anymore. In order to use Ceph on XCP 7.5 you'll need to install the patched QEMU package: Install qemu-dp from the extras_testing repository:

yum install qemu-dp --enablerepo="xcp-ng-extras_testing*"

To go back to the base version of qemu-dp: yum downgrade qemu-dp

Once installed, note that future updates of qemu-dp will get installed from the base repository, not from extras_testing, unless you activate the extras_testing repository for updates. So you will have to run the above yum install command again after an update of qemu-dp. If you choose to activate the extras_testing repository for updates, any package that has an experimental version in extras_testing will get installed instead of that of base.

Then install the RBD plugin from this repo: https://github.com/rposudnevskiy/RBDSR

You can follow this discussion about build an open driver for XCP 7.6 here: https://github.com/xcp-ng-rpms/qemu-dp/issues/5

Clone this wiki locally
You can’t perform that action at this time.