Join GitHub today
GitHub is home to over 36 million developers working together to host and review code, manage projects, and build software together.Sign up
Ceph on XCP ng 7.5 or later
Warning: in XCP-ng 7.6, no modified version of qemu-dp is provided yet, due to the discussion at https://github.com/xcp-ng-rpms/qemu-dp/issues/5 not being resolved in the RBD on QEMU connection. However work is in progress to understand and build on the storage stack. Help welcome.
For the full discussion about Ceph in XCP-ng, see this forum thread: https://xcp-ng.org/forum/topic/4/ceph-on-xcp-ng
Following some working setup and installation guide.
Install EPEL repo
In order to install the needed package we'll need to add extra repos. The first one is EPEL:
yum install epel-release -y --enablerepo=extras
WARNING: Remember to disable EPEL (which is automatically turned on after installation). Because EPEL will conflict with XCP upgrade scripts.
Install Ceph Release Repo
We need to had the repository of the Ceph Version we decided to use.
This can be done by installing the release package
Of course you must install only one version of these repository:
# yum install centos-release-ceph-nautilus --enablerepo=extras # yum install centos-release-ceph-mimic --enablerepo=extras # yum install centos-release-ceph-luminous --enablerepo=extras
Install the Ceph Packages
Install the yum-plugin-priority:
yum install yum-plugin-priorities --enablerepo='base'
Note from the release manager (stormi): I do not advise installing yum-plugin-priorities in dom0, due to possible upgrade issues in the future. It would be good to test whether the procedure really needs this step.
Install the Ceph packages needed:
yum install ceph-common --enablerepo='base,extras,epel'
Note from stormi, still: make sure this doesn't overwrite existing libs or packages from original dom0.
Choose your connection mode Ceph
Warning: Ceph is not yet supported at it's best. Currently there are many concurrent solution in order to plugin. No one of theme are fully tested.
CephFS with Kernel
CephFS is not the best way to connect but it's probably the most easy way to setup. You can read about the patch to apply here: https://xcp-ng.org/forum/topic/1006/ceph-fs-storage-driver/2 Please be aware that on XCP 7.6 with kernel 4.4.0+10 you'll need to downgrade the tunables due to lack features on this kernel. You can downgrade your Ceph Feature Profile by running this command on a ceph-mon:
ceph osd crush tunables hammer
CephFS with Fuse
If you don't want to downgrade your Ceph Cluster Feature than you'll need to connect avoiding Kernel support. However this will decrease your performance: you can do by installing FUSE and Ceph-FUSE support.
yum install ceph-fuse --enablerepo='base,extras,epel,centos-release-ceph-nautilus'
RBD on QEMU
This implementation is currently working on XCP7.5, but broken on XCP7.6 due to the lack of a driver which is not opensource anymore. In order to use Ceph on XCP 7.5 you'll need to install the patched QEMU package:
Install qemu-dp from the
yum install qemu-dp --enablerepo="xcp-ng-extras_testing*"
To go back to the base version of qemu-dp:
yum downgrade qemu-dp
Once installed, note that future updates of
qemu-dpwill get installed from the
baserepository, not from
extras_testing, unless you activate the
extras_testingrepository for updates. So you will have to run the above
yum installcommand again after an update of
qemu-dp. If you choose to activate the
extras_testingrepository for updates, any package that has an experimental version in
extras_testingwill get installed instead of that of
Then install the RBD plugin from this repo: https://github.com/rposudnevskiy/RBDSR
You can follow this discussion about build an open driver for XCP 7.6 here: https://github.com/xcp-ng-rpms/qemu-dp/issues/5