-
Notifications
You must be signed in to change notification settings - Fork 5
Mitaka Openstack: ceph backend configuration
Tutorial Goal: learn how to configure Glance, Nova and Cinder to use Ceph RBD as storage backend.
You need:
- a running Openstack instances
- a running Ceph cluster
Install crudini on Openstack nodes (this tool will be used to edit the configuration files):
sudo apt-get install -y crudini
Install the key and the repo:
wget -q -O- 'https://download.ceph.com/keys/release.asc' | sudo apt-key add -
echo deb http://download.ceph.com/debian-jewel/ $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list
sudo apt-get update
Install python-rbd and ceph-common:
sudo apt-get install -y python-rbd ceph-common
On the nodes running glance-api, cinder-volume, nova-compute copy the ceph.conf file /etc/ceph/ceph.conf
From one of the ceph cluster nodes run the following commands:
ceph osd pool create volumes <pg_num>
ceph osd pool create images <pg_num>
ceph osd pool create backups <pg_num>
ceph osd pool create vms <pg_num>
ceph auth get-or-create client.cinder mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=vms, allow rwx pool=images'
ceph auth get-or-create client.glance mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=images'
ceph auth get-or-create client.cinder-backup mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=backups'
Add the keyrings for client.cinder, client.glance, and client.cinder-backup to the appropriate nodes and change their ownership:
# on the cinder-volume and nova-api servers:
ssh ceph-adm-$GN ceph auth get-or-create client.cinder | sudo tee /etc/ceph/ceph.client.cinder.keyring
sudo chown cinder:cinder /etc/ceph/ceph.client.cinder.keyring
# on the glance-api server:
ssh ceph-adm-$GN ceph auth get-or-create client.glance | sudo tee /etc/ceph/ceph.client.glance.keyring
sudo chown glance:glance /etc/ceph/ceph.client.glance.keyring
Nodes running nova-compute need the keyring file for the nova-compute process:
ssh ceph-adm-$GN ceph auth get-or-create client.cinder | sudo tee /etc/ceph/ceph.client.cinder.keyring
Moreover the nova-compute nodes need to store the secret key of the client.cinder user in libvirt:
ssh ceph-adm-$GN ceph auth get-key client.cinder | tee client.cinder.key
Then, on the compute nodes, add the secret key to libvirt and remove the temporary copy of the key:
uuidgen
457eb676-33da-42ec-9a8c-9293d545c337
cat > secret.xml <<EOF
<secret ephemeral='no' private='no'>
<uuid>457eb676-33da-42ec-9a8c-9293d545c337</uuid>
<usage type='ceph'>
<name>client.cinder secret</name>
</usage>
</secret>
EOF
sudo virsh secret-define --file secret.xml
sudo virsh secret-set-value --secret 457eb676-33da-42ec-9a8c-9293d545c337 --base64 $(cat client.cinder.key)
Edit /etc/glance/glance-api.conf:
[DEFAULT]
...
...
[glance_store]
default_store = rbd
stores = rbd
rbd_store_pool = images
rbd_store_user = glance
rbd_store_ceph_conf = /etc/ceph/ceph.conf
rbd_store_chunk_size = 8
Using crudini:
crudini --set /etc/glance/glance-api.conf glance_store default_store rbd
crudini --set /etc/glance/glance-api.conf glance_store stores rbd
crudini --set /etc/glance/glance-api.conf glance_store rbd_store_pool images
crudini --set /etc/glance/glance-api.conf glance_store rbd_store_user glance
crudini --set /etc/glance/glance-api.conf glance_store rbd_store_ceph_conf /etc/ceph/ceph.conf
crudini --set /etc/glance/glance-api.conf glance_store rbd_store_chunk_size 8
If you want to enable copy-on-write cloning of images, also add under the [DEFAULT] section:
show_image_direct_url = True
using crudini:
crudini --set /etc/glance/glance-api.conf DEFAULT show_image_direct_url True
Disable the Glance cache management to avoid images getting cached under /var/lib/glance/image-cache/, assuming your configuration file has flavor = keystone+cachemanagement:
[paste_deploy]
flavor = keystone
Recommended properties for glance images:
hw_scsi_model=virtio-scsi: add the virtio-scsi controller and get better performance and support for discard operation
hw_disk_bus=scsi: connect every cinder block devices to that controller
hw_qemu_guest_agent=yes: enable the QEMU guest agent
os_require_quiesce=yes: send fs-freeze/thaw calls through the QEMU guest agent
Finally restart glance:
service glance-api restart
service glance-registry restart
OpenStack requires a driver to interact with Ceph block devices. You must also specify the pool name for the block device. Edit /etc/cinder/cinder.conf enabling the new backend rbddriver in the [DEFAULT] section:
enabled_backends=...,rbddriver
[rbddriver]
volume_backend_name=RBD
volume_driver=cinder.volume.drivers.rbd.RBDDriver
rbd_pool=volumes
rbd_ceph_conf=/etc/ceph/ceph.conf
rbd_flatten_volume_from_snapshot=false
rbd_max_clone_depth=5
rbd_store_chunk_size = 4
rados_connect_timeout = -1
glance_api_version = 2
rbd_user=cinder
rbd_secret_uuid=457eb676-33da-42ec-9a8c-9293d545c337
Using crudini:
crudini --set /etc/cinder/cinder.conf rbddriver volume_backend_name RBD
crudini --set /etc/cinder/cinder.conf rbddriver volume_driver cinder.volume.drivers.rbd.RBDDriver
crudini --set /etc/cinder/cinder.conf rbddriver rbd_pool volumes
crudini --set /etc/cinder/cinder.conf rbddriver rbd_ceph_conf /etc/ceph/ceph.conf
crudini --set /etc/cinder/cinder.conf rbddriver rbd_flatten_volume_from_snapshot false
crudini --set /etc/cinder/cinder.conf rbddriver rbd_max_clone_depth 5
crudini --set /etc/cinder/cinder.conf rbddriver rbd_store_chunk_size 4
crudini --set /etc/cinder/cinder.conf rbddriver rados_connect_timeout -1
crudini --set /etc/cinder/cinder.conf rbddriver glance_api_version 2
crudini --set /etc/cinder/cinder.conf rbddriver rbd_user cinder
crudini --set /etc/cinder/cinder.conf rbddriver rbd_secret_uuid 457eb676-33da-42ec-9a8c-9293d545c337
Restart cinder:
service cinder-api restart
service cinder-scheduler restart
service cinder-volume restart
On every compute node, edit /etc/nova/nova.conf and add the following lines in the [DEFAULT] section:
rbd_user=cinder
rbd_secret_uuid=457eb676-33da-42ec-9a8c-9293d545c337
Using crudini:
crudini --set /etc/nova/nova.conf DEFAULT rbd_user cinder
crudini --set /etc/nova/nova.conf DEFAULT rbd_secret_uuid 457eb676-33da-42ec-9a8c-9293d545c337
In order to boot all the virtual machines directly into Ceph, you must configure the ephemeral backend for Nova.
Edit /etc/nova/nova.conf
[libvirt]
images_type = rbd
images_rbd_pool = vms
images_rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_user = cinder
rbd_secret_uuid = 457eb676-33da-42ec-9a8c-9293d545c337
disk_cachemodes="network=writeback"
hw_disk_discard = unmap # enable discard support
live_migration_flag= "VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE,VIR_MIGRATE_PERSIST_DEST,VIR_MIGRATE_TUNNELLED"
It is also a good practice to disable file injection. While booting an instance, Nova usually attempts to open the rootfs of the virtual machine. Then, Nova injects values such as password, ssh keys etc. directly into the filesystem. However, it is better to rely on the metadata service and cloud-init.
Aad the following under the [libvirt] section:
inject_password = false
inject_key = false
inject_partition = -2
Using crudini:
crudini --set /etc/nova/nova.conf libvirt images_type rbd
crudini --set /etc/nova/nova.conf libvirt images_rbd_pool vms
crudini --set /etc/nova/nova.conf libvirt images_rbd_ceph_conf /etc/ceph/ceph.conf
crudini --set /etc/nova/nova.conf libvirt rbd_user cinder
crudini --set /etc/nova/nova.conf libvirt rbd_secret_uuid 457eb676-33da-42ec-9a8c-9293d545c337
crudini --set /etc/nova/nova.conf libvirt disk_cachemodes "network=writeback"
crudini --set /etc/nova/nova.conf libvirt inject_password false
crudini --set /etc/nova/nova.conf libvirt inject_key false
crudini --set /etc/nova/nova.conf libvirt inject_partition -2
crudini --set /etc/nova/nova.conf libvirt hw_disk_discard unmap
crudini --set /etc/nova/nova.conf libvirt live_migration_flag "VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE,VIR_MIGRATE_PERSIST_DEST,VIR_MIGRATE_TUNNELLED"
Restart nova-compute:
service nova-compute restart
Upload an image into glance:
wget https://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-disk1.img
#convert qcow2 to raw:
qemu-img convert -O raw trusty-server-cloudimg-amd64-disk1.img trusty-server-cloudimg-amd64-disk1.raw
#upload the image to glance:
openstack image create --container-format bare --disk-format raw --file trusty-server-cloudimg-amd64-disk1.raw --public ubuntu-trusty-rbd
Check the images pool:
rbd -p images ls
Create a block device on the RBD backend. First of all, you need to define a new volume type associated to the RBD backend:
openstack volume type create --property volume_backend_name=RBD CEPH
Create a volume of type CEPH:
openstack volume create --type CEPH --size 2 testVolume
Check:
root@stack-group-00:~# openstack volume list
+--------------------------------------+--------------+-----------+------+-------------+
| ID | Display Name | Status | Size | Attached to |
+--------------------------------------+--------------+-----------+------+-------------+
| 8c8d1f17-740a-4a8d-a697-805126a8c0b5 | testVolume | available | 2 | |
+--------------------------------------+--------------+-----------+------+-------------+
Check the pool volumes:
rbd -p volumes ls
volume-8c8d1f17-740a-4a8d-a697-805126a8c0b5
Start an instance from the image you have uploaded.
Check the vmspool:
# rbd -p vms ls -l
NAME SIZE PARENT FMT PROT LOCK
30641cb1-4c5d-4443-a458-a6e55bd0ffde_disk 20480M images/22e808de-72a4-43ff-9a40-d6cbe60f2930@snap 2 excl
Get info about the image:
root@ceph-adm-0:~# rbd info vms/30641cb1-4c5d-4443-a458-a6e55bd0ffde_disk
rbd image '30641cb1-4c5d-4443-a458-a6e55bd0ffde_disk':
size 20480 MB in 5120 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.11de782f4706
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
flags:
parent: images/22e808de-72a4-43ff-9a40-d6cbe60f2930@snap
overlap: 2252 MB
Note the copy-on-write cloning of the image.
Enable live snapshot editing /etc/nova/nova.conf:
[workarounds]
disable_libvirt_livesnapshot = False
then restart nova-compute:
service nova-compute restart
Create a snapshot of the VM, then check the images pool:
root@ceph-adm-0:~# rbd -p images ls -l
NAME SIZE PARENT FMT PROT LOCK
0617262a-cdbd-40ab-8571-cb8040b6d965 20480M vms/30641cb1-4c5d-4443-a458-a6e55bd0ffde_disk@883894ec3e9f47d2a55c399d0d3042bb 2 excl
Note: The Mitaka snapshot fails using kernel 3.1x with the following message:
2016-09-30 22:28:41.106 29772 ERROR oslo_messaging.rpc.dispatcher libvirtError: internal error: unable to execute QEMU command 'migrate': Migration disabled: failed to allocate shared memory
On each compute node you have to
-
Enable passwordless login for the user
nova -
update the libvirt configurations. Modify /etc/libvirt/libvirtd.conf as follows:
before : #listen_tls = 0
after : listen_tls = 0
before : #listen_tcp = 1
after : listen_tcp = 1
add: auth_tcp = "none"
- Modify /etc/init/libvirt-bin.conf:
before : exec /usr/sbin/libvirtd -d
after : exec /usr/sbin/libvirtd -d -l
-l is short for –listen
- Modify /etc/default/libvirt-bin:
before :libvirtd_opts=" -d"
after :libvirtd_opts=" -d -l"
Restart libvirt. After executing the command, ensure that libvirt is successfully restarted.
$ stop libvirt-bin && start libvirt-bin
$ ps -ef | grep libvirt