DRBD Manage Storage Driver
This driver allows for highly available storage using DRBD9 + DRBD Manage in OpenNebula.
To contribute bug patches or new features, you can use the github Pull Request model. It is assumed that code and documentation are contributed under the Apache License 2.0.
Hayley Swimelar email@example.com
- This addon is compatible with OpenNebula versions up to 5.0.2
- It was tested with versions 4.14 and 5.0.2
- This version of README.md describes the installation process for ONE 5.0.2 environments
- This is intended for use as an images datastore for use with an NFS system datastore
- DRBD9 9.0.0+
- DRBD Manage 0.99.2+
- quickly attaches images to VMs
- fast image clones
- transfers images over the network in diskless mode
- snapshots of images are not available
- this driver does not support the ssh system datastore
Follow these steps on the Front-End node only.
Clone the Repository and Run the Install Script.
Run the following commands as either oneadmin or root:
git clone git://git.linbit.com/addon-drbdmanage.git && cd addon-drbdmanage chmod u+x install.sh ./install.sh
To upgrade the driver, simply run the installation script again.
Configure the Driver in OpenNebula
Modify the following sections of
Add drbdmanage to the list of drivers in the
TM_MAD = [ executable = "one_tm", arguments = "-t 15 -d dummy,lvm,shared,fs_lvm,qcow2,ssh,vmfs,ceph,drbdmanage" ]
DATASTORE_MAD = [ EXECUTABLE = "one_datastore", ARGUMENTS = "-t 15 -d dummy,fs,lvm,ceph,dev,iscsi_libvirt,vcenter,drbdmanage -s shared,ssh,ceph,fs_lvm,qcow2"
Add new TM_MAD_CONF and DS_MAD_CONF sections:
TM_MAD_CONF = [ name = "drbdmanage", ln_target = "NONE", clone_target = "SELF", shared = "yes" ]
DS_MAD_CONF = [ NAME = "drbdmanage", REQUIRED_ATTRS = "BRIDGE_LIST", PERSISTENT_ONLY = "NO", MARKETPLACE_ACTIONS = "export" ]
After making these changes, restart the opennebula service.
Configuring the Nodes
Overview of node Roles
The Front-End node issues commands to the Storage and Host nodes via DRBD Manage.
Storage nodes hold disk images of VMs locally.
Host nodes are responsible for running instantiated VMs and typically have the storage for the images they need attached across the network via DRBD Manage diskless mode.
All nodes must have DRBD9 and DRBD Manage installed. This process is detailed in the User's Guide for DRBD9
It is possible to have Front-End and Host nodes act as storage nodes in addition to their primary role as long as they the meet all the requirements for both roles.
If you do not intend for the Front-End or Host nodes to be used as storage nodes in addition to their primary role, they should be added to the DRBD Manage cluster as pure controller nodes.
The Front-End node must be a control node with it's own copy of the control volume, this means that you must provide a small, approximately 4Gb, volume for the drbdpool volume group, even if you do not plan to use this node for DRBD storage.
The Host nodes may also be configured as
pure client nodes
without a local control volume by adding the
--satellite option while adding
the node the the DRBD Manage cluster. This allows hosts to be added without
preparing local storage for DRBD on that node.
Storage Node Configuration
Only the Front-End and Host nodes require OpenNebula to be installed, but the oneadmin user must be able to passwordlessly access storage nodes. Refer to the OpenNebula install guide for your distribution on how to manually configure the oneadmin user account.
The Storage nodes must use one of the storage plugins that support snapshots, this means one of the thin LVM plugins or ZFS. The merits of the different plugins are discussed in the User's Guide.
In this example preparation of thinly-provisioned storage using LVM for DRBD Manage, you must create a volume group and thinLV using LVM on each storage node.
Example of this process using two physical volumes (/dev/sdX and /dev/sdY) and the default names for the volume group and thinpool. Make sure to set the thinLV's metadata volume to a reasonable size, once it becomes full it can be difficult to resize:
pvcreate /dev/sdX /dev/sdY vgcreate drbdpool /dev/sdX /dev/sdY lvcreate -l 95%VG --poolmetadatasize 8g -T /dev/drbdpool/drbdthinpool
Instructions on how to configure DRBD Manage to use a storage plugin can be found in the cluster configuration section of the User's Guide.
Additonal Driver Configuration
Additional configuration for the driver can be found in the
datastore/drbdmanage.conf file in the driver director or in the install path,
Permissions for Oneadmin
The oneadmin user must have passwordless sudo access to the
drbdmanage program on the Front-End node
oneadmin ALL=(root) NOPASSWD: /usr/bin/drbdmanage
mkfs command on the Storage nodes
oneadmin ALL=(root) NOPASSWD: /sbin/mkfs
A policy section for the oneadmin user must also be added in
/etc/dbus-1/system.d/org.drbd.drbdmanaged.conf on the Front-End node. Be
sure to leave the original policy section intact!
<!DOCTYPE busconfig PUBLIC "-//freedesktop//DTD D-BUS Bus Configuration 1.0//EN" "http://www.freedesktop.org/standards/dbus/1.0/busconfig.dtd"> <busconfig> <policy user="0"> <allow own="org.drbd.drbdmanaged"/> <allow send_interface="org.drbd.drbdmanaged"/> <allow send_destination="org.drbd.drbdmanaged"/> </policy> <policy user="oneadmin"> <allow own="org.drbd.drbdmanaged"/> <allow send_interface="org.drbd.drbdmanaged"/> <allow send_destination="org.drbd.drbdmanaged"/> </policy> </busconfig>
Be sure to consider the groups that oneadmin should be added to in order to
gain access to the devices and programs needed to access storage and
instantiate VMs. For this addon, the oneadmin user must belong to the
group on all nodes in order to access the DRBD devices where images are held.
usermod -a -G disk oneadmin
Creating a New DRBD Manage Datastore
Create a datastore configuration file named ds.conf and use the
tool to create a new datastore based on that configuration. There are two
mutually exclusive deployment options: DRBD_REDUNDANCY and
DRBD_DEPLOYMENT_NODES. For both of these options, BRIDGE_LIST must be a space
separated list of all storage nodes in the DRBD Manage cluster.
Deploying to a Redundancy Level
The DRBD_REDUNDANCY option takes a level of redundancy which is a number between one and the total number of storage nodes. Resources are assigned to storage nodes automatically based on the level of redundancy and DRBD Manage's deployment policy. The following example shows a cluster with three storage nodes that will deploy new resources to two of the nodes in the BRIDGE_LIST based on the free space available on the storage nodes.
cat >ds.conf <<EOI NAME = drbdmanage_redundant DS_MAD = drbdmanage TM_MAD = drbdmanage DRBD_REDUNDANCY = 2 BRIDGE_LIST = "alice bob charlie" EOI onedatastore create ds.conf
Deploying to a List of Nodes
Using DRBD_DEPLOYMENT_NODES allows you to select a group of nodes that resources will always be assigned to. In the following example, new resources will always be assigned to the nodes alice and charlie. Please note that the bridge list still contains all of the storage nodes in the DRBD Manage cluster.
cat >ds.conf <<EOI NAME = drbdmanage_nodes DS_MAD = drbdmanage TM_MAD = drbdmanage DRBD_DEPLOYMENT_NODES = "alice charlie" BRIDGE_LIST = "alice bob charlie" EOI onedatastore create ds.conf
Restricting Where Resources Can Be Deployed
Using DRBD_DEPLOYMENT_SITE allows you to select a site defined in DRBD MANAGE to restrict deployment of resources. This optional setting works in tandem with either DRBD_DEPLOYMENT_NODES or DRBD_REDUNDANCY. When deploying to a redundancy level, only nodes within the site are considered when deciding which nodes to deploy on. When deploying to a list of nodes, this option blocks deployment to nodes listed in DRBD_DEPLOYMENT_NODES that are not also in the site.
cat >ds.conf <<EOI NAME = drbdmanage_nodes DS_MAD = drbdmanage TM_MAD = drbdmanage DRBD_REDUNDANCY = 2 BRIDGE_LIST = "alice bob charlie" DRBD_DEPLOYMENT_SITE = "alpha" EOI onedatastore create ds.conf
There are three additional attributes that you may add to a datastore's
template. These can be used to overwrite the options of the same name in the
DRBD_MIN_COUNT is the minimum number of nodes that a resource must be deployed on for the deployment of a new resource to be considered a success. This should be an integer between 0 and the total number of storage nodes in your DRBD Manage cluster.
DRBD_MIN_RATIO is the ratio of nodes a resource must be deployed on for the deployment of a new resource to be considered a success. This should be a number between 0.0 and 1.0.
DRBD_SUPPORT_LIVE_MIGRATION enables the live migration of VMs. Valid options are "yes" and "no" (default). If this option is enabled, you must also configure your DRBD Manage cluster to allow dual primary. To do this, run the following command:
drbdmanage net-options --allow-two-primaries yes --common
Please note that images that were created before live migration support has been enabled may not be available to all hosts. These images may be assigned to hosts after the fact using DRBD Manage directly.
Validating a Datastore Configuration
It is recommended to validate your configuration using the validation tool provided with the driver. The tool will report any errors it finds in your configuration. Simply pass the configuration file you wish to validate as an argument.
This driver will use DRBD Manage to create new images and transfer them to hosts. Images are attached to hosts across the network using diskless mode. Images are replicated according to the deployment policy set in the datastore template.
##DRBD9 and DRBD Manage User's Guide
If you have any questions about setting up, tuning, or administrating DRBD9 or DRBD Manage, be sure to check out the information provided in the User's Guide