New playbook and roles for installing GlusterFS.

docs/playbooks.rst

@@ -329,30 +329,31 @@ A *snippet* of a typical configuration for an Hadoop cluster is::
 GlusterFS
 =========
 
-Tested on:
+Supported on:
 
-* Ubuntu 12.04
-* CentOS 6.3
+* Ubuntu 14.04 and later
+* RHEL/CentOS 6.x, 7.x
 
 +--------------------+----------------------------------------------------+
-| ansible groups     | role                                               |
+| ansible groups     | action                                             |
 +====================+====================================================+
-| ``gluster_data``   | Run a gluster *brick*                              |
+|``glusterfs_server``| Run a GlusterFS server with a single *brick*       |
 +--------------------+----------------------------------------------------+
-| ``gluster_client`` | Install gluster client and install a gluster       |
-|                    | filesystem on ``/glusterfs``                       |
+|``glusterfs_client``| Install gluster client and (optionally) mount      |
+|                    | a GlusterFS filesystem.                            |
 +--------------------+----------------------------------------------------+
 
-This will install a GlusterFS using all the ``gluster_data`` nodes as
-*bricks*, and any ``gluster_client`` to mount this filesystem in
+This will install a GlusterFS using all the ``glusterfs_server`` nodes
+as servers with a single brick located in directory `/srv/glusterfs`,
+and any ``glusterfs_client`` to mount this filesystem over directory
 ``/glusterfs``.
 
-Setup is very basic, and by default no replicas is set.
+To manage the GlusterFS filesystem you need to connect to a
+``gluster_server`` node.
 
-To manage the gluster filesystem you need to connect to a
-``gluster_data`` node.
-
-Extra variables can be set, by editing the `setup/` section:
+By default the volume is neither replicated nor striped, i.e., replica
+and stripe number is set to 1.  This can be changed by defining the
+following variables in the `setup/` section:
 
 +----------------------+------------+-----------------------------------------+
 | variable name        | default    | description                             |
@@ -362,21 +363,30 @@ Extra variables can be set, by editing the `setup/` section:
 | ``gluster_replicas`` | no replica | set replica value for default volume    |
 +----------------------+------------+-----------------------------------------+
 
-A *snippet* of a typical configuration for a Gluster cluster is::
+The following example configuration sets up a GlusterFS cluster using 8 data nodes
+and providing 2 replicas for each file::
 
     [cluster/gluster]
-    gluster-frontend_nodes=1
-    gluster-data_nodes=8
-    setup_provider=ansible_gluster
+    client_nodes=1
+    server_nodes=8
+    setup_provider=gluster
     ssh_to=gluster-frontend
-    ...
 
-    [setup/ansible_gluster]
-    gluster-frontend_groups=gluster_client
-    gluster-data_groups=gluster_data,gluster_client
-    gluster-data_var_gluster_replicas=2
-    gluster-data_var_gluster_stripes=1
-    ...
+    [setup/gluster]
+    client_groups=glusterfs_client
+    server_groups=glusterfs_server,glusterfs_client
+
+    # set replica and stripe parameters
+    server_var_gluster_replicas=2
+    server_var_gluster_stripes=1
+
+The "GlusterFS" playbook depends on the following Ansible roles being
+available:
+
+* `glusterfs-common <https://github.com/gc3-uzh-ch/elasticluster/tree/master/elasticluster/share/playbooks/roles/glusterfs-common>`_
+* `glusterfs-client <https://github.com/gc3-uzh-ch/elasticluster/tree/master/elasticluster/share/playbooks/roles/glusterfs-client>`_
+* `glusterfs-server <https://github.com/gc3-uzh-ch/elasticluster/tree/master/elasticluster/share/playbooks/roles/glusterfs-server>`_
+
 
 
 OrangeFS/PVFS2

elasticluster/conf.py

@@ -22,6 +22,7 @@
 ])
 
 # System imports
+from collections import defaultdict
 import os
 import re
 import sys
@@ -30,6 +31,7 @@
 except ImportError:
     # Python 3
     StringTypes = (str,)
+import warnings
 
 # External modules
 from ConfigParser import RawConfigParser
@@ -249,6 +251,40 @@ def load_cluster(self, cluster_name):
         )
         return cluster
 
+    @staticmethod
+    def _read_node_groups(conf):
+        """
+        """
+        result = defaultdict(list)
+        for key, value in conf.items():
+            if not key.endswith('_groups'):
+                continue
+            node_kind = key[:-len('_groups')]
+            group_names = [group_name.strip()
+                           for group_name in value.split(',')]
+            for group_name in group_names:
+                # handle renames
+                if group_name in Configurator._renamed_node_groups:
+                    old_group_name = group_name
+                    group_name, remove_at = Configurator._renamed_node_groups[group_name]
+                    warnings.warn(
+                        "Group `{0}` was renamed to `{1}`;"
+                        " please fix your configuration file."
+                        " Support for automatically renaming"
+                        " this group will be removed in {2}."
+                        .format(old_group_name, group_name,
+                                (("ElastiCluster {0}".format(remove_at))
+                                 if remove_at
+                                 else ("a future version of ElastiCluster"))))
+                result[node_kind].append(group_name)
+        return result
+
+    _renamed_node_groups = {
+        # old name     ->  (new name             will be removed in...
+        'gluster_data' :   ('glusterfs_server',  '1.4'),
+        'gluster_client':  ('glusterfs_client',  '1.4'),
+    }
+
     def create_setup_provider(self, cluster_template, name=None):
         """Creates the setup provider for the given cluster template.
 
@@ -273,8 +309,7 @@ def create_setup_provider(self, cluster_template, name=None):
             del conf['playbook_path']
         else:
             playbook_path = None
-        groups = dict((k[:-7], v.split(',')) for k, v
-                      in conf.items() if k.endswith('_groups'))
+        groups = self._read_node_groups(conf)
         environment = dict()
         for nodekind, grps in groups.iteritems():
             if not isinstance(grps, list):

elasticluster/share/playbooks/roles/glusterfs-client/defaults/main.yml

@@ -0,0 +1,23 @@
+---
+
+# A list of GlusterFS filesystems to mount.
+#
+# Each filesystem is defined by an associative array with the
+# following key/value pairs: 
+#
+# - fs: the remote filesystem to mount, in the form "server:volume"
+# - mountpoint: path to the local mountpoint
+# - options: mount options, defaults to `rw,async` if not given
+# - state: see documentation for Ansible module `mount`; the default value here is `mounted`
+#
+# For example::
+#     
+#     GLUSTERFS_MOUNTS:
+#       - fs: 'master:data'
+#         mountpoint: '/data'
+#         options: 'ro'
+#         state: mounted
+#     
+# By default, this parameter is the empty list, i.e., no GlusterFS filesystems are mounted.
+#
+GLUSTERFS_MOUNTS: []

elasticluster/share/playbooks/roles/glusterfs-client/meta/main.yml

@@ -0,0 +1,4 @@
+---
+
+dependencies:
+  - glusterfs-common

elasticluster/share/playbooks/roles/glusterfs-client/tasks/main.yml

@@ -0,0 +1,33 @@
+---
+
+# for once, the package name on Debian/Ubuntu and RHEL/CentOS is the same
+- name: Install GlusterFS client
+  package:
+    name=glusterfs-client
+    state=installed
+  tags:
+    - gluster
+    - glusterfs-client
+
+
+- name: Create mountpoint directory {{item.mountpoint}}
+  file:
+    path='{{item.mountpoint}}'
+    state=directory
+  with_items: '{{GLUSTERFS_MOUNTS}}'
+  tags:
+      - gluster
+      - glusterfs-client
+
+
+- name: Mount GlusterFS filesystem {{item.mountpoint}}
+  mount:
+    src='{{item.fs}}'
+    name='{{item.mountpoint}}'
+    fstype=glusterfs
+    opts='{{item.options|default("defaults")}}'
+    state='{{item.state|default("mounted")}}'
+  with_items: '{{GLUSTERFS_MOUNTS}}'
+  tags:
+      - gluster
+      - glusterfs-client