Add support for autoscaling

.github/workflows/ci.yml

@@ -43,6 +43,8 @@ jobs:
           - test10
           - test11
           - test12
+          - test13
+          - test14
 
         exclude:
           - image: 'centos:7'
@@ -61,6 +63,10 @@ jobs:
             scenario: test11
           - image: 'centos:7'
             scenario: test12
+          - image: 'centos:7'
+            scenario: test13
+          - image: 'centos:7'
+            scenario: test14
 
     steps:
       - name: Check out the codebase.

README.md

@@ -39,32 +39,35 @@ package in the image.
 
 `openhpc_module_system_install`: Optional, default true. Whether or not to install an environment module system. If true, lmod will be installed. If false, You can either supply your own module system or go without one.
 
-`openhpc_ram_multiplier`: Optional, default `0.95`. Multiplier used in the calculation: `total_memory * openhpc_ram_multiplier` when setting `RealMemory` for the partition in slurm.conf. Can be overriden on a per partition basis using `openhpc_slurm_partitions.ram_multiplier`. Has no effect if `openhpc_slurm_partitions.ram_mb` is set.
-
 ### slurm.conf
 
 `openhpc_slurm_partitions`: list of one or more slurm partitions.  Each partition may contain the following values:
 * `groups`: If there are multiple node groups that make up the partition, a list of group objects can be defined here.
   Otherwise, `groups` can be omitted and the following attributes can be defined in the partition object:
   * `name`: The name of the nodes within this group.
   * `cluster_name`: Optional.  An override for the top-level definition `openhpc_cluster_name`.
+  * `extra_nodes`: Optional. A list of additional node definitions, e.g. for nodes in this group/partition not controlled by this role. Each item should be a dict, with keys/values as per the ["NODE CONFIGURATION"](https://slurm.schedmd.com/slurm.conf.html#lbAE) docs for slurm.conf. Note the key `NodeName` must be first.
   * `ram_mb`: Optional.  The physical RAM available in each server of this group ([slurm.conf](https://slurm.schedmd.com/slurm.conf.html) parameter `RealMemory`) in MiB. This is set using ansible facts if not defined, equivalent to `free --mebi` total * `openhpc_ram_multiplier`.
-
-  For each group (if used) or partition there must be an ansible inventory group `<cluster_name>_<group_name>`, with all nodes in this inventory group added to the group/partition. Note that:
-  - Nodes may have arbitrary hostnames but these should be lowercase to avoid a mismatch between inventory and actual hostname.
-  - Nodes in a group are assumed to be homogenous in terms of processor and memory.
-  - An inventory group may be empty, but if it is not then the play must contain at least one node from it (used to set processor information).
-* `ram_multiplier`: Optional.  An override for the top-level definition `openhpc_ram_multiplier`. Has no effect if `ram_mb` is set.
+  * `ram_multiplier`: Optional.  An override for the top-level definition `openhpc_ram_multiplier`. Has no effect if `ram_mb` is set.
 * `default`: Optional.  A boolean flag for whether this partion is the default.  Valid settings are `YES` and `NO`.
 * `maxtime`: Optional.  A partition-specific time limit in hours, minutes and seconds ([slurm.conf](https://slurm.schedmd.com/slurm.conf.html) parameter `MaxTime`).  The default value is
   given by `openhpc_job_maxtime`.
 
+For each group (if used) or partition any nodes in an ansible inventory group `<cluster_name>_<group_name>` will be added to the group/partition. Note that:
+- Nodes may have arbitrary hostnames but these should be lowercase to avoid a mismatch between inventory and actual hostname.
+- Nodes in a group are assumed to be homogenous in terms of processor and memory.
+- An inventory group may be empty, but if it is not then the play must contain at least one node from it (used to set processor information).
+- Nodes may not appear in more than one group.
+- A group/partition definition which does not have either a corresponding inventory group or a `extra_nodes` will raise an error.
+
 `openhpc_job_maxtime`: A maximum time job limit in hours, minutes and seconds.  The default is `24:00:00`.
 
 `openhpc_cluster_name`: name of the cluster
 
 `openhpc_config`: Mapping of additional parameters and values for `slurm.conf`. Note these will override any included in `templates/slurm.conf.j2`.
 
+`openhpc_ram_multiplier`: Optional, default `0.95`. Multiplier used in the calculation: `total_memory * openhpc_ram_multiplier` when setting `RealMemory` for the partition in slurm.conf. Can be overriden on a per partition basis using `openhpc_slurm_partitions.ram_multiplier`. Has no effect if `openhpc_slurm_partitions.ram_mb` is set.
+
 `openhpc_state_save_location`: Optional. Absolute path for Slurm controller state (`slurm.conf` parameter [StateSaveLocation](https://slurm.schedmd.com/slurm.conf.html#OPT_StateSaveLocation))
 
 #### Accounting

defaults/main.yml

@@ -12,6 +12,8 @@ openhpc_resume_timeout: 300
 openhpc_retry_delay: 10
 openhpc_job_maxtime: 24:00:00
 openhpc_config: "{{ openhpc_extra_config | default({}) }}"
+openhpc_slurm_configless: "{{ 'enable_configless' in openhpc_config.get('SlurmctldParameters', []) }}"
+
 openhpc_state_save_location: /var/spool/slurm
 
 # Accounting
@@ -49,7 +51,6 @@ ohpc_slurm_services:
 ohpc_release_repos:
   "7": "https://github.com/openhpc/ohpc/releases/download/v1.3.GA/ohpc-release-1.3-1.el7.x86_64.rpm" # ohpc v1.3 for Centos 7
   "8": "http://repos.openhpc.community/OpenHPC/2/CentOS_8/x86_64/ohpc-release-2-1.el8.x86_64.rpm" # ohpc v2 for Centos 8
-openhpc_slurm_configless: false
 openhpc_munge_key:
 openhpc_login_only_nodes: ''
 openhpc_module_system_install: true

filter_plugins/group_hosts.py → filter_plugins/slurm_conf.py

@@ -12,6 +12,8 @@
 # License for the specific language governing permissions and limitations
 # under the License.
 
+# NB: To test this from the repo root run:
+#   ansible-playbook -i tests/inventory -i tests/inventory-mock-groups tests/filter.yml
 
 from ansible import errors
 import jinja2
@@ -30,11 +32,25 @@ def _get_hostvar(context, var_name, inventory_hostname=None):
         namespace = context["hostvars"][inventory_hostname]
     return namespace.get(var_name)
 
-@jinja2.contextfilter
-def group_hosts(context, group_names):
-    return {g:_group_hosts(context["groups"].get(g, [])) for g in sorted(group_names)}
+def hostlist_expression(hosts):
+    """ Group hostnames using Slurm's hostlist expression format.
+
+        E.g. with an inventory containing:
+
+            [compute]
+            dev-foo-0 ansible_host=localhost
+            dev-foo-3 ansible_host=localhost
+            my-random-host
+            dev-foo-4 ansible_host=localhost
+            dev-foo-5 ansible_host=localhost
+            dev-compute-0 ansible_host=localhost
+            dev-compute-1 ansible_host=localhost
+
+        Then "{{ groups[compute] | hostlist_expression }}" will return:
+            
+            ["dev-foo-[0,3-5]", "dev-compute-[0-1]", "my-random-host"]
+    """
 
-def _group_hosts(hosts):
     results = {}
     unmatchable = []
     for v in hosts:
@@ -58,9 +74,16 @@ def _group_numbers(numbers):
         prev = v
     return ','.join(['{}-{}'.format(u[0], u[-1]) if len(u) > 1 else str(u[0]) for u in units])
 
+def error(condition, msg):
+    """ Raise an error if condition is not True """
+
+    if not condition:
+        raise errors.AnsibleFilterError(msg)
+
 class FilterModule(object):
 
     def filters(self):
         return {
-            'group_hosts': group_hosts
+            'hostlist_expression': hostlist_expression,
+            'error': error,
         }

molecule/README.md

@@ -21,6 +21,7 @@ test10 | 1            | N                       | As for #5 but then tries to ad
 test11 | 1            | N                       | As for #5 but then deletes a node (actually changes the partition due to molecule/ansible limitations)
 test12 | 1            | N                       | As for #5 but enabling job completion and testing `sacct -c`
 test13 | 1            | N                       | As for #5 but tests `openhpc_config` variable.
+test14 | 1            |                         | As for #5 but also tests `extra_nodes` via State=DOWN nodes.
 
 # Local Installation & Running
 

molecule/test13/verify.yml

@@ -13,8 +13,8 @@
     command: scontrol show config
     register: slurm_config
   - assert:
-      that: "item in slurm_config.stdout"
+      that: "item in (slurm_config.stdout_lines | map('replace', ' ', ''))"
       fail_msg: "FAILED - {{ item }} not found in slurm config"
     loop:
       - SlurmctldSyslogDebug=error
-      - SlurmctldSyFirstJobId=13
+      - FirstJobId=13

molecule/test14/converge.yml

@@ -0,0 +1,30 @@
+---
+- name: Converge
+  hosts: all
+  tasks:
+    - name: "Include ansible-role-openhpc"
+      include_role:
+        name: "{{ lookup('env', 'MOLECULE_PROJECT_DIRECTORY') | basename }}"
+      vars:
+        openhpc_enable:
+          control: "{{ inventory_hostname in groups['testohpc_login'] }}"
+          batch: "{{ inventory_hostname in groups['testohpc_compute'] }}"
+          runtime: true
+        openhpc_slurm_control_host: "{{ groups['testohpc_login'] | first }}"
+        openhpc_slurm_partitions:
+          - name: "compute"
+            extra_nodes:
+              # Need to specify IPs for the non-existent State=DOWN nodes, because otherwise even in this state slurmctld will exclude a node with no lookup information from the config.
+              # We use invalid IPs here (i.e. starting 0.) to flag the fact the nodes shouldn't exist.
+              # Note this has to be done via slurm config rather than /etc/hosts due to Docker limitations on modifying the latter.
+              - NodeName: fake-x,fake-y
+                NodeAddr: 0.42.42.0,0.42.42.1
+                State: DOWN
+                CPUs: 1
+              - NodeName: fake-2cpu-[3,7-9]
+                NodeAddr: 0.42.42.3,0.42.42.7,0.42.42.8,0.42.42.9
+                State: DOWN
+                CPUs: 2
+        openhpc_cluster_name: testohpc
+        openhpc_slurm_configless: true
+

molecule/test14/molecule.yml

@@ -0,0 +1,60 @@
+---
+name: single partition, group is partition
+driver:
+  name: docker
+platforms:
+  - name: testohpc-login-0
+    image: ${MOLECULE_IMAGE}
+    pre_build_image: true
+    groups:
+      - testohpc_login
+    command: /sbin/init
+    tmpfs:
+      - /run
+      - /tmp
+    volumes:
+      - /sys/fs/cgroup:/sys/fs/cgroup:ro
+    networks:
+      - name: net1
+    docker_networks:
+      - name: net1
+        driver_options:
+          com.docker.network.driver.mtu: ${DOCKER_MTU:-1500} # 1500 is docker default
+  - name: testohpc-compute-0
+    image: ${MOLECULE_IMAGE}
+    pre_build_image: true
+    groups:
+      - testohpc_compute
+    command: /sbin/init
+    tmpfs:
+      - /run
+      - /tmp
+    volumes:
+      - /sys/fs/cgroup:/sys/fs/cgroup:ro
+    networks:
+      - name: net1
+    docker_networks:
+      - name: net1
+        driver_options:
+          com.docker.network.driver.mtu: ${DOCKER_MTU:-1500} # 1500 is docker default
+  - name: testohpc-compute-1
+    image: ${MOLECULE_IMAGE}
+    pre_build_image: true
+    groups:
+      - testohpc_compute
+    command: /sbin/init
+    tmpfs:
+      - /run
+      - /tmp
+    volumes:
+      - /sys/fs/cgroup:/sys/fs/cgroup:ro
+    networks:
+      - name: net1
+    docker_networks:
+      - name: net1
+        driver_options:
+          com.docker.network.driver.mtu: ${DOCKER_MTU:-1500} # 1500 is docker default
+provisioner:
+  name: ansible
+verifier:
+  name: ansible

molecule/test14/verify.yml

@@ -0,0 +1,12 @@
+---
+
+- name: Check slurm hostlist
+  hosts: testohpc_login
+  tasks:
+  - name: Get slurm partition info
+    command: sinfo --noheader --format="%P,%a,%l,%D,%t,%N" # using --format ensures we control whitespace
+    register: sinfo
+  - name: 
+    assert:                        # PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
+      that: "sinfo.stdout_lines == ['compute*,up,60-00:00:00,6,down*,fake-2cpu-[3,7-9],fake-x,fake-y', 'compute*,up,60-00:00:00,2,idle,testohpc-compute-[0-1]']"
+      fail_msg: "FAILED - actual value: {{ sinfo.stdout_lines }}"

molecule/test5/molecule.yml

@@ -16,6 +16,10 @@ platforms:
       - /sys/fs/cgroup:/sys/fs/cgroup:ro
     networks:
       - name: net1
+    docker_networks:
+      - name: net1
+        driver_options:
+          com.docker.network.driver.mtu: ${DOCKER_MTU:-1500} # 1500 is docker default
   - name: testohpc-compute-0
     image: ${MOLECULE_IMAGE}
     pre_build_image: true
@@ -29,6 +33,10 @@ platforms:
       - /sys/fs/cgroup:/sys/fs/cgroup:ro
     networks:
       - name: net1
+    docker_networks:
+      - name: net1
+        driver_options:
+          com.docker.network.driver.mtu: ${DOCKER_MTU:-1500} # 1500 is docker default
   - name: testohpc-compute-1
     image: ${MOLECULE_IMAGE}
     pre_build_image: true
@@ -42,6 +50,10 @@ platforms:
       - /sys/fs/cgroup:/sys/fs/cgroup:ro
     networks:
       - name: net1
+    docker_networks:
+      - name: net1
+        driver_options:
+          com.docker.network.driver.mtu: ${DOCKER_MTU:-1500} # 1500 is docker default
 provisioner:
   name: ansible
 verifier:

molecule/test6/molecule.yml

@@ -16,6 +16,10 @@ platforms:
       - /sys/fs/cgroup:/sys/fs/cgroup:ro
     networks:
       - name: net1
+    docker_networks:
+      - name: net1
+        driver_options:
+          com.docker.network.driver.mtu: ${DOCKER_MTU:-1500} # 1500 is docker default
 provisioner:
   name: ansible
   inventory:

tasks/runtime.yml

@@ -58,6 +58,7 @@
     state: touch
     owner: slurm
     group: slurm
+    mode: 0644
     access_time: preserve
     modification_time: preserve
   when: openhpc_slurm_job_comp_type == 'jobcomp/filetxt'
@@ -85,6 +86,7 @@
     src: slurm.conf.j2
     dest: "{{ _slurm_conf_tmpfile.path }}"
     lstrip_blocks: true
+    mode: 0644
   delegate_to: localhost
   when: openhpc_enable.control | default(false) or not openhpc_slurm_configless
   changed_when: false # so molecule doesn't fail
@@ -95,7 +97,7 @@
     path: "{{ _slurm_conf_tmpfile.path }}"
     option: "{{ item.key }}"
     section: null
-    value: "{{ item.value }}"
+    value: "{{ (item.value | join(',')) if (item.value is sequence and item.value is not string) else item.value }}"
     no_extra_spaces: true
     create: no
     mode: 0644

templates/slurm.conf.j2

@@ -106,39 +106,34 @@ NodeName={{ node }}
 PropagateResourceLimitsExcept=MEMLOCK
 Epilog=/etc/slurm/slurm.epilog.clean
 {% for part in openhpc_slurm_partitions %}
+    {% set nodelist = [] %}
     {% for group in part.get('groups', [part]) %}
+
         {% set group_name = group.cluster_name|default(openhpc_cluster_name) ~ '_' ~ group.name %}
-        {% if groups[group_name] | length > 0 %}
-            {# If using --limit, the first host in each group may not have facts available. Find one that does, or die: #}
-            {% set group_hosts = groups[group_name] | intersect(play_hosts) %}
-            {% set first_host = group_hosts | first | mandatory('Group "' + group_name + '" contains no hosts in this play - was --limit used?') %}
+# openhpc_slurm_partitions group: {{ group_name }}
+        {% set inventory_group_hosts = groups.get(group_name, []) %}
+        {% if inventory_group_hosts | length > 0 %}
+            {% set play_group_hosts = inventory_group_hosts | intersect (play_hosts) %}
+            {% set first_host = play_group_hosts | first | mandatory('Group "' ~ group_name ~ '" contains no hosts in this play - was --limit used?') %}
             {% set first_host_hv = hostvars[first_host] %}
-
-NodeName=DEFAULT State=UNKNOWN \
-    RealMemory={% if 'ram_mb' in group %}{{group.ram_mb}}{% else %}{{ (first_host_hv['ansible_memory_mb']['real']['total'] * group.ram_multiplier | default(openhpc_ram_multiplier)) | int }}{% endif %} \
-    Sockets={{first_host_hv['ansible_processor_count']}} \
-    CoresPerSocket={{first_host_hv['ansible_processor_cores']}} \
-    ThreadsPerCore={{first_host_hv['ansible_processor_threads_per_core']}}
-            {% for node in groups[group_name] %}
-NodeName={{ node }}
+            {% set ram_mb = (first_host_hv['ansible_memory_mb']['real']['total'] * (group.ram_multiplier | default(openhpc_ram_multiplier))) | int %}
+            {% for hostlist in (inventory_group_hosts | hostlist_expression) %}
+NodeName={{ hostlist }} State=UNKNOWN RealMemory={{ group.get('ram_mb', ram_mb) }} Sockets={{first_host_hv['ansible_processor_count']}} CoresPerSocket={{ first_host_hv['ansible_processor_cores'] }} ThreadsPerCore={{ first_host_hv['ansible_processor_threads_per_core'] }}
+                {% set _ = nodelist.append(hostlist) %}
             {% endfor %}{# nodes #}
-        {% else %}
-NodeName=-nonesuch
-        {% endif %}{# non-empty group #}
+        {% endif %}{# inventory_group_hosts #}
+        {% for extra_node_defn in group.get('extra_nodes', []) %}
+{{ extra_node_defn.items() | map('join', '=') | join(' ') }}
+            {% set _ = nodelist.append(extra_node_defn['NodeName']) %}
+        {% endfor %}
     {% endfor %}{# group #}
-PartitionName={{part.name}} \
-    Default={% if 'default' in part %}{{ part.default }}{% else %}YES{% endif %} \
-    MaxTime={% if 'maxtime' in part %}{{ part.maxtime }}{% else %}{{ openhpc_job_maxtime }}{% endif %} \
-    State=UP \
-    Nodes=\
-        {% for group in part.get('groups', [part]) %}
-            {% set group_name = group.cluster_name|default(openhpc_cluster_name) ~ '_' ~ group.name %}
-            {% if groups[group_name] | length > 0 %}{{ groups[group_name] | join(",\\\n") }}{% if not loop.last %}{{ ",\\\n" }}{% endif %}{% else %}-nonesuch{% endif %}
-        {% endfor %}{# group #}
+{% if not nodelist %}  {# empty partition - define an invalid hostname which slurm accepts #}
+    {% set nodelist = ['-nonesuch'] %}
+NodeName={{ nodelist[0] }}
+{% endif %}
+PartitionName={{part.name}} Default={{ part.get('default', 'YES') }} MaxTime={{ part.get('maxtime', openhpc_job_maxtime) }} State=UP Nodes={{ nodelist | join(',') }}
 {% endfor %}{# partitions #}
 
 # Want nodes that drop out of SLURM's configuration to be automatically
 # returned to service when they come back.
 ReturnToService=2
-
-# Parameters from openhpc_config (which do not override values templated above) will be below here: