Adding new node in slurm DeepOPS cluster failing

[karanveersingh5623]

This -l option is not working when adding a new node to Slurm Cluster
Below is the error trace

(env) (base) [root@adas-ml deepops]# ansible-playbook -l adas-ml-2 playbooks/slurm-cluster.yml -vvv

TASK [DeepOps.hosts : Install custom hosts file] ***************************************************************************************************************
task path: /root/deepops/roles/galaxy/DeepOps.hosts/tasks/main.yml:23
[WARNING]: conditional statements should not include jinja2 templating delimiters such as {{ }} or {% %}. Found: {{ slurm_configure_etc_hosts | default(true)
}}
<192.168.61.3> ESTABLISH SSH CONNECTION FOR USER: None
<192.168.61.3> SSH: EXEC ssh -o ControlMaster=auto -o ControlPersist=5m -o ConnectionAttempts=100 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=60 -o 'ControlPath=~/.ssh/ansible-%r@%h:%p' 192.168.61.3 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /tmp `"&& mkdir "` echo /tmp/ansible-tmp-1654670502.5229077-43700-14549810058325 `" && echo ansible-tmp-1654670502.5229077-43700-14549810058325="` echo /tmp/ansible-tmp-1654670502.5229077-43700-14549810058325 `" ) && sleep 0'"'"''
<192.168.61.3> (0, b'ansible-tmp-1654670502.5229077-43700-14549810058325=/tmp/ansible-tmp-1654670502.5229077-43700-14549810058325\n', b'')
<192.168.61.3> ESTABLISH SSH CONNECTION FOR USER: None
<192.168.61.3> SSH: EXEC ssh -o ControlMaster=auto -o ControlPersist=5m -o ConnectionAttempts=100 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=60 -o 'ControlPath=~/.ssh/ansible-%r@%h:%p' 192.168.61.3 '/bin/sh -c '"'"'rm -f -r /tmp/ansible-tmp-1654670502.5229077-43700-14549810058325/ > /dev/null 2>&1 && sleep 0'"'"''
<192.168.61.3> (0, b'', b'')
fatal: [adas-ml-2]: FAILED! => changed=false
  msg: 'AnsibleUndefinedVariable: ''dict object'' has no attribute ''adas-ml-2'''

PLAY RECAP *****************************************************************************************************************************************************
adas-ml-2                  : ok=19   changed=0    unreachable=0    failed=1    skipped=7    rescued=0    ignored=1

[karanveersingh5623]

@ajdecon
Please let me know if you need more information .

[ajdecon]

To add a new node, we generally recommend re-running the slurm-cluster.yml playbook across the full cluster, rather than just on the new node.

We recommend this because some of the playbooks need to gather information from all hosts in the cluster in order to generate configuration files. The hosts file playbook is a case in point, and this is the one you ran into.

If you want to be able to add hosts without a full-cluster Ansible run, you can work around this by following the documentation for static configuration files on large clusters.

However, this does require you to manually manage the contents of these files, rather than relying on DeepOps to generate these for you.

[karanveersingh5623]

@ajdecon , thanks for update