[Feature Request] hostname in .ini

[zeekus]

It would be nice feature if a hostname could be set from the [global] section of the ini file.

Setting a hostname in pcluster is bit more complicated then just doing a 'hostnamectl set-hostname'.

It seems slurm doesn't always work as expected after changes to the hostname.

[lukeseawalker]

The node names are a critical part of the system and our logic rely on hostname to determine the queue, node type, and instance type information of a node.

Can you please provide more details on your use case?

Thanks,
L

[zeekus]

Summary of issue:

As I write this I wonder if this is a user error. I built a my prototype system with a post_install.sh setting the hostname.

Upon the initial boot, everything worked. After a restart of slurm, the hostname change started becoming an issue.

Details on first runs:

My test was:

srun -n 2 'hostname -f'

Here is what the post_install.sh looked like:

if [[ ${cfn_node_type} == MasterServer ]]; then

  #add the hostname - only works in 'post install' cloud-init overwrites if in pre-install 
  cluster_name=example
  hostnamectl set-hostname $cluster_name
  echo 127.0.0.1 $cluster_name.us-east.aws.example.net $cluster_name.pcluster $cluster_name >> /etc/hosts

After the initial boot, we added the following changes to slurm:

  #EnforcePartLimits changed to NO
  sed -i 's/EnforcePartLimits=ALL/EnforcePartLimits=NO/g' /opt/slurm/etc/slurm.conf
  #Memory value added 
  sed -i 's/CPUs=36 State=CLOUD/CPUs=36 RealMemory=60211 State=CLOUD/g' /opt/slurm/etc/pcluster/slurm_parallelcluster_compute_partition.conf

Upon a slurm restart, slurm was complaining the hostname 'example' is not the master.
So, my colleague, reset the host-name to the original host-name of 'ip-10-128-64-218'. It appears slurm reverted to the original cloud-init hostname after a restart.

His summary of events:

1. Hostname changed by post_install
2. adjusted the slurm compute node config
   attempted to restart slurm, then tried a reboot after it failed.
   "not the master errors continue"
3. Manually changed hostname back to original 'ip-10-128-64-218'.
4. restarted slurm success
5. reboot successful

Our assumption was we can't change the hostname without breaking slurm.

PS.
I don't think the cluster was stopped before making these changes

[zeekus]

possible reason:

Maybe this holds the original hostname:
File: /opt/slurm/etc/slurm_parallelclsuter.conf

# slurm_parallelcluster.conf is managed by the pcluster processes.
# Do not modify.
# Please add user-specific slurm configuration options in slurm.conf

SlurmctldHost=ip-10-128-64-218(10.128.64.218)
SuspendTime=600

include /opt/slurm/etc/pcluster/slurm_parallelcluster_compute_partition.conf

[zeekus]

i manually changed the hostname in attempt to get the errors.

It appears slurm gets all confused.

My change:
1. stopped cluster
2. created file /etc/slurm/etc/slurm_hostname_override.conf
2. appended slurm_hostname_override.conf to the last line of slurm.conf

           include slurm_hostname_override.conf

contents of slurm_hostname_override.conf

SlurmctldHost=bluefish(10.128.64.218)

 5. changed hostname

hostnamectl set-hostname bluefish

tknab@bluefish$ sudo cat /var/log/slurmctld.log | grep -i host
[2021-05-20T17:30:10.365] error: This host (bluefish/bluefish) not a valid controller
[2021-05-20T17:46:38.751] error: _ping_ctld_thread: BackupController# index mismatch (0 != 1) from host ip-10-128-64-218
[2021-05-20T17:48:18.769] error: _ping_ctld_thread: BackupController# index mismatch (0 != 1) from host ip-10-128-64-218
[2021-05-20T17:49:58.789] error: _ping_ctld_thread: BackupController# index mismatch (0 != 1) from host ip-10-128-64-218
[2021-05-20T17:51:38.808] error: _ping_ctld_thread: BackupController# index mismatch (0 != 1) from host ip-10-128-64-218
[2021-05-20T17:53:18.827] error: _ping_ctld_thread: BackupController# index mismatch (0 != 1) from host ip-10-128-64-218
[2021-05-20T17:54:58.847] error: _ping_ctld_thread: BackupController# index mismatch (0 != 1) from host ip-10-128-64-218
[2021-05-20T17:56:38.866] error: _ping_ctld_thread: BackupController# index mismatch (0 != 1) from host ip-10-128-64-218
[2021-05-20T17:58:18.885] error: _ping_ctld_thread: BackupController# index mismatch (0 != 1) from host ip-10-128-64-218
[2021-05-20T17:59:58.904] error: _ping_ctld_thread: BackupController# index mismatch (0 != 1) from host ip-10-128-64-218
[2021-05-20T18:01:38.924] error: _ping_ctld_thread: BackupController# index mismatch (0 != 1) from host ip-10-128-64-218
[2021-05-20T18:02:58.213] error: _ping_ctld_thread: BackupController# index mismatch (0 != 1) from host ip-10-128-64-218
[2021-05-20T18:04:38.232] error: _ping_ctld_thread: BackupController# index mismatch (0 != 1) from host ip-10-128-64-218
[2021-05-20T18:06:18.252] error: _ping_ctld_thread: BackupController# index mismatch (0 != 1) from host ip-10-128-64-218
[2021-05-20T18:07:58.271] error: _ping_ctld_thread: BackupController# index mismatch (0 != 1) from host ip-10-128-64-218
[2021-05-20T18:08:39.551] error: This host (bluefish/bluefish) not a valid controller
[2021-05-20T18:08:54.734] error: This host (bluefish/bluefish) not a valid controller
[2021-05-20T18:09:44.415] error: _ping_ctld_thread: BackupController# index mismatch (0 != 1) from host ip-10-128-64-218
[2021-05-20T18:12:54.553] error: Unable to resolve "compute-dy-c48xlarge-2": Unknown host

[zeekus]

Could this be related to munge ?

root@bluefish$ munge -n | unmunge
STATUS: Success (0)
ENCODE_HOST: ip-10-128-64-218.ec2.internal (10.128.64.218)
ENCODE_TIME: 2021-05-20 18:48:45 +0000 (1621536525)
DECODE_TIME: 2021-05-20 18:48:45 +0000 (1621536525)
TTL: 300
CIPHER: aes128 (4)
MAC: sha256 (5)
ZIP: none (0)
UID: root (0)
GID: root (0)
LENGTH: 0

[demartinofra]

Hi @zeekus sorry for the delayed response. Were you able to make any progress on customizing the headnode hostname?

[jwillis0720]

@demartinofra is using post_install to add to /opt/slurm/etc/slurm.conf the recommended way to dynamically add more options to slurm.conf on pcluster create? In my case just need to add MaxJobsSubmit

[demartinofra]

@jwillis0720 yes it si. Make sure you apply such a change on the head node only and restart slurmctld after changing the config.

[no-response]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.