[Examples] Update nccl-tests

[un-def]

• Generate hostfile with slots option
• Add MCA parameters from https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-start-nccl.html#nccl-start-base-test

[r4victor]

What was the effect of missing slots? I got errors when using a hostfile without slots with TCPRX on 8xH100.

[un-def]

Error message:

here are not enough slots available in the system to satisfy the N
slots that were requested by the application:

  ./all_reduce_perf

Either request fewer slots for your application, or make more slots
available for use.

A "slot" is the Open MPI term for an allocatable unit where we can
launch a process.  The number of slots available are defined by the
environment in which Open MPI processes are run:

  1. Hostfile, via "slots=N" clauses (N defaults to number of
     processor cores if not provided)
  2. The --host command line parameter, via a ":N" suffix on the
     hostname (N defaults to 1 if not provided)
  3. Resource manager (e.g., SLURM, PBS/Torque, LSF, etc.)
  4. If none of a hostfile, the --host command line parameter, or an
     RM is present, Open MPI defaults to the number of processor cores

If none of a hostfile, the --host command line parameter, or an RM is present, Open MPI defaults to the number of processor core

For example, n1-highmem-2 has 4 T4 but only 2 CPU, meaning that slots defaults to 2 while we request 4 processes per node.