-
Notifications
You must be signed in to change notification settings - Fork 1
site docs
This document provides usage instructions for installations of UPC++ at various computing centers.
This document is a continuous work-in-progress, the purpose of which is to provide up-to-date information on public installs maintained by (or in collaboration with) the UPC++ team. However, systems are constantly changing. So, please report of any errors or omissions in the issue tracker.
Typically installs of UPC++ are maintained only for the current default versions of the system-provided environment modules such as compilers and CUDA. If you find one of the installs described in this document to be out-of-date with respect to the current defaults, please report using the issue tracker link above.
This document is not a replacement for the documentation provided by the centers, and assumes general familiarity with the use of the systems.
This document is intended to describe use of existing UPC++ installations and is not a guide to configuring or installing UPC++.
Stable installs are available through environment modules. A wrapper is used
to transparently dispatch commands such as upcxx to an install appropriate to
the currently loaded PrgEnv-{intel,gnu,cray}, craype-{haswell,mic-knl} and
compiler (intel, gcc, or cce) environment modules.
By default, batch jobs on Cori inherit both $PATH and the $MODULEPATH from
the environment at the time the job is submitted/requested using sbatch or
salloc. So, no additional steps are needed to use upcxx-run if a upcxx
environment module was loaded when sbatch or salloc ran.
cori$ module load upcxx
cori$ module switch craype-haswell craype-mic-knl # both work
cori$ upcxx --version
UPC++ version 2020.3.0 / gex-2020.3.0
Copyright (c) 2020, The Regents of the University of California,
through Lawrence Berkeley National Laboratory.
https://upcxx.lbl.gov
icpc (ICC) 19.0.3.199 20190206
Copyright (C) 1985-2019 Intel Corporation. All rights reserved.
cori$ upcxx -O hello-world.cpp -o hello-world.x
cori$ salloc -C knl -q interactive --nodes 2
salloc: Granted job allocation 28703076
salloc: Waiting for resource configuration
salloc: Nodes nid0[2350-2351] are ready for job
nid02350$ upcxx-run -n 4 -N 2 ./hello-world.x
Hello world from process 0 out of 4 processes
Hello world from process 2 out of 4 processes
Hello world from process 1 out of 4 processes
Hello world from process 3 out of 4 processesA UPCXX CMake package is provided in the UPC++ install on Cori, as
described in README.md. Thus with the upcxx environment
module loaded, CMake should "just work" on Cori. However, /usr/bin/cmake on
Cori is fairly old and users may want to use a newer version via module load cmake.
Stable installs are available through environment modules. A wrapper is used
to transparently dispatch commands such as upcxx to an install appropriate
to the currently loaded pgi or gcc environment module.
There are two distinct environment modules available for UPC++: upcxx and
upcxx-cuda. The latter includes initial support for "CUDA Memory Kinds",
which will be improved in a future release.
Please note that UPC++ does not yet work with the IBM XL compilers (the default compiler family on Summit).
Because the UPC++ installation is not yet as well-integrated as on Cori, one
must run module use /gpfs/alpine/world-shared/csc296/summit/modulefiles to
add a non-default directory to the MODULEPATH before the UPC++ environment
modules will be accessible. We recommend inclusion of this command in ones
shell startup files, such as $HOME/.login or $HOME/.bash_profile.
If not adding the command to ones shell startup files, the module use ...
command will be required once per login shell in which you need a upcxx
environment module.
By default, batch jobs on Summit inherit both $PATH and the $MODULEPATH
from the environment at the time the job is submitted using bsub. So, no
additional steps are needed in batch jobs using upcxx-run if a upcxx or
upcxx-cuda environment module was loaded when the job was submitted.
summit$ module load gcc # since default `xl` is not supported
summit$ module load upcxx-cuda
summit$ upcxx -V
UPC++ version 2020.3.0 / gex-2020.3.0
Copyright (c) 2020, The Regents of the University of California,
through Lawrence Berkeley National Laboratory.
https://upcxx.lbl.gov
g++ (GCC) 6.4.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
summit$ upcxx -O hello-world.cpp -o hello-world.x
summit$ bsub -W 5 -nnodes 4 -P [project] -Is bash
Job <714297> is submitted to default queue <batch>.
<<Waiting for dispatch ...>>
<<Starting on batch2>>
bash-4.2$ upcxx-run -n4 ./hello-world.x
Hello world from process 0 out of 4 processes
Hello world from process 2 out of 4 processes
Hello world from process 3 out of 4 processes
Hello world from process 1 out of 4 processesBy default, the installation of PGI compilers on Summit uses the libstdc++
from the (extremely old) /usr/bin/g++. This has been seen to lead to
errors compiling and linking modern C++ code. We strongly recommend only
using PGI compilers with the additional pgi-cxx14 environment module loaded:
summit$ module load pgi pgi-cxx14A UPCXX CMake package is provided in the UPC++ install on Summit, as
described in README.md. CMake is available on Summit via
module load cmake. With the upcxx and cmake environment modules both
loaded, CMake will additionally require either CXX=mpicxx in the environment
or -DCMAKE_CXX_COMPILER=mpicxx on the command line.
The upcxx-run utility provided with UPC++ is a relatively simple wrapper
around the jsrun job launcher on Summit. Since the majority of the resource
allocation/placement capabilities of jsrun have no equivalent in upcxx-run,
and due to the complexity of a Summit compute node, we strongly recommended
to use jsrun directly, for all but the simplest cases.
For instructions on launch of UPC++ applications using a system-provided
"native" spawner, such as IBM's jsrun on Summit, see the section
Advanced Job Launch
in the UPC++ Programmer's Guide.
To become familiar with use of jsrun on Summit, you should read the
Summit User Guide.
Each Summit compute node has two POWER9 CPUs, each with its own I/O bus. Each
I/O bus has a connection to the single InfiniBand Host Channel Adapter (HCA).
The HCA is connected to two "rails" (network ports). This combination of two
I/O buses and two network rails results in four distinct paths between memory
and network. The software stack exposes these paths as four (logical) HCAs
named mlx5_0 through mlx5_3.
| HCA | I/O bus | rail |
|---|---|---|
| mlx5_0 | CPU 0 | A |
| mlx5_1 | CPU 0 | B |
| mlx5_2 | CPU 1 | A |
| mlx5_3 | CPU 1 | B |
Which HCAs are used in a UPC++ application is determined at run time by the
GASNET_IBV_PORTS environment variable. Which ports are used can have a
measurable impact on network performance, but unfortunately there is no "one
size fits all" optimal setting. For instance, the lowest latency is obtained
by having each process use only the two HCAs on the I/O bus of the CPU where
it is executing. Meanwhile, obtaining the maximum bandwidth of a given network
rail from a single CPU requires use of both I/O buses.
By default, the upcxx and upcxx-cuda environment modules will set
GASNET_IBV_PORTS=mlx5_0+mlx5_1. This utilizes both network rails, using
only the I/O bus of CPU0. This penalizes transfers instantiated by CPU1, and
cannot reach peak bandwidth due to using only a single I/O bus. However, for
latency-sensitive applications running on only CPU0, this is a good default.
Additionally, this setting is believed to be free of a corner-case problem
described at the end of this section.
The following are some scenarios and their recommended settings, based on slides which describe the analogous situation for MPI applications on Summit. However, the manner in which MPI and UPC++ use multiple HCAs differs, which accounts for small differences in the recommendations made below.
-
Processes each bound to a single CPU -- latency-sensitive.
To get the best latency from both CPU sockets requires different settings for processes running on each, in order to use both network rails and the I/O bus nearest to the CPU.- CPU0:
GASNET_IBV_PORTS=mlx5_0+mlx5_1 - CPU1:
GASNET_IBV_PORTS=mlx5_2+mlx5_3
- CPU0:
-
Processes each bound to a single CPU -- bandwidth-sensitive.
How to get the full bandwidth from both CPU sockets depends on the communication behaviors of the application. If both CPUs are communicating at the same time, then the latency-optimized settings immediately above are typically sufficient to achieve peak aggregate bandwidth. However, if a single communicating CPU (at a given time) is to achieve the peak bandwidth a different pair of process-specific settings is required (which comes at the cost of slightly increased mean latency).- CPU0:
GASNET_IBV_PORTS=mlx5_0+mlx5_3 - CPU1:
GASNET_IBV_PORTS=mlx5_1+mlx5_2
- CPU0:
-
Processes each bound to a single CPU -- mixed or unknown behavior.
In relative terms, the bandwidth penalty is greater when using only a single I/O bus than is the latency penalty for use of the farther I/O bus. For this reason the bandwidth-optimizing settings (immediately above) are the nearest thing to a "generic" application recommendation. -
Processes unbound or individually spanning both CPUs.
In this case the best average performance comes from using only half of the available paths (and using all four incurs a measurable penalty). This setting also provides a reasonable balance when one is unable to establish per-CPU settings (see below).GASNET_IBV_PORTS=mlx5_0+mlx5_3
The recommendations above include cases in which one should provide distinct environment variables to different processes. In the future we hope this can be automated. However, until that happens one can use a simple bash shell script such as the following:
#!/bin/bash
socket=$(hwloc-calc -I Node $(hwloc-bind --get))
case $socket in
1) export GASNET_IBV_PORTS=mlx5_2+mlx5_3 ;;
*) export GASNET_IBV_PORTS=mlx5_0+mlx5_1 ;;
esac
exec "$@"This example script implements the latency-optimizing settings (see below for
an analogous bandwidth-optimizing version) with the additional behavior of
assigning the CPU 0 setting to processes which span CPUs. To demonstrate use
of this example script, let us assume it has been saved as wrapper.sh in the
current directory and made executable (as with chmod +x wrapper.sh). You
can then use it to prefix the executable (./my_app in the following) when
running with jsrun (see also section "Job launch on Summit", above):
$ jsrun [jsrun options] ./wrapper.sh ./my_app [application args]As mentioned briefly above, the default was chosen in part to avoid a
corner-case correctness problem. The latency-optimizing settings are believed
to be highly resistant (but not entirely immune) to this problem. However,
the other settings described above (ones involving mlx5_0+mlx5_3 or
mlx5_1+mlx5_2) use both I/O buses in a single process, which can lead to
data corruption in some cases.
The issue is that, by default, the use of multiple I/O buses may permit an
rput which has signaled operation completion to be overtaken by a subsequent
rput, rget or rpc. When an rput is overtaken by another rput to the
same location, the earlier value may be stored rather than the latter. When
an rget overtakes an rput targeting the same location, it may fail to
observe the value stored by the rput. When an rpc overtakes an rput,
CPU accesses to the location targeted by the rput is subject to both of the
preceding problems.
If you suspect your application is seeing such data corruption (or just want to
be certain that it cannot), we recommend setting GASNET_USE_FENCED_PUTS=1
in your environment at run time. This introduces a penalty in both latency and
bandwidth, but the bandwidth penalty is tiny when compared to the increase due
to using both I/O buses. With this in mind, the following is the example
wrapper script for bandwidth-optimized runs.
#!/bin/bash
socket=$(hwloc-calc -I Node $(hwloc-bind --get))
case $socket in
1) export GASNET_IBV_PORTS=mlx5_1+mlx5_2 ;;
*) export GASNET_IBV_PORTS=mlx5_0+mlx5_3 ;;
esac
export GASNET_USE_FENCED_PUTS=1
exec "$@"Stable installs are available through environment modules. A wrapper is used
to transparently dispatch commands such as upcxx to an install appropriate
to the currently loaded PrgEnv-{intel,gnu,cray} and compiler (intel,
gcc, or cce) environment modules.
Because the UPC++ installation is not yet as well-integrated as on Cori,
one must module use ... to add a non-default directory to the MODULEPATH
before the UPC++ environment modules will be accessible. We recommend
inclusion of the required command in ones $HOME/.modulerc to make it
persistent. This file must begin with a #%Module line to be accepted by
the module command.
A complete .modulerc suitable for Theta:
#%Module
module use /projects/CSC250STPM17/modulefiles
If not using .modulerc, the module use ... command will be required once
per login shell in which you need a upcxx environment module.
COBALT jobs (both batch and interactive) do not inherit the necessary
settings from the submit-time environment, meaning both the module use ...
and module load upcxx may be required in batch jobs which use upcxx-run.
This is shown in the example below.
theta$ module load upcxx
theta$ upcxx --version
UPC++ version 2020.3.0 / gex-2020.3.0
Copyright (c) 2020, The Regents of the University of California,
through Lawrence Berkeley National Laboratory.
https://upcxx.lbl.gov
icpc (ICC) 19.0.5.281 20190815
Copyright (C) 1985-2019 Intel Corporation. All rights reserved.
theta$ upcxx -O hello-world.cpp -o hello-world.x
theta$ qsub -q debug-cache-quad -t 10 -n 2 -A CSC250STPM17 -I
Connecting to thetamom3 for interactive qsub...
Job routed to queue "debug-cache-quad".
Memory mode set to cache quad for queue debug-cache-quad
Wait for job 418194 to start...
Opening interactive session to 3833,3836
thetamom3$ # Note that modules have reset to defaults
thetamom3$ module load upcxx
thetamom3$ upcxx-run -n 4 -N 2 ./a.out
Hello from 0 of 4
Hello from 1 of 4
Hello from 3 of 4
Hello from 2 of 4A UPCXX CMake package is provided in the UPC++ install on Theta, as
described in README.md. While /usr/bin/cmake is too old,
sufficiently new CMake versions are available on Theta via module load cmake.
With the upcxx and cmake environment modules both loaded, CMake should
"just work" on Theta.