The Delta programming environment supports the GNU, AMD (AOCC), Intel and NVIDIA HPC compilers. Support for the HPE/Cray Programming environment is forthcoming.
Modules provide access to the compiler + MPI environment.
The default environment includes the GCC 11.4.0 compiler + OpenMPI; nvcc is in the CUDA module and is loaded by default.
AMD recommended compiler flags for GNU, AOCC, and Intel compilers for Milan processors can be found in the AMD Compiler Options Quick Reference Guide for Epyc 7xx3 processors.
To build (compile and link) a serial program in Fortran, C, and C++:
Serial Program CommandsGCC | AOCC | NVHPC |
---|---|---|
gfortran myprog.f | flang myprog.f | nvfortran myprog.f |
gcc myprog.c | clang myprog.c | nvc myprog.c |
g++ myprog.cc | clang myprog.cc | nvc++ myprog.cc |
To build (compile and link) a MPI program in Fortran, C, and C++:
MPI Program CommandsMPI Implementation | Module Files for MPI/Compiler | Build Commands |
---|---|---|
OpenMPI |
|
|
Cray MPICH
|
|
|
To build an OpenMP program, use the -fopenmp / -mp option.
OpenMP Program CommandsGCC | AOCC | NVHPC |
---|---|---|
gfortran -fopenmp myprog.f | flang -fopenmp myprog.f | nvfortran -mp myprog.f |
gcc -fopenmp myprog.c | clang -fopenmp myprog.c | nvc -mp myprog.c |
g++ -fopenmp myprog.cc | clang -fopenmp myprog.cc | nvc++ -mp myprog.cc |
To build an MPI/OpenMP hybrid program, use the -fopenmp / -mp option with the MPI compiling commands.
Hybrid MPI/OpenMP Program CommandsGCC | PGI/NVHPC |
---|---|
mpif77 -fopenmp myprog.f |
|
mpif90 -fopenmp myprog.f90 |
|
mpicc -fopenmp myprog.c |
|
mpic++ -fopenmp myprog.cc |
|
Document - XC Series User Application Placement Guide CLE6..0UP01 S-2496 | HPE Support
This code can be compiled using the methods show above. The code appears in some of the batch script examples below to demonstrate core placement options.
#define _GNU_SOURCE
#include <stdio.h>
#include <unistd.h>
#include <string.h>
#include <sched.h>
#include <mpi.h>
#include <omp.h>
/* Borrowed from util-linux-2.13-pre7/schedutils/taskset.c */
static char *cpuset_to_cstr(cpu_set_t *mask, char *str)
{
char *ptr = str;
int i, j, entry_made = 0;
for (i = 0; i < CPU_SETSIZE; i++) {
if (CPU_ISSET(i, mask)) {
int run = 0;
entry_made = 1;
for (j = i + 1; j < CPU_SETSIZE; j++) {
if (CPU_ISSET(j, mask)) run++;
else break;
}
if (!run)
sprintf(ptr, "%d,", i);
else if (run == 1) {
sprintf(ptr, "%d,%d,", i, i + 1);
i++;
} else {
sprintf(ptr, "%d-%d,", i, i + run);
i += run;
}
while (*ptr != 0) ptr++;
}
}
ptr -= entry_made;
*ptr = 0;
return(str);
}
int main(int argc, char *argv[])
{
int rank, thread;
cpu_set_t coremask;
char clbuf[7 * CPU_SETSIZE], hnbuf[64];
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
memset(clbuf, 0, sizeof(clbuf));
memset(hnbuf, 0, sizeof(hnbuf));
(void)gethostname(hnbuf, sizeof(hnbuf));
#pragma omp parallel private(thread, coremask, clbuf)
{
thread = omp_get_thread_num();
(void)sched_getaffinity(0, sizeof(coremask), &coremask);
cpuset_to_cstr(&coremask, clbuf);
#pragma omp barrier
printf("Hello from rank %d, thread %d, on %s. (core affinity = %s)\n",
rank, thread, hnbuf, clbuf);
}
MPI_Finalize();
return(0);
}
A version of xthi is also available from ORNL:
% git clone https://github.com/olcf/XC30-Training/blob/master/affinity/Xthi.c
To build an OpenACC program, use the -acc option and the -mp option for multi-threaded:
OpenACC Program CommandsNon-Multi-threaded | Multi-threaded |
---|---|
nvfortran -acc myprog.f |
|
nvc -acc myprog.c |
|
nvc++ -acc myprog.cc |
|
CUDA compilers (nvcc) are included in the CUDA module which is loaded by default under modtree/gpu. For the CUDA Fortran compiler and other NVIDIA development tools, load the nvhpc module.
[arnoldg@dt-login03 namd]$ nv
nvaccelerror nvidia-bug-report.sh nvlink
nvaccelinfo nvidia-cuda-mps-control nv-nsight-cu
nvc nvidia-cuda-mps-server nv-nsight-cu-cli
nvc++ nvidia-debugdump nvprepro
nvcc nvidia-modprobe nvprof
nvcpuid nvidia-persistenced nvprune
nvcudainit nvidia-powerd nvsize
nvdecode nvidia-settings nvunzip
nvdisasm nvidia-sleep.sh nvvp
nvextract nvidia-smi nvzip
nvfortran nvidia-xconfig
See the NVIDIA HPC SDK page for more information.
Note: The Multi-Process Service (MPS) is not currently enabled on Delta GPU nodes. Submit a support request <help>
for assistance if you have questions about MPS status.
Note
If using hipcc on the login nodes, add --offload-arch=gfx908 to the flags to match the gpu on the MI100 node.
To access the development environment for the gpuMI100x8 partition, start a job on the node with srun
or sbatch
.
Next, set your PATH to prefix /opt/rocm/bin where the HIP and ROCm tools are installed.
A sample batch script to obtain an xterm (interactive xterm batch script for Slurm) is shown below:
#!/bin/bash -x
MYACCOUNT=$1
GPUS=--gpus-per-node=1
PARTITION=gpuMI100x8-interactive
srun --tasks-per-node=1 --nodes=1 --cpus-per-task=4 \
--mem=16g \
--partition=$PARTITION \
--time=00:30:00 \
--account=account_name \ # <- match to a "Project" returned by the "accounts" command
$GPUS --x11 \
xterm
AMD HIP development environment on gpud01 (setting the path on the compute node):
[arnoldg@gpud01 bin]$ export PATH=/opt/rocm/bin:$PATH
[arnoldg@gpud01 bin]$ hipcc
No Arguments passed, exiting ...
[arnoldg@gpud01 bin]$
See the AMD HIP documentation and AMD ROCm documentation for more information.
Note
The Delta Open OnDemand (OOD) portal provides an easy method to use VS Code in a web browser.
Go to the OOD Code Server (VS Code) interactive app <ood-code-server>
page for instructions on how to start an OOD VS Code session.
Microsoft VS Code documentation
The code-server for VS Code can be run on Delta in manual mode (without Open OnDemand) by following these steps:
Start the server.
/sw/external/vscode/code-server/bin/code-server:[arnoldg@dt-login03 bin]$ ./code-server --bind-addr dt-login03:8899 [2023-04-14T15:57:03.059Z] info code-server 4.11.0 85e083580dec27ef19827ff42d3c9257d56ea7e3 [2023-04-14T15:57:03.060Z] info Using user-data-dir ~/.local/share/code-server [2023-04-14T15:57:03.132Z] info Using config file ~/.config/code-server/config.yaml [2023-04-14T15:57:03.133Z] info HTTP server listening on http://141.142.140.196:8899/ [2023-04-14T15:57:03.133Z] info - Authentication is enabled [2023-04-14T15:57:03.133Z] info - Using password from ~/.config/code-server/config.yaml [2023-04-14T15:57:03.133Z] info - Not serving HTTPS [10:57:12]
SSH to the login node where the server is waiting. Read the config.yaml noted above and copy the password to your clipboard.
SSH tunnel to login node running code-server:(base) galen@macbookair-m1-042020 ~ % ssh -l arnoldg -L 127.0.0.1:8899:dt-login03.delta.ncsa.illinois.edu:8899 dt-login03.delta.ncsa.illinois.edu ... Success. Logging you in... dt-login03.delta.internal.ncsa.edu (141.142.140.196) OS: RedHat 8.6 HW: HPE CPU: 128x RAM: 252 GB ΔΔΔΔΔ ΔΔΔΔΔΔ ΔΔ ΔΔΔΔΔΔ ΔΔ ΔΔ ΔΔ ΔΔ ΔΔ ΔΔ ΔΔΔΔ ΔΔ ΔΔ ΔΔΔΔ ΔΔ ΔΔ ΔΔ ΔΔ ΔΔ ΔΔ ΔΔ ΔΔ ΔΔ ΔΔΔΔΔΔ ΔΔΔΔΔ ΔΔΔΔΔΔ ΔΔΔΔΔΔ ΔΔ ΔΔ ΔΔ [arnoldg@dt-login03 ~]$ more ~/.config/code-server/config.yaml bind-addr: 127.0.0.1:8080 auth: password password: 9e8081e80d9999c3c525fe26 cert: false
Open a local browser on your desktop system with URL = http://127.0.0.1:8899. Log in with the password copied from above and begin using VS Code in your browser.
Follow the Visual Studio Code remote development using SSH guide.
As stated in the guide, install "Remote - SSH" into Visual Studio:
Continue to follow the guide to set up a remote connection to Delta. It helps if you have a local $HOME/.ssh/config with your commonly used hosts already present on the laptop and SSH client where you will be using Visual Studio. Here is an example entry for Delta, change your username to your login name on Delta. Visual Studio will show hosts in your config in a pick list.
SSH config:Host delta HostName login.delta.ncsa.illinois.edu User arnoldg ForwardX11 True
Once connected, you can work with the remote system as if it were local. When Visual Studio needs to install extension items on the remote system, it will go into your $HOME/.vscode-server on Delta. Visual Studio takes care of all the details for you:
remote server VS extensions:[arnoldg@dt-login03 ~]$ du -sh .vscode-server/ 523M .vscode-server/ [arnoldg@dt-login03 ~]$
Proceed to F1 → Remote SSH and connect to Delta. Then, following the guide, use Visual Studio as normal.
Windows users: The login box of vscode will display your login as 2fa<delta_username>, and you may not see a 2nd login box for 2fa Duo until you press the "details" link at lower right after you enter your password. Use the Duo passcode after pressing "details" link when the next password prompt appears at the top. Also see the Visual Studio Code remote development troubleshooting guide and search for "two-factor".
If VS Code fails to connect after you have been using it for a while, check your quota (a full $HOME can prevent it from updating $HOME/.vscode on Delta). We have also seen cases where it was necessary to remove $HOME/.vscode on Delta similar the VS Code documentation - clean uninstall.
Example of working with a C file remote on Delta:
See the Visual Studio Code working with Juypter Notebooks guide and jupyter
(open two new browser tabs).
- Install the Jupyter extension for Visual Studio, if you have not already done so.
- Complete the first step from the Delta user guide (second link above) where you srun a jupyter-notebook on a compute node.
Make note of and copy the first URL after the job is running, that is the URI you will provide to Visual Studio's "Connect to a Remote Jupyter Server" after clicking the Kernels button.
You may also need to select the remote jupyter kernel under the kernels in VScode.