.. seealso:: You need to have all :ref:`dependencies installed <install-dependencies>` to complete this chapter.
.. sectionauthor:: Axel Huebl, Klaus Steiniger, Sergei Bastrakov
We recommend to use a picongpu.profile file, located directly in your $HOME/ directory,
to set up the environment within which PIConGPU will run by conviently performing
source $HOME/picongpu.profileon the command line after logging in to a system.
PIConGPU is shipped with a number of ready-to-use profiles for different systems which are located in
etc/picongpu/<cluster>-<institute>/ within PIConGPU's main folder.
Have a look into this directory in order to see for which HPC systems profiles are already available.
If you are working on one of these systems, just copy the respective *_picongpu.profile.example
from within this directory into your $HOME and make the necessary changes, such as e-mail address
or PIConGPU source code location defined by $PICSRC.
If you are working on an HPC system for which no profile is available, feel free to create one and
contribute it to PIConGPU by opening a pull request.
A selection of available profiles is presented below, after some general notes on using CPUs. Beware, these may not be up-to-date with the latest available software on the respective system, as we do not have continuous access to all of these.
On CPU systems we strongly recommend using MPI + OpenMP parallelization.
It requires building PIConGPU with the OpenMP 2 backend.
Additionally it is recommended to add an option for target architecture, for example, pic-build -b omp2b:znver3 for AMD Zen3 CPUs.
When building on a compute node or a same-architecture node, one could use -b omp2b:native instead.
The default value for option -b can be set with environment variable $PIC_BACKEND in the profile.
With respect to selecting an optimal MPI + OpenMP configuration please refer to documentation of your system.
As a reasonable default strategy, we recommend running an MPI rank per NUMA node, using 1 or 2 OpenMP threads per core depending on simultaneous multithreading being enabled, and binding threads to cores through affinity settings.
This approach is used, for example, in the defq partition of Hemera as shown below.
The properties of OpenMP parallelization, such as number of threads used, are controlled via OpenMP environment variables.
In particular, the number of OpenMP threads to be used (per MPI rank) can be set via $OMP_NUM_THREADS.
Beware that task launch wrappers used on your system may effectively override this setting.
Particularly, a few systems require running PIConGPU with mpirun --bind-to none in order to properly use all CPU cores.
For setting thread affinity, we provide a helper wrapper cpuNumaStarter.sh that should be applicable to most systems.
This is a very basic picongpu.profile enabling compilation on CPUs by setting the OpenMP backend, declaring commonly required directories,
and providing default parameters for :ref:`TBG <usage-tbg>`.
.. literalinclude:: profiles/bash/bash_picongpu.profile.example :language: bash
System overview: link
Production directory: usually $PROJWORK/$proj/ (link).
Note that $HOME is mounted on compute nodes as read-only.
For this profile to work, you need to download the :ref:`PIConGPU source code <install-dependencies-picongpu>` and install :ref:`PNGwriter and openPMD <install-dependencies>` manually.
.. literalinclude:: profiles/crusher-ornl/batch_hipcc_picongpu.profile.example :language: bash
.. literalinclude:: profiles/crusher-ornl/batch_craycc_picongpu.profile.example :language: bash
System overview: link (internal)
User guide: None
Production directory: /bigdata/hplsim/ with external/, scratch/, development/ and production/
Profile for HZDR's home cluster hemera.
Sets up software environment, i.e. providing libraries to satisfy PIConGPU's dependencies, by loading modules,
setting common paths and options, as well as defining the getDevice() and getNode() aliases.
The latter are shorthands to request resources for an interactive session from the batch system.
Together with the -s bash option of :ref:`TBG <usage-tbg>`, these allow to run PIConGPU interactively on an HPC system.
For this profile to work, you need to download the :ref:`PIConGPU source code <install-dependencies-picongpu>` manually.
.. literalinclude:: profiles/hemera-hzdr/defq_picongpu.profile.example :language: bash
.. literalinclude:: profiles/hemera-hzdr/gpu_picongpu.profile.example :language: bash
.. literalinclude:: profiles/hemera-hzdr/fwkt_v100_picongpu.profile.example :language: bash
.. literalinclude:: profiles/hemera-hzdr/k20_picongpu.profile.example :language: bash
.. literalinclude:: profiles/hemera-hzdr/k80_picongpu.profile.example :language: bash
System overview: link
User guide: link
Production directory: usually $PROJWORK/$proj/ (link).
Note that $HOME is mounted on compute nodes as read-only.
For this profile to work, you need to download the :ref:`PIConGPU source code <install-dependencies-picongpu>` and install :ref:`PNGwriter <install-dependencies>` manually.
.. literalinclude:: profiles/summit-ornl/gpu_picongpu.profile.example :language: bash
System overview: link
User guide: link
Production directory: $SCRATCH (link).
For this profile to work, you need to download the :ref:`PIConGPU source code <install-dependencies-picongpu>` and install :ref:`boost, libpng, PNGwriter and ADIOS2 <install-dependencies>` manually.
Note
The MPI libraries are lacking Fortran bindings (which we do not need anyway).
During the install of ADIOS, make sure to add to configure the --disable-fortran flag.
Note
Please find a Piz Daint quick start from August 2018 here.
.. literalinclude:: profiles/pizdaint-cscs/picongpu.profile.example :language: bash
System overview: link
User guide: link
Production directory: /scratch/$USER/ and /scratch/$proj/
For these profiles to work, you need to download the :ref:`PIConGPU source code <install-dependencies-picongpu>` and install :ref:`PNGwriter <install-dependencies>` manually.
.. literalinclude:: profiles/taurus-tud/k80_picongpu.profile.example :language: bash
For this profile, you additionally need to compile and install everything for the power9-architecture including your own :ref:`boost <install-dependencies>`, :ref:`HDF5 <install-dependencies>`, c-blosc and :ref:`ADIOS <install-dependencies>`.
Note
Please find a Taurus ml quick start here.
Note
You need to compile the libraries and PIConGPU on an ml node since
only nodes in the ml queue are Power9 systems.
.. literalinclude:: profiles/taurus-tud/V100_picongpu.profile.example :language: bash
System overview: link
User guide: link
Production directory: $SCRATCH (link).
For these profiles to work, you need to download the :ref:`PIConGPU source code <install-dependencies-picongpu>` and install :ref:`PNGwriter <install-dependencies>` manually.
.. literalinclude:: profiles/cori-nersc/a100_picongpu.profile.example :language: bash
System overview: link
User guide: link
Production directory: /ptmp/$USER/
For this profile to work, you need to download the :ref:`PIConGPU source code <install-dependencies-picongpu>` and install :ref:`libpng and PNGwriter <install-dependencies>` manually.
.. literalinclude:: profiles/draco-mpcdf/picongpu.profile.example :language: bash
System overview: link
User guide: link
Production directory: $CINECA_SCRATCH/ (link)
For this profile to work, you need to download the :ref:`PIConGPU source code <install-dependencies-picongpu>` manually.
.. literalinclude:: profiles/davide-cineca/gpu_picongpu.profile.example :language: bash
System overview: link
User guide: link
Production directory: $SCRATCH (link)
For these profiles to work, you need to download the :ref:`PIConGPU source code <install-dependencies-picongpu>` and install :ref:`PNGwriter and openPMD <install-dependencies>`, for the gpus partition also :ref:`Boost and HDF5 <install-dependencies>`, manually.
.. literalinclude:: profiles/jureca-jsc/batch_picongpu.profile.example :language: bash
.. literalinclude:: profiles/jureca-jsc/gpus_picongpu.profile.example :language: bash
.. literalinclude:: profiles/jureca-jsc/booster_picongpu.profile.example :language: bash
System overview: link
User guide: link
Production directory: $SCRATCH (link)
For these profiles to work, you need to download the :ref:`PIConGPU source code <install-dependencies-picongpu>` and install :ref:`PNGwriter and openPMD <install-dependencies>`, for the gpus partition also :ref:`Boost and HDF5 <install-dependencies>`, manually.
.. literalinclude:: profiles/juwels-jsc/batch_picongpu.profile.example :language: bash
.. literalinclude:: profiles/juwels-jsc/gpus_picongpu.profile.example :language: bash
System overview: link
User guide: link
Production directory: $WORKDIR (link)
For these profiles to work, you need to download the :ref:`PIConGPU source code <install-dependencies-picongpu>`.
.. literalinclude:: profiles/aris-grnet/gpu_picongpu.profile.example :language: bash
System overview and user guide: link
Production directory: usually $PROJWORK/$proj/ (as on summit link).
For this profile to work, you need to download the :ref:`PIConGPU source code <install-dependencies-picongpu>` and install :ref:`openPMD-api and PNGwriter <install-dependencies>` manually or use pre-installed libraries in the shared project directory.
.. literalinclude:: profiles/ascent-ornl/gpu_picongpu.profile.example :language: bash