-
Notifications
You must be signed in to change notification settings - Fork 233
Installation on ARCHER
Jeroen van Hunen, Philip Heron and ARCHER support staff.
Email: jeroen.van-hunen@durham.ac.uk, philip.j.heron@durham.ac.uk
13th November 2019
As mentioned, you will need a project account to be able to run the setup scripts. In the submit scripts, the project account will be given as a A99-xacc, please replace this with your own.
The instructions below utilise deal.ii 9.0.0, which is installed on Archer at /work/y07/y07/cse/dealii/9.0.0/deal_II/9.0.0-gnu-32-trilinos
- First, go to your working directory:
pwd
/work/A99/A99/username/
where A99 and username will need to be replaced with your specific group and username.
- Set up the modules that are needed:
. /work/y07/y07/cse/dealii/9.0.0/source/modules.sh
(Note the .[space] before the /work... part). This will also set the LD_LIBRARY_PATH.
- Now set other paths and system variables:
export CPATH=$CPATH:/work/y07/y07/cse/dealii/9.0.0/deal_II/9.0.0-gnu-32-trilinos/include:/work/y07/y07/cse/dealii/9.0.0/deal_II/9.0.0-gnu-32-trilinos/p4est/FAST/include
- Before ASPECT is built, the environment needs to be set so that dynamic linking can occur:
export CRAYPE_LINK_TYPE=dynamic
- Wherever you would like to download ASPECT:
git clone --recursive https://github.com/geodynamics/aspect.git
- Enter into the ASPECT folder, and start the CMAKE by linking to the deal-ii build folder.
cd aspect
cmake -DDEAL_II_DIR=/work/y07/y07/cse/dealii/9.0.0/deal_II/9.0.0-gnu-32-trilinos/lib/cmake/deal.II .
(Note the dot at the end of the line)
- After this cmake, you can use the make command to generate the executable:
make -j<N>
(where is the number of processors you have available to run the ASPECT make). We would recommend running -j12.
- This will build the release executable, if the release version is needed, re-make using
mv aspect aspect.dbg
make release
make -j<N>
Running jobs on ARCHER is explained well in the ARCHER user guide. An example script is given below - it was generated using the package Bolt and is described in step 5.3 of the user guide.
The user guide gives an example as such:
For example, to generate a job script to run an executable called 'my_prog.x' for 3 hours using 3072 parallel tasks and 12 tasks per compute node, you would use:
bolt -n 3072 -N 12 -t 3:0:0 -o my_job.bolt -j my_job -A z01-budget my_prog.x arg1 arg2
This generates the job script 'my_job.bolt' with the correct options to run 'my_prog.x' with command line arguments 'arg1' and 'arg2'. The project code against which the job will be charged is specified with the ' -A ' option.
In this instance, we want to generate a 72 CPU job lasting 3 hours for the van-keken-smooth cookbook. If we went into the cookbook folder, which is where we could like to run the van-keken-smooth file, we can generate a script:
cd cookbook
bolt -n 72 -N 12 -t 3:0:0 -o aspect_example.bolt -j Cookbook_1 -A A99-xacc ../aspect van-keken-smooth.prm
This generates the file aspect_example.bolt
#!/bin/bash --login
#
# Parallel script produced by bolt
# Resource: ARCHER (Cray XC30 (24-core per node))
# Batch system: PBSPro_select
#
# bolt is written by EPCC (http://www.epcc.ed.ac.uk)
#
#PBS -l select=6
#PBS -N Cookbook_1
#PBS -A A99-xacc
#PBS -l walltime=3:0:0
# Switch to current working directory
cd $PBS_O_WORKDIR
# Run the parallel program
aprun -n 72 -N 12 -S 6 -d 1 ../aspect van-keken-smooth.prm
We can execute the file using qsub:
qsub aspect_example.bolt
This setup was graciously supported by Adrian Jackson from the Archer support staff. Any queries on the build can be made to jeroen.van-hunen@durham.ac.uk or philip.j.heron@durham.ac.uk