Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Installation on ARCHER
Philip Heron and ARCHER support staff.
Written 10th January 2018
Setting up ASPECT on ARCHER requires a number of hours. It is best setup using submit scripts to run on the appropriate nodes. As a result, an account to a project is required for this setup.
The ARCHER support staff has made a number of submit scripts available. The zip file of the folder can be found here at https://github.com/heronphi/Archer_dealii_aspect_install. Below will go through the different files within it and how they are used.
The instructions below utilize deal.ii 8.5.1 - in order for deal.ii to be compatible with ARCHER a number of changes have been made. If a newer version of deal.ii is required, please get in contact and another setup can be obtained.
As mentioned, you will need a project account to be able to run the setup scripts. In the submit scripts, the project account will be given as a A99-xacc, please replace this with your own.
Setup - deal.ii
The outline below compiles deal.ii in the work directory.
In your work directory, make a deal_ii folder that the build we take place.
Copy the contents of the Archer_dealii_aspect_install folder into deal_ii –
build.pbs master.sh build_p4est.sh modules.sh change.sh p4est-1.1.tar.gz configure.sh p4est-setup.sh dealii-8.5.1.changes step-32_aprun_CMakeLists.txt dealii-8.5.1.changes.tar.gz tar.sh dealii-8.5.1.tar.gz test.pbs download.sh unpack.sh env.sh
A master script (master.sh) has been made available to work through all of the setup required for deal.ii. The master script is given here:
. unpack.sh . change.sh . env.sh &> env.log . modules.sh &> modules.log module list >> modules.log 2>&1 . build_p4est.sh &> build_p4est.log mkdir -p $build_dir cd $build_dir . ../configure.sh &> ../configure.log && qsub ../build.pbs cd ..
Below I will outline what each script does, and what need to be done to ensure the build occurs in your work directory. After going through these edits, the master script should then be executed.
. unpack.shthis run the unpack script, which unpacks the deal.ii 8.5.1 source code.
. change.shthis runs the change script, which unpacks the changes to deal.ii 8.5.1 source code.
. env.shruns the environment script. To set up the build of deal.ii, we need to edit the env.sh build script that configures the variables of deal.ii. We need to edit install_dir line to point to the correct build directory:
. module.shsets up the modules that are needed. The module list will be (including the initial modules already loaded):
Currently Loaded Modulefiles: 1) modules/126.96.36.199 2) eswrap/1.3.3-1.020200.1280.0 3) switch/1.0-1.0502.60522.1.61.ari 4) craype-network-aries 5) craype-ivybridge 6) craype/2.5.10 7) pbs/12.2.401.141761 8) cray-mpich/7.5.5 9) packages-archer 10) bolt/0.6 11) nano/2.2.6 12) leave_time/1.3.0 13) quickstart/1.0 14) ack/2.14 15) xalt/0.6.0 16) epcc-tools/6.0 17) gcc/5.3.0 18) cray-libsci/16.11.1 19) udreg/2.3.2-1.0502.10518.2.17.ari 20) ugni/6.0-1.0502.10863.8.29.ari 21) pmi/5.0.12 22) dmapp/7.0.1-1.0502.11080.8.76.ari 23) gni-headers/4.0-1.0502.10859.7.8.ari 24) xpmem/0.1-2.0502.64982.5.3.ari 25) dvs/2.5_0.9.0-1.0502.2188.1.116.ari 26) alps/5.2.4-2.0502.9774.31.11.ari 27) rca/1.0.0-2.0502.60530.1.62.ari 28) atp/2.1.0 29) PrgEnv-gnu/5.2.82 30) cmake/3.5.2 31) cray-hdf5-parallel/188.8.131.52 32) cray-netcdf-hdf5parallel/184.108.40.206 33) cray-tpsl/17.04.1 34) cray-trilinos/220.127.116.11
. build_p4est.shthis builds the p4est library that has been downloaded into the original tar file.
configure.shthe configure script for deal.ii and the ARCHER Cray Linux Environment.
build.pbs starts the make file in a qsub script to allow for a long build on a node. Here we need to change the account number:
#PBS -A A99-xacc
- Before the master script is run, we need to set the Cray linking properties to dynamic. In the terminal window, type:
- After the making the edits in points (3), (7) and (8), then master script can be executed:
Please note – it is important to execute this script as it is written. Please use “.[space]master.sh” and not ./master.sh, or anything else.
The unpacking, patches, and configures will take about 40 minutes. However, then the build job will then take 2 hours.
- We can run some tests using the test.pbs script - but the account name must be changed:
#PBS -A A99-xacc
From the build script the test script can be executed:
cd deal_ii/build qsub ../test.pbs
However, the affinity test fails because I think it is testing the wrong thing - it compares the numbers of CPUs DEAL II is running on with the total number of CPUs * SMP threads.
Setup - ASPECT
Once deal.ii is built, we can download ASPECT and make and install.
- Before installing, run the modules script in the deal.ii folder:
as above, this is .[space]modules.sh Check the module loaded are the same as that shown in point 4 in the deal.ii install.
module list Currently Loaded Modulefiles: 1) modules/18.104.22.168 2) eswrap/1.3.3-1.020200.1280.0 3) switch/1.0-1.0502.60522.1.61.ari 4) craype-network-aries 5) craype-ivybridge 6) craype/2.5.10 7) pbs/12.2.401.141761 8) cray-mpich/7.5.5 9) packages-archer 10) bolt/0.6 11) nano/2.2.6 12) leave_time/1.3.0 13) quickstart/1.0 14) ack/2.14 15) xalt/0.6.0 16) epcc-tools/6.0 17) gcc/5.3.0 18) cray-libsci/16.11.1 19) udreg/2.3.2-1.0502.10518.2.17.ari 20) ugni/6.0-1.0502.10863.8.29.ari 21) pmi/5.0.12 22) dmapp/7.0.1-1.0502.11080.8.76.ari 23) gni-headers/4.0-1.0502.10859.7.8.ari 24) xpmem/0.1-2.0502.64982.5.3.ari 25) dvs/2.5_0.9.0-1.0502.2188.1.116.ari 26) alps/5.2.4-2.0502.9774.31.11.ari 27) rca/1.0.0-2.0502.60530.1.62.ari 28) atp/2.1.0 29) PrgEnv-gnu/5.2.82 30) cmake/3.5.2 31) cray-hdf5-parallel/22.214.171.124 32) cray-netcdf-hdf5parallel/126.96.36.199 33) cray-tpsl/17.04.1 34) cray-trilinos/188.8.131.52
- Before ASPECT is built, the environment needs to be set so that dynamic linking can occur:
- Wherever you would like to download ASPECT:
git clone --recursive https://github.com/geodynamics/aspect.git
- Enter into the ASPECT folder, and start the CMAKE by linking to the dealii build folder.
cd aspect cmake -DDEAL_II_DIR=/work/A99/A99/username/deal_ii/build . -- ==================================================== -- ============ Configuring ASPECT ==================== -- ==================================================== -- Found deal.II version 8.5.1 at '/work/A99/A99/username/deal_ii/build/lib/cmake/deal.II' -- Using ASPECT_USE_PETSC = 'OFF' -- The C compiler identification is GNU 5.3.0 -- The CXX compiler identification is GNU 5.3.0 -- Cray Programming Environment 2.5.10 C -- Check for working C compiler: /opt/cray/craype/2.5.10/bin/cc -- Check for working C compiler: /opt/cray/craype/2.5.10/bin/cc -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Detecting C compile features -- Detecting C compile features - done -- Cray Programming Environment 2.5.10 CXX -- Check for working CXX compiler: /opt/cray/craype/2.5.10/bin/CC -- Check for working CXX compiler: /opt/cray/craype/2.5.10/bin/CC -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Detecting CXX compile features -- Detecting CXX compile features - done -- Setting up test project, see tests/setup_tests.log for details. -- Performing Test HAVE_DLOPEN -- Performing Test HAVE_DLOPEN - Success -- Enabling dynamic loading of plugins from the input file -- Looking for include file link.h -- Looking for include file link.h - found -- Enabling checking of compatible deal.II library when loading plugins -- Autopilot invoked ### # # Project aspect set up with deal.II-8.5.1 found at # /work/A99/A99/username/deal_ii/build # # CMAKE_BUILD_TYPE: Debug # # You can now run # $ make - to compile and link the program # $ make run - to (compile, link and) run the program # # $ make debug - to switch the build type to 'Debug' # $ make release - to switch the build type to 'Release' # # $ make edit_cache - to change (cached) configuration variables # and rerun the configure and generate phases of CMake # # $ make strip_comments - to strip the source files in this # directory off the documentation comments # $ make clean - to remove the generated executable as well as # all intermediate compilation files # $ make runclean - to remove all output generated by the program # $ make distclean - to clean the directory from _all_ generated # files (includes clean, runclean and the removal # of the generated build system) # $ make info - to view this message again # # Have a nice day! # ### -- Performing Test HAVE_FP_EXCEPTIONS -- Performing Test HAVE_FP_EXCEPTIONS - Success -- Runtime floating point checks enabled. -- Disabling cotire. -- Parameter GUI not found: install and provide a hint using -D PARAMETER_GUI_DIR or set -D PARAMETER_GUI_EXECUTABLE directly. -- Writing config into detailed.log... -- Configuring done -- Generating done -- Build files have been written to: /work/A99/A99/username/aspect
- After this cmake, you can use the make command to generate the exectuable:
(where is the number of processors you have available to run the ASPECT make). I would recommend running -j8.
- This will build the release executable, if the release version is needed, re-make using
The user guide gives an example as such:
For example, to generate a job script to run an executable called 'my_prog.x' for 3 hours using 3072 parallel tasks and 12 tasks per compute node, you would use:
bolt -n 3072 -N 12 -t 3:0:0 -o my_job.bolt -j my_job -A z01-budget my_prog.x arg1 arg2
This generates the job script 'my_job.bolt' with the correct options to run 'my_prog.x' with command line arguments 'arg1' and 'arg2'. The project code against which the job will be charged is specified with the ' -A ' option.
In this instance, we want to generate a 72 CPU job lasting 3 hours for the van-keken-smooth cookbook. If we went into the cookbook folder, which is where we could like to run the van-keken-smooth file, we can generate a script:
cd cookbook bolt -n 72 -N 12 -t 3:0:0 -o aspect_example.bolt -j Cookbook_1 -A A99-xacc ../aspect van-keken-smooth.prm
This generates the file aspect_example.bolt
#!/bin/bash --login # # Parallel script produced by bolt # Resource: ARCHER (Cray XC30 (24-core per node)) # Batch system: PBSPro_select # # bolt is written by EPCC (http://www.epcc.ed.ac.uk) # #PBS -l select=6 #PBS -N Cookbook_1 #PBS -A A99-xacc #PBS -l walltime=3:0:0 # Switch to current working directory cd $PBS_O_WORKDIR # Run the parallel program aprun -n 72 -N 12 -S 6 -d 1 ../aspect van-keken-smooth.prm
We can execute the file using qsub:
This setup was graciously made by Mark at the Archer support staff. Any queries on the build can be made to firstname.lastname@example.org