Skip to content

CMake infrastructure update for Psi4

Jump to bottom Edit New page
Lori A. Burns edited this page Jun 7, 2015 · 15 revisions

This page summarizes the latest infrastructure updates for CMake support in Psi4.

How to help CMake find libraries

In general, if you have a library in an esoteric location on your machine, you have to help CMake by passing the appropriate hint variables. This can be done either by using the appropriate setup script options, by exporting environment variables or by appending -DPATH_TO_THIS=/path/to/this -DPATH_TO_THAT=/path/to/that options to the setup script invocation.

LAB-added

Based on https://gcc.gnu.org/projects/cxx0x.html and talking with Ryan, gcc-4.8 is safe, 4.7 is probably safe, and 4.6 should be, too, though we've heard a couple reports of problems in which 4.6 was blamed, fairly or not.

C++11

If the project is set up using --cxx11=on (or -DENABLE_CXX11_SUPPORT=ON), CMake will probe the compiler for C++11 support. The C++11 support macros will check if the compiler has a keyword to enable the new standard in the form -std=c++11 or -std=c++0x. If that is the case the macro -DHAS_CXX11 is defined. This can be used to enable/disable code that depends on the new standard. The detected flag is also added to the C++ compiler flags. If C++11 is not supported -std=gnu++98 is used instead.

The CMake macro also checks for the new language features introduced by the new standard. If supported, a macro is defined, so that a further level of fine-tuning can be used by developers. These are the features CMake checks:

  • __func__, macro -DHAS_CXX11_FUNC
  • auto, macro -DHAS_CXX11_AUTO
  • auto_ret_type, macro -DHAS_CXX11_AUTO_RET_TYPE
  • class_override_final, macro -DHAS_CXX11_CLASS_OVERRIDE
  • constexpr, macro -DHAS_CXX11_CONSTEXPR
  • cstdint, macro -DHAS_CXX11_CSTDINT_H
  • decltype, macro -DHAS_CXX11_DECLTYPE
  • initializer_list, macro -DHAS_CXX11_INITIALIZER_LIST
  • lambda, macro -DHAS_CXX11_LAMBDA
  • long_long, macro -DHAS_CXX11_LAMBDA
  • nullptr, macro -DHAS_CXX11_NULLPTR
  • regex, macro -DHAS_CXX11_LIB_REGEX
  • rvalue-references, macro -DHAS_CXX11_RVALUE_REFERENCES
  • sizeof_member, macro -DHAS_CXX11_SIZEOF_MEMBER
  • static_assert, macro -DHAS_CXX11_STATIC_ASSERT
  • variadic_templates, macro -DHAS_CXX11_VARIADIC_TEMPLATES

The available features are listed at configure time as output of the setup or cmake commands. The macro definitions enabled are also listed in the same output.

C++11 support can be difficult to detect for Intel compilers, because they rely on the headers from GCC compilers. I still haven't figured out a general and bullet-proof strategy to disable C++11 support in these difficult cases.

Fortran codes in Psi

How to include a Fortran code in Psi

The Fortran codes currently included in Psi can be divided into three categories:

  1. optional replacement of C/C++ standard code, e.g. ERD replacing libint;
  2. optional components, e.g. PCMSolver;
  3. components to be compiled whenever a Fortran compiler is explicitly specified by the user at configuration. This is the case for the Douglas-Kroll-Hess integrals subroutines.

Type 1 and type 2 Fortran projects can be enabled using the appropriate setup script option or the corresponding CMake variable ENABLE_project (e.g. ENABLE_ERD for ERD) Fortran is enabled at the beginning of the topmost CMakeLists.txt

# Decide if Fortran is needed
if(ENABLE_LIBERD)
   if(CMAKE_Fortran_COMPILER)
      enable_language(Fortran)
      add_definitions(-DHAVE_FORTRAN)
      # This is to use the CMake generated macros and not those based on FC_SYMBOL
      add_definitions(-DUSE_FCMANGLE_H)
      set(FORTRAN_ENABLED TRUE)
   else()
      message(FATAL_ERROR "The Fortran compiler has to be explicitly specified!")
   endif()
endif()

Some preprocessor definitions are added:

  1. -DHAVE_FORTRAN this will trigger enabling of the type 3 Fortran projects in the source code; COLLECTED
  2. -DUSE_FCMANGLE_H to use CMake-style determined Fortran name mangling conventions;
  3. -DHAVE_project to be used in the source code whenever segregation of code is to be achieved.

Detection of the Fortran name mangling conventions is to be done separately, by calling the fortran_enabler() macro defined in cmake/FortranEnabler.cmake A compatibility check of the Fortran and C++ ABIs will also be performed by this macro. The call to the macro has to be in the leaf CMakeLists.txt. For ERD it is in src/lib/CMakeLists.txt:

if(ENABLE_LIBERD)
   fortran_enabler()
   add_subdirectory(liberd)
endif()

Examples can be found in the source code and CMakeLists.txt files:

  1. git grep HAVE_ERD and git grep ENABLE_ERD for a type 1 Fortran code;
  2. git grep HAVE_PCMSOLVER and git grep ENABLE_PCMSOLVER for a type 2 Fortran code (not yet on master);
  3. git grep HAVE_FORTRAN for a type 3 Fortran code, currently only DKH.

How CMake determines Fortran mangling conventions

By means of the FortranCInterface.cmake module it is possible to setup the correct name mangling conventions for the chosen Fortran compiler, without having to know them before hand. That is, the --with-f77-symbol option is no longer needed. Furthermore, CMake can check if the Fortran and C++ compilers are compatible by compiling a small mixed-language project at configure time. If the compilers are not compatible CMake will exit with a fatal error.

Using the FCMangle.h header

CMake will also produce an header file FCMangle.h located in obj/include that contains the macros to be used to perform name mangling of:

  • global Fortran names without underscores. If the symbol to be mangled is amazing you would use #define amazing FC_GLOBAL(amazing, AMAZING)
  • global Fortran names with underscores. If the symbol to be mangled is amazing_stuff you would use #define amazing_stuff FC_GLOBAL_(amazing_stuff, AMAZING_STUFF)
  • module Fortran names without underscores. If the symbol to be mangled is amazing, included in the module fantastic, you would use #define amazing FC_MODULE(fantastic, amazing, FANTASTIC, AMAZING)
  • module Fortran names with underscores. If the symbol to be mangled is amazing_stuff, included in the module fantastic_stuff, you would use #define amazing_stuff FC_MODULE_(fantastic_stuff, amazing_stuff, FANTASTIC_STUFF, AMAZING_STUFF)

This CMake module can generate more complicated mangling patterns, based on arguments to its invocation in the root CMakeLists.txt, see here For the purposes of resolving name mangling of BLAS/LAPACK the basic usage seems to be enough.

This new system will have to coexist with the system based on autotools. To this end, I introduced the preprocessor variable USE_FCMANGLE_H that directs between the use of the autotools or CMake setup in the header files that declare the name mangled aliases to Fortran routines. For details on the modifications I made have a look at this commit

WARNING If you add any new Fortran subroutines that need resolution of name mangling you have to follow the current framework. The new source code would otherwise work only with either of the two configure&build frameworks!!

Adding source files

Instead of explicitly listing the source files in the directory, I preferred to use the file(GLOB ...) function provided by CMake. The macros defined in cmake/GlobUtils.cmake will glob all files with a given extension in the current directory and generate a CMake list (a ;-separated list of strings) I reverted to the explicit lists of source files in the CMakeLists.txt, since the globbing could potentially lead to a slowdown in compilation times. A template CMakeLists.txt can be generated by using the make_cmake_files.py Python script. The list of files thus generated can either be used verbatim or some of the files might be removed using the list(REMOVE_ITEM ...) function.

With the globbing approach one doesn't need to remember to add the source files in the list in the CMakeLists.txt. One disadvantage is that when adding new files to a directory, this doesn't trigger CMake to re-perform the globbing. To circumvent this, use the script touch_cmakelists.py script that is configured by CMake and resides in obj/bin. This script traverses the whole project tree and touches all CMakeLists.txt. You can alternatively just touch the CMakeLists.txt in the directory where new files where added.

With this approach, we can use the cloc Perl script to count lines of code in the various subdirectories and use the data to produce a graph like this, showing the number of blank, comment and code lines in Psi4. Applying compiler flags and compile definitions on a per-target basis can be made easier. Also, it scales better with programmer time, meaning that less time (none actually) is wasted in creating the source lists.

Number of blank, comment and code lines in Psi4 as counted by the cloc Perl script

How to add new sources?

The recursive nature of CMake is again exploited. The root directory for sources is src, its CMakeLists.txt directs CMake further down to bin or lib via add_subdirectory(...) commands. The CMakeLists.txt in these intermediates leaves is again a list of add_subdirectory(...) commands. If you added a new subdirectory to bin or src, you have to add an add_subdirectory(...) invocation to that CMakeLists.txt. Then go to your directory and invoke the Python script make_cmake_files.py (the one src/lib or in src/bin depending on you case) This will produce a template CMakeLists.txt that you will then need to modify to suit your needs. Let's look at a full example. The directory src/lib/amazingstuff was created. Open src/lib/CMakeLists.txt and add the following line:

add_subdirectory(amazingstuff)

now you can cd src/lib/amazingstuff and use the Python script:

python ../make_cmake_files.py amazingstuff --lang=CXX

The usage help for the script is:

usage: make_cmake_files.py [-h] [--lang [{CXX,C,F}]] [LIBNAME]

Create CMakeLists.txt template

positional arguments:
   LIBNAME             Name of the library to be created

optional arguments:
  -h, --help          show this help message and exit
  --lang [{CXX,C,F}]  Source file language

The script will create the template CMakeLists.txt.try and remind you that:

Template for amazingstuff created
Don't forget to fix excluded files and dependencies!!!

meaning that you have to explicitly specify which files are to be excluded from compilation (based on some CMake variable or whatever else) and which are the dependencies for the current target. Once that is done mv CMakeLists.txt.try CMakeLists.txt.

The final CMakeLists.txt should look like this:

set(headers_list "")
# List of headers
list(APPEND headers_list chkpt.h config.h )

# If you want to remove some headers specify them explictly here
if(DEVELOPMENT_CODE)
   list(REMOVE_ITEM headers_list "")
else()
   list(REMOVE_ITEM headers_list "")
endif()
# Sort alphabetically
list(SORT headers_list)

set(sources_list "")
# List of sources
list(APPEND sources_list grad.cc max_am.cc exist.cc natom_per_fragment.cc num_unique_atom.cc nshell.cc puream.cc eccsd.cc zmat.cc nallatom.cc openpi.cc usotao.cc frzcpi.cc sprim.cc num_unique_shell.cc rottype.cc ncalcs.cc fgeom.cc ecorr.cc e_labeled.cc natom.cc atom_dummy.cc label.cc nso.cc escf.cc fock.cc clsdpi.cc disp_irrep.cc override_occ.cc frzvpi.cc cdsalc2cd.cc symoper.cc nfragment.cc emp2.cc sym_label.cc nmo.cc keyword.cc atom_pos.cc rotconst.cc scf.cc efzc.cc iopen.cc lagr.cc cdsalcpi.cc sopi.cc prefix.cc disp.cc rot_symm_num.cc us2s.cc nprim.cc geom.cc close.cc nsymhf.cc init.cc exps.cc phase_check.cc nfzv.cc eref.cc contr.cc snuc.cc enuc.cc shells_per_am.cc fragment_coeff.cc shell_transm.cc nallatom_per_fragment.cc nirreps.cc usotbf.cc stype.cc ua2a.cc sloc_new.cc contr_full.cc nfzc.cc zvals.cc orbspi.cc ccvecs.cc rref.cc sloc.cc cartrep.cc e_t.cc statespi.cc exist_add_prefix.cc irr_labs.cc vib_freqs.cc ict.cc evals.cc felement.cc nref_per_fragment.cc ref.cc snumg.cc etot.cc nao.cc am2canon_shell_order.cc )

# If you want to remove some sources specify them explictly here
if(DEVELOPMENT_CODE)
   list(REMOVE_ITEM sources_list "vib_freqs.cc")
else()
   list(REMOVE_ITEM sources_list "vib_freqs.cc")
endif()

# Write list of files to be passed to cloc for counting lines of code.
# Only files that are actually compiled are counted.
set(to_count "${sources_list}" "${headers_list}")
write_to_cloc_list("${to_count}")

# Build static library
add_library(chkpt STATIC ${sources_list})
# Specify dependencies for the library (if any)
#add_dependencies(chkpt )
set(libs_to_merge chkpt ${libs_to_merge} PARENT_SCOPE)
if(BUILD_CUSTOM_BOOST)
   add_dependencies(chkpt custom_boost)
endif()
install(TARGETS chkpt ARCHIVE DESTINATION lib)

# Sets install directory for all the headers in the list
install_list_FILES("${headers_list}" include/libchkpt)

For further examples, look around in the various subdirectories of src/bin and src/lib.

Regression testing

The system I've put in place, traverses recursively the directories in tests to go look for CMakeLists.txt The root CMakeLists.txt is in tests and is just a list of add_subdirectory(test_dir) commands. The external executable dependent tests are added in the exact same way, if the external executable was found on the system. Each subdirectory (leaf) added in the root CMakeLists.txt has to have a CMakeLists.txt of its own, either directing CMake further down one level with additional add_subdirectory(test_dir) commands or giving the commands necessary to add a test (vide infra)

I've added the macro add_regression_test (found in cmake/testing/TestingMacros.cmake) to make addition of tests easier. The macro accepts two arguments: a name and a set of labels. Labels are used in order to categorize tests in subsets that can be run using CTest. The labels must be passed as a list, i.e. a ;-separated list of strings enclosed in double quotation marks. For example: add_regression_test(amazing_test "psi;amazing;cc") It is possible to specify more than one label per test. All tests must carry the label "psi". The following labels are also defined and describe how fast the test is run:

longtests = The (as of right now) 4 really long tests longertests = Every test in the test folder less longtests quicktests = Proper subset of tests deemed to run quickly

It is mandatory for all tests to carry one (and only one) length label! It is suggested to add at least a second label to specify the method tested.

Running tests

CTest produces the following final report:

97% tests passed, 8 tests failed out of 246

Label Time Summary: 
 adc            =  52.00 sec
 autotest       = 464.73 sec
 casscf         = 104.18 sec
 cc             = 4415.34 sec
 cdomp2         =   8.17 sec
 cepa           =  41.19 sec
 cisd           = 405.39 sec
 dcft           = 2018.97 sec
 df             = 129.69 sec
 dfmp2          =  70.65 sec
 dfomp2         =  29.18 sec
 dfscf          =  29.86 sec
 dft            = 447.46 sec
 docs           =   2.44 sec
 fci            =  31.73 sec
 findif         = 715.60 sec
 fnocc          = 504.10 sec
 freq           =  17.04 sec
 gradient       = 188.32 sec
 longertests    = 9831.32 sec
 longtests      = 297.29 sec
 mcscf          =   3.72 sec
 mints          =  14.24 sec
 misc           = 882.81 sec
 mp2            = 175.91 sec
 ocepa          = 120.44 sec
 omp            = 1050.27 sec
 opt            = 554.26 sec
 properties     =  49.68 sec
 psi            = 15718.70 sec
 psimrcc        =  54.03 sec
 pywrap         = 2576.88 sec
 quicktests     = 5590.09 sec
 rasci          =  33.58 sec
 sapt           = 424.44 sec
 scf            = 1266.79 sec
 tutorial       =  95.91 sec

Total Test time (real) = 800.83 sec

The following tests FAILED:
114 - dft2 (Failed)
126 - fd-freq-energy-large (Failed)
134 - frac (Failed)
137 - large_atoms (Failed)
184 - opt-lindep-change (Failed)
206 - psithon2 (Failed)
229 - sapt4 (Failed)
231 - scf-bz2 (Failed)
Errors while running CTest

CTest kills execution of tests that take more than a given time to complete. Currently, this timeout is set to half an hour. The value can be changed in the file cmake/ConfigTesting.cmake by setting the variable DART_TESTING_TIMEOUT (expressed in seconds)

Additional labels currently defined

  • scf
  • adc
  • cc
  • mcscf
  • casscf
  • dft
  • dcft
  • sapt
  • fci
  • cisd
  • cepa
  • cdomp2
  • dfmp2
  • df
  • dfscf
  • findif
  • fnocc
  • mints
  • mp2
  • ocepa
  • mp3
  • omp
  • opt
  • prop
  • psimrcc
  • pywrap
  • rasci
  • tutorial
  • misc
  • ext-exe-dep This is to be used for external executable dependent tests, i.e. Grimme's DFTD3, Stanton and Gauss' CFOUR and Kallay's MRCC tests.
  • dftd This is to be used with Grimme's DFTD3 tests
  • cfour This is to be used with Stanton and Gauss' CFOUR tests
  • mrcc This is to be used with Kallay's MRCC tests

Adding new tests

The mechanism is the same as for addition of new sources. We use add_subdirectory in the intermediate leaf CMakeLists.txt between the root and the directory actually containing the test. In the directory containing the test we add a CMakeLists.txt containing the following two lines:

include(TestingMacros)

add_regression_test(scf1 "psi;quicktests;scf")

That's it!

CDash

With CTest in place, it is possible to submit information about configuration, compilation and testing to a dashboard. Using the dashboard, it is possible to:

  • keep track of failing configurations;
  • keep track of failing builds;
  • keep track of failing tests;

so that it's possible to spot when one of the three broke, pinpoint the offending commit(s) and produce a fix. It is furthermore possible to:

  • run coverage builds, to analyse how much code functionality is covered by the regression test suite;
  • run dynamic analysis builds, either using Valgrind or sanitizers (sanitizers only work with Clang 3.6)

The dashboard is currently hosted on a server at KTH in Stockholm, courtesy of dr. Radovan Bast. The URL is: https://testboard.org/cdash/index.php?project=Psi To submit data for a build, you will need to register. The following is an example submission script:

#!/bin/bash

source /Users/roberto/.bashrc
export PATH=/opt/intel/composer_xe_2015.0.077/bin/intel64:/opt/intel/composer_xe_2015.0.077/mpirt/bin/intel64:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/opt/intel/composer_xe_2015.0.077/debugger/gdb/intel64/bin

PSI4_TMPDIR=/Users/roberto/Scratch/RDR-clang3.5-accelerate-debug
mkdir -p $PSI4_TMPDIR
export PSI4_TMPDIR
export NPROCS=`sysctl -n hw.ncpu`
export CTEST_MAKE_NUM_PROCS=$NPROCS

TMP_DIR=/Users/roberto/Scratch/tmprunpsi4/RDR-clang3.5-accelerate-debug
mkdir -p $TMP_DIR

git clone git@github.com:robertodr/psi4.git $TMP_DIR

cd $TMP_DIR

./setup --fc=gfortran --cc=clang --cxx=clang++ --type=debug --plugins=on --accelerate -D BUILDNAME=RDR-clang3.5-accelerate-debug -D SITE=merzbow

cd $TMP_DIR/obj

export PSI4DATADIR=$TMP_DIR/lib
mkdir -p $PSI4_TMPDIR/psi4scr
export PSI4_SCRATCH=$PSI4_TMPDIR/psi4scr

ctest -D Nightly -j$NPROCS

cd
rm -rf $PSI4_TMPDIR $TMP_DIR

exit 0

Important points to note:

  • a fresh clone of the repository is made in some temporary directory. Never run a regular dashboard submission build in your current working directory;
  • the additional -D BUILDNAME=RDR-clang3.5-accelerate-debug -D SITE=merzbow parameters to the setup script. These are needed to show where the build was executed (the name of the machine in this case, but it can really be anything) and some basic info on configuration. For BUILDNAME, I use this naming convention: {name initials}-{C++ compiler name and version}-{BLAS library used}-{build type}
  • the dashboard to submit to is in this case Nightly. The nightly builds are those executed every night (time zone is currently CET) so they are executed as cron jobs (most probably)

The available dashboards are Nightly, Experimental and Continuous. Experimental builds are those run irregularly or with a frequency other than daily. For example, you have configured, built and tested Psi4 locally and want to submit the result to the dashboard. You would then run ctest -D Experimental in you build directory. Dynamic analyses (with Valgrind and/or sanitizers) are run weekly. The results of configuration, build and testing are thus listed in the Experimental dashboard. Defects found by the dynamic analysis tools are instead listed under the "Dynamic Analysis" header. Code coverage analysis using gcov are instead run daily. The results of configuration, build and testing are listed in the Nightly dashboard. Code coverage statistics is available under the "Coverage" header. It is possible to expand the coverage results and see which lines are covered and which are not.

The Continuous dashboard is the one to be used for continuous integration server(s), server(s) that configure, build and test the project whenever a push to master (or some other branch) occurs. This is not currently set up for Psi4, but can be done in the future, possibly using Jenkins CI

I have setup a small repository containing "recipes" for dashboard submission that you can adapt to your machine(s)

CPack

Add a custom footer
Add a custom sidebar
Clone this wiki locally