Help needed for compilation with cmake

[abaillod]

Hello,

I am trying to use the cmake option to compile both SPEC and the python wrappers (latest master branch, commit dec8eb7
). The reason is that compiling SPEC, then the python wrappers is usually long and requires me to change the python wrapper makefile for every new commit (see #151), which is not optimal. So I thought maybe the cmake compilation is the solution to all of my problems! However I get a compilation error related to the FFTW library... What I did:

mkvirtualenv test
pip install cmake
cd SPEC
mkdir build
cd build
cmake ../.

I get:

...
[100%] Built target findFFTW_download
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.2") 
CMake Error at /home/abaillod/.virtualenvs/simsopt/lib/python3.6/site-packages/cmake/data/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find FFTW (missing: FFTW_INCLUDE_DIRS)
Call Stack (most recent call first):
  /home/abaillod/.virtualenvs/simsopt/lib/python3.6/site-packages/cmake/data/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
  build/findFFTW-src/FindFFTW.cmake:399 (find_package_handle_standard_args)
  CMakeLists.txt:99 (find_package)


-- Configuring incomplete, errors occurred!
See also "/home/abaillod/SPEC/build/CMakeFiles/CMakeOutput.log".
See also "/home/abaillod/SPEC/build/CMakeFiles/CMakeError.log".

Am I missing some kind of environment variable (FFTW_INCLUDE_DIRS) ? I tried printing some environment variable related to FFTW, I get

(simsopt) abaillod@spcpc602:~/SPEC/build> printenv | grep fftw
FFTW_DIR=/usr/local/fftw3
FFTW=/usr/local/fftw-2.1.5-gcc

Adding a new environment variable with

export FFTW_INCLUDE_DIRS=/usr/local/fftw3

does not work.

Anyone has a clue ? Note that compiling with make BUILD_ENV=intel_spc is working flawlessly. I am a total beginner with cmake and I don't know where to look. Do I need to write a setup file for cmake on my machine?

[mbkumar]

Could you please follow the steps posted at https://github.com/PrincetonUniversity/SPEC/wiki/Stellar-Compilation-Instructions and see if those instructions help?

[jonathanschilling]

I got the Makefile setup for building the python wrapper to work again.
Here are the steps to reproduce this.

1. Clone the latest state of Caoxiang's f90wrap and install it:

   git clone git@github.com:zhucaoxiang/f90wrap.git
   cd f90wrap
   pip install --user .

2. I assume that you start from a clean clone of the SPEC repository on the pywrap_makefile branch:
   git clone git@github.com:PrincetonUniversity/SPEC.git -b pywrap_makefile SPEC_pywrap_makefile
3. Go into the python_wrapper directory:
   cd SPEC_pywrap_makefile/Utilities/python_wrapper
4. Build SPEC itself and the Python wrapper (make sure to adjust BUILD_ENV to your targeted machine):
   make BUILD_ENV=gfortran_arch
   At the end, the SPEC version should be printed:

   ... lots of messages ...
   SPEC version: 3.1
   

5. Check the setup:
   1. Check that the version output works on the command line:
      python -c "from spec import spec; print('SPEC version: {:}'.format(spec.constants.version))"
   2. Check that SPEC can be run in stand-alone mode via the Python wrapper:
      python spec/core.py ../../InputFiles/TestCases/G3V01L0Fi.001.sp
   3. Check that the first example (optimization) for the Python wrapper works:
      python examples/example.py
   4. Check that the second example (iterative re-convergence) for the Python wrapper works:
      OMP_NUM_THREADS=1 python examples/example_2.py

@abaillod @zhucaoxiang
Can you please check if this works on your machines as well?

[abaillod]

@mbkumar, I am not sure to fully understand your link, especially point 6. As I said, I am a beginner with cmake and I have no idea what I am doing when I modify these files. Anyway since the Makefile seems to be fixed I will work with that. I am happy to test the cmake on my machine though if needed!

@jonathanschilling , thanks for fixing the Makefile! I however had to do some changes so that it works on my machine. I did the following changes:

@@ -7,11 +7,11 @@ include ${SOURCES}/SPECfile
 #                   define the compiler names
 #=======================================================================
 
-CC       = gcc
+#CC       = gcc
 
 # Fortran compiler
-F90      = gfortran
-#F90      = ifort
+#F90      = gfortran
+F90      = ifort
 #F90      =  /opt/intel/composer_xe_2015.3.187/bin/intel64/ifort
 
 PYTHON   = python3
@@ -28,16 +28,14 @@ ifeq ($(F90),gfortran)
 endif
 
 ifeq ($(F90),ifort)
-       # gfortran f90wrap temp files only. not compilation
-       FPP      = gfortran -E
-
-       FPP_F90FLAGS = -x f95-cpp-input -fPIC
-
-       # use 1 and 0 for True and False
-       F90FLAGS = -fpscomp logicals -fPIC
-
-       # for f2py
-       FCOMP    = intelem
+        FPP      = gfortran -E # gfortran f90wrap temp files only. not compilation
+        FPP_F90FLAGS = -x f95-cpp-input -fPIC
+        F90FLAGS = -fpscomp logicals -fPIC # use 1 and 0 for True and False
+        FCOMP    = intelem # for f2py
+        LINKS=-L${MKLROOT}/lib/intel64 -lmkl_rt -lpthread -lm -ldl
+        LINKS+=-L$(HDF5_HOME)/lib -lhdf5_fortran -lhdf5 -lpthread -lz -lm 
+        LINKS+=-L$(FFTW_DIR)/lib -lfftw3
+        LINKS+=-liomp5
 endif
 
 #=======================================================================

Then all the tests you propose were successful!

[jonathanschilling]

@abaillod Great to hear that this setup works for you! Can you maybe modify the Makefile in python_wrapper in a similar way as done in the SPECfile where flags are selected based on BUILD_ENV so that we do not need to edit it all the time?

[zhucaoxiang]

@jonathanschilling @abaillod I pushed one commit to adopt the Intel compiler. Now, you can use something like make all BUILD_ENV=intel_stellar F90=ifort.

But I do have a question testing python spec/core.py ../../InputFiles/TestCases/G3V01L0Fi.001.sp, while I can run xspec normally. I got the following error.

Invalid line in screenlist:  Wpp00aa = T
readin :            :
readin : ********** : Igeometry=  3 ; Istellsym=  1 ; Lreflect=  0 ;
readin :            : Lfreebound=  0 ; phiedge=  2.000000000000000E+00 ; curtor=  1.038123580200000E-09 ; curpol=  0.000000000000000E+00 ;
readin :            : gamma=  0.000000000000000E+00 ;
readin :            : Nfp=  5 ; Nvol=  1 ; Mvol=  1 ; Mpol=  4 ; Ntor=  4 ;
readin :            : pscale=  0.00000E+00 ; Ladiabatic= 0 ; Lconstraint=  0 ; mupf: tol,its=  1.00E-12 , 128 ;
readin :            : Lrad =  8,
readin :            :
readin : ********** : Linitialize=  0 ;LautoinitBn=  1 ; Lzerovac= 0 ; Ndiscrete= 2 ;
readin :            : Nquad=  -1 ; iMpol=  -4 ; iNtor=  -4 ;
readin :            : Lsparse= 0 ; Lsvdiota= 0 ; imethod= 3 ; iorder= 2 ; iprecon= 1 ; iotatol= -1.00000E+00 ;
readin :            : Lextrap= 0 ; Mregular= -1 ; Lrzaxis= 1 ; Ntoraxis= 3 ;
readin :            :
readin : ********** : LBeltrami= 4 ; Linitgues= 1 ; Lmatsolver= 3 ; LGMRESprec= 1 ; NiterGMRES= 200 ; epsGMRES=  1.00000E-14 ; epsILU=  1.00000E-12 ;
readin :            :
readin : ********** : Lfindzero= 2 ;
readin :            : escale=  0.00000E+00 ; opsilon=  1.00000E+00 ; pcondense=  4.000 ; epsilon=  1.00000E+00 ; wpoloidal= 1.0000 ; upsilon=  1.00000E+00 ;
readin :            : forcetol=  1.00000E-12 ; c05xmax=  1.00000E-06 ; c05xtol=  1.00000E-12 ; c05factor=  1.00000E-04 ; LreadGF= F ;
readin :            : mfreeits=   0 ; gBntol=  1.00000E-06 ; gBnbld=  6.66000E-01 ;
readin :            : vcasingeps=  1.00000E-12 ; vcasingtol=  1.00000E-08 ; vcasingits=     8 ; vcasingper=     1 ;
readin :            :
readin : ********** : odetol=  1.00E-07 ; nPpts=   500 ;
readin :            : LHevalues= F ; LHevectors= F ; LHmatrix= F ; Lperturbed= 0 ; dpp= -1 ; dqq= -1 ; dRZ=  1.00000000E-05 ; Lcheck=  0 ; Ltiming= F ;
readin :            :
readin :            : myid=  0 ; Rscale= 1.000000000000000E+01 ;
preset :            : myid=  0 ; Mrad=  8 : Lrad=  8,
preset : ********** : LBsequad= F , LBnewton= F , LBlinear= T ;
preset :            :
preset : ********** : Nquad=  -1 ; mn=   41 ; NGdof=     0 ; NAdof=   361,
preset :            :
preset : ********** : Nt=    32 ; Nz=    32 ; Ntz=     1024 ;
INTEL MKL ERROR: /opt/intel/oneapi/mkl/2021.1.1/lib/intel64/libmkl_avx512.so.1: undefined symbol: mkl_sparse_optimize_bsr_trsm_i8.
Intel MKL FATAL ERROR: Cannot load libmkl_avx512.so.1 or libmkl_def.so.1.
srun: error: stellar-k08n20: task 0: Exited with exit code 2
srun: launch/slurm: _step_signal: Terminating StepId=96299.0

The first line is interesting. It doesn't show in the xspec execution. The error seems happening with a linking error, so there must be an error. Do you guys have any clue? (I think I might have some issues with my mpi4py)

3. I think Jonathan's fix breaks CMAKE that @mbkumar set up. My understanding is that the CMAKE system or the steup.py script is compiling a library/package named spec and put it in the /spec folder. That's why it was using import spec.spec as spec in core.py. Right now, the import line is different. I haven't tested it on my side, but I guess the python setup.py install way doesn't work.

I didn't push a fix because a) I was waiting for the merge of my f90warp fork; b) a solution works for both Makefile and CMAKE. But I think we are really close. (Just use a consistent name on both sides).

[jonathanschilling]

@zhucaoxiang I think we should keep the adjusted name of the auto-generated Python wrapper as "spec_f90wrapped".
Otherwise, I feel that we have too many slightly different items named "spec" in this setup.

@mbkumar Can you adjust the CMake setup to use this different name?

[mbkumar]

I have no issues with using different names if that reduces confusion for everyone. I'll push a fix. Bharat Medasani Engineer Princeton Plasma Physics Lab (PPPL)

…

On Mon, Jan 24, 2022 at 6:14 AM Jonathan Schilling ***@***.***> wrote: @zhucaoxiang <https://github.com/zhucaoxiang> I think we should keep the adjusted name of the auto-generated Python wrapper as "spec_f90wrapped". Otherwise, I feel that we have too many slightly different items named "spec" in this setup. @mbkumar <https://github.com/mbkumar> Can you adjust the CMake setup to use this different name? — Reply to this email directly, view it on GitHub <#160 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA62VEC5MQLLGJ76IMVRYLTUXUYB7ANCNFSM5LVTAZSA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[abaillod]

This issue has been solved on my machine thanks to the recent fixes proposed by @jonathanschilling. I am closing it!