Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Re-enable MKL support for FFTs #371

Merged
merged 1 commit into from
Nov 10, 2020
Merged

Re-enable MKL support for FFTs #371

merged 1 commit into from
Nov 10, 2020

Conversation

tskisner
Copy link
Member

@tskisner tskisner commented Nov 7, 2020

Our internal FFT routines have had support for directly using MKL for years. However, this was disabled when we moved to our modern CMake build system. This PR adds checks for MKL and if found does not look for FFTW. This works around the frequent issue where MKL ships with a partial FFTW interface and so linking to both libraries can break our code.

Tested on cori.nersc.gov with Intel compilers and MKL.

@tskisner
Copy link
Member Author

tskisner commented Nov 7, 2020

Although statistical unit tests pass with both FFT frameworks, I am working on comparing generated timestream values to make sure that they are the same. However, the build system changes in this PR can be reviewed already.

cmake/FindMKL.cmake Outdated Show resolved Hide resolved
@tskisner
Copy link
Member Author

For the record, I ran a comparison using the toast_satellite_sim.py example script and the following parameters:

--group-size
4
--nside
256
--no-day-maps
--no-season-maps
--no-sky
--simulate-noise
--sample-rate
20.0
--obs-time-h
1.0
--obs-num
10
--hwp-rpm
5.0
--madam
--madam-iter-max
10
--madam-baseline-length
60.0
--no-destripe
--no-wcov
--no-wcov-inv
--binmap
--hits
--focalplane
../fp_7.pkl

With a synthetic focalplane made with

toast_fake_focalplane.py \
    --minpix 7 \
    --out fp \
    --fwhm 10.0 \
    --fov 10.0 \
    --psd_fknee 0.1 \
    --psd_alpha 1.0 \
    --psd_fmin 1.0e-5 \
    --psd_NET 1.0 \
    --bandcenter_ghz 150.0 \
    --bandwidth_ghz 40.0

I used one cori KNL node with 16 processes. I ran this script with the cmbenv-gcc/20200630 and cmbenv-intel/20200630 software stacks (which have toast 2.3.8) and verified that the outputs agreed. Next I took the cmbenv-intel stack and compiled this branch of toast using just MKL without any linking to FFTW. However, note that the pipeline above does have other python packages that link to FFTW (Madam). This compilation of toast passes all unit tests and the outputs of the above pipeline agree with the other two stacks.

 * Add a cmake check for libmkl_rt.  This uses MKLROOT if set.
 * If MKL is found use it for BLAS/LAPACK and skip those checks.
 * If MKL is found disable checks for FFTW.
 * re-enable support for MKL DFFT.
 * In the TOAST environment setup, set the MKL threading layer
   to GNU or INTEL based on the compiler used.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants