Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for the STRUMPACK sparse direct solver and preconditioner [strumpack-dev] #222

Merged
merged 12 commits into from Jul 7, 2017

Conversation

@pghysels
Copy link
Contributor

@pghysels pghysels commented Jun 14, 2017

Pull request for comments about the integration of strumpack-dev into master.

  • For STRUMPACK, see: https://github.com/pghysels/STRUMPACK.
  • Look in example_build.sh and INSTALL.txt for installation instructions.
  • This branch requires metis, scotch, parmetis, ptscotch, blas, lapack, scalapack.

Test with the new Example 18p:

mpirun -np 4 ex18p -m ../data/square-disc.mesh --strumpack

TODO

@mlstowell

This comment has been minimized.

Copy link
Member

@mlstowell mlstowell commented on 27cdf66 Jun 10, 2017

Hi, Pieter,

I just had a chance to look through this and I think its an excellent start. It seems to closely follow the SuperLU implementation as we had planned. The issue you had mentioned regarding copying the matrix is also an issue with the SuperLU implementation. Perhaps in the future we can find a common solution but I see no immediate need to address this.

I would like to try to build and test the new code using the same version of STRUMPACK that you have developed against. Is "STRUMPACK-Sparse 1.1.0" from November 8, 2016 (on http://portal.nersc.gov/project/sparse/strumpack/) a suitable choice or is there something more recent?

Thanks,
Mark

to set STRUMPACK solver options.

Pass the command line arguments to STRUMPACK.
Remove some debugging statements.
Make the indentation more consistent.
@pghysels

This comment has been minimized.

Copy link
Contributor Author

@pghysels pghysels commented on a81eb2d Jun 11, 2017

Dear @mlstowell

Last week I moved the STRUMPACK code to a public github repository:
https://github.com/pghysels/STRUMPACK
For now, at least until we can release a new version of STRUMPACK, I think you best use the git version.
The STRUMPACK code has a script (example_build.sh) with examples on how to build the STRUMPACK code on the different machines I use. Please let me know if you have any issues with that, or have any other feedback. You can also look in the INSTALL.txt file or the user manual in doc/manual.tex.

And here are the steps I took in order to compile mfem with STRUMPACK. In the same folder as where I have the mfem/ source I run the following commands:
(note that I manually pass the parmetis, scotch and scalapack include folders and libs for now, we might have to make some changes to the build system for it to find those)

mkdir mfem-build
cd mfem-build
cmake ../mfem
-DCMAKE_BUILD_TYPE=Debug
-DMFEM_USE_MPI=YES
-DMFEM_USE_METIS_5=YES
-DMFEM_USE_OPENMP=YES
-DMFEM_THREAD_SAFE=ON
-DMFEM_USE_STRUMPACK=YES
-DMETIS_DIR=/home/pieterg/local/parmetis-4.0.3/build/Linux-x86_64/metis
-DMETIS_LIBRARIES="-L/home/pieterg/local/parmetis-4.0.3/build/Linux-x86_64/libparmetis/ -lparmetis
-L/home/pieterg/local/parmetis-4.0.3/build/Linux-x86_64/libmetis/ -lmetis"
-DSTRUMPACK_INCLUDE_DIRS="/home/pieterg/LBL/STRUMPACK/STRUMPACK-install/include;/home/pieterg/local/parmetis-4.0.3/metis/include;/home/pieterg/local/parmetis-4.0.3/include;/home/pieterg/local/scotch_6.0.4/include"
-DSTRUMPACK_LIBRARIES="-L/home/pieterg/LBL/STRUMPACK/STRUMPACK-install/lib/ -lstrumpack
-L/usr/lib/x86_64-linux-gnu/openmpi/lib/ -lmpi -lmpi_cxx -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lopenblas -lscalapack-openmpi -lblacs-openmpi
-L/home/pieterg/local/scotch_6.0.4/lib/ -lscotch -lscotcherr -lptscotch -lptscotcherr -lc -lstdc++ -lm -lc -lm -lquadmath -lm -lc -lgfortran"

make -j 4 VERBOSE=1
make ex18p VERBOSE=1

cd examples
mpirun -np 4 ex18p -m ../data/square-disc.mesh --strumpack

I will run some benchmarks next week with STRUMPACK as a preconditioner. This requires tuning some options. I think for now, it would be good if you can just try it as a direct solver, which is the default setting for now.

Pieter

@pghysels pghysels requested review from tzanio and mlstowell Jun 14, 2017
@tzanio tzanio requested a review from v-dobrev Jun 14, 2017
@tzanio tzanio changed the title Strumpack dev Support for the STRUMPACK sparse direct solver and preconditioner [strumpack-dev] Jun 14, 2017
tzanio added 2 commits Jun 14, 2017
@@ -0,0 +1,328 @@
// MFEM Example 18 - Parallel Version

This comment has been minimized.

@tzanio

tzanio Jun 14, 2017
Member

It will be best if we add the --strumpack option to ex11p instead of having a separate example. Can we do that?

This comment has been minimized.

@pghysels

pghysels Jun 16, 2017
Author Contributor

Yes, I can do that. Will do it soon.

@tzanio tzanio mentioned this pull request Jun 15, 2017
@tzanio
tzanio approved these changes Jun 28, 2017
@tzanio
Copy link
Member

@tzanio tzanio commented Jun 28, 2017

@pghysels, can you check the edits in the INSTALL and CHANGELOG files as well as this PR for updating the documentation? (Feel free to change anything as you see fit.)

I'd like to try to merge this today if possible. @mlstowell, @v-dobrev -- Any objections?

@pghysels
Copy link
Contributor Author

@pghysels pghysels commented Jun 28, 2017

STRUMPACK also depends on ParMetis and Metis.
In defaults.mk STRUMPACK_LIB includes SCOTCH_LIB and SCALAPACK_LIB, but not METIS_LIB. In defaults.mk, SUPERLU_LIB also does not depend include the ParMetis library. But in defaults.cmake, ParMETIS is listed as a requirement for SuperLUDist.

In INSTALL, it says that "STRUMPACK requires the PT-Scotch and Scalapack libraries." But STRUMPACK also requires ParMetis and Metis, just like SuperLUDist.
I just want to make sure you are aware of the dependency of STRUMPACK on (Par)METIS.

@tzanio
Copy link
Member

@tzanio tzanio commented Jun 28, 2017

@pghysels: ParMetis issues fixed in db368dd?

Copy link
Member

@mlstowell mlstowell left a comment

Everything looks great to me. Thanks for the new solver!

tzanio added 2 commits Jun 28, 2017
@tzanio tzanio added the in-next label Jun 28, 2017
@tzanio
Copy link
Member

@tzanio tzanio commented Jun 28, 2017

Merged in the next branch for testing. Thanks @pghysels and @xiaoyeli!

@tzanio tzanio merged commit 5b0ab29 into master Jul 7, 2017
3 checks passed
3 checks passed
continuous-integration/appveyor/branch AppVeyor build succeeded
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@tzanio tzanio deleted the strumpack-dev branch Jul 7, 2017
@tzanio tzanio added this to the Next release milestone Oct 22, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants
You can’t perform that action at this time.