New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for the STRUMPACK sparse direct solver and preconditioner [strumpack-dev] #222

Merged
merged 12 commits into from Jul 7, 2017

Conversation

Projects
None yet
3 participants
@pghysels
Contributor

pghysels commented Jun 14, 2017

Pull request for comments about the integration of strumpack-dev into master.

  • For STRUMPACK, see: https://github.com/pghysels/STRUMPACK.
  • Look in example_build.sh and INSTALL.txt for installation instructions.
  • This branch requires metis, scotch, parmetis, ptscotch, blas, lapack, scalapack.

Test with the new Example 18p:

mpirun -np 4 ex18p -m ../data/square-disc.mesh --strumpack

TODO

@mlstowell

This comment has been minimized.

Show comment
Hide comment
@mlstowell

mlstowell Jun 10, 2017

Member

Hi, Pieter,

I just had a chance to look through this and I think its an excellent start. It seems to closely follow the SuperLU implementation as we had planned. The issue you had mentioned regarding copying the matrix is also an issue with the SuperLU implementation. Perhaps in the future we can find a common solution but I see no immediate need to address this.

I would like to try to build and test the new code using the same version of STRUMPACK that you have developed against. Is "STRUMPACK-Sparse 1.1.0" from November 8, 2016 (on http://portal.nersc.gov/project/sparse/strumpack/) a suitable choice or is there something more recent?

Thanks,
Mark

Member

mlstowell commented on 27cdf66 Jun 10, 2017

Hi, Pieter,

I just had a chance to look through this and I think its an excellent start. It seems to closely follow the SuperLU implementation as we had planned. The issue you had mentioned regarding copying the matrix is also an issue with the SuperLU implementation. Perhaps in the future we can find a common solution but I see no immediate need to address this.

I would like to try to build and test the new code using the same version of STRUMPACK that you have developed against. Is "STRUMPACK-Sparse 1.1.0" from November 8, 2016 (on http://portal.nersc.gov/project/sparse/strumpack/) a suitable choice or is there something more recent?

Thanks,
Mark

Add a number of routines to the STRUMPACKSolver class
to set STRUMPACK solver options.

Pass the command line arguments to STRUMPACK.
Remove some debugging statements.
Make the indentation more consistent.
@pghysels

This comment has been minimized.

Show comment
Hide comment
@pghysels

pghysels Jun 11, 2017

Contributor

Dear @mlstowell

Last week I moved the STRUMPACK code to a public github repository:
https://github.com/pghysels/STRUMPACK
For now, at least until we can release a new version of STRUMPACK, I think you best use the git version.
The STRUMPACK code has a script (example_build.sh) with examples on how to build the STRUMPACK code on the different machines I use. Please let me know if you have any issues with that, or have any other feedback. You can also look in the INSTALL.txt file or the user manual in doc/manual.tex.

And here are the steps I took in order to compile mfem with STRUMPACK. In the same folder as where I have the mfem/ source I run the following commands:
(note that I manually pass the parmetis, scotch and scalapack include folders and libs for now, we might have to make some changes to the build system for it to find those)

mkdir mfem-build
cd mfem-build
cmake ../mfem
-DCMAKE_BUILD_TYPE=Debug
-DMFEM_USE_MPI=YES
-DMFEM_USE_METIS_5=YES
-DMFEM_USE_OPENMP=YES
-DMFEM_THREAD_SAFE=ON
-DMFEM_USE_STRUMPACK=YES
-DMETIS_DIR=/home/pieterg/local/parmetis-4.0.3/build/Linux-x86_64/metis
-DMETIS_LIBRARIES="-L/home/pieterg/local/parmetis-4.0.3/build/Linux-x86_64/libparmetis/ -lparmetis
-L/home/pieterg/local/parmetis-4.0.3/build/Linux-x86_64/libmetis/ -lmetis"
-DSTRUMPACK_INCLUDE_DIRS="/home/pieterg/LBL/STRUMPACK/STRUMPACK-install/include;/home/pieterg/local/parmetis-4.0.3/metis/include;/home/pieterg/local/parmetis-4.0.3/include;/home/pieterg/local/scotch_6.0.4/include"
-DSTRUMPACK_LIBRARIES="-L/home/pieterg/LBL/STRUMPACK/STRUMPACK-install/lib/ -lstrumpack
-L/usr/lib/x86_64-linux-gnu/openmpi/lib/ -lmpi -lmpi_cxx -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lopenblas -lscalapack-openmpi -lblacs-openmpi
-L/home/pieterg/local/scotch_6.0.4/lib/ -lscotch -lscotcherr -lptscotch -lptscotcherr -lc -lstdc++ -lm -lc -lm -lquadmath -lm -lc -lgfortran"

make -j 4 VERBOSE=1
make ex18p VERBOSE=1

cd examples
mpirun -np 4 ex18p -m ../data/square-disc.mesh --strumpack

I will run some benchmarks next week with STRUMPACK as a preconditioner. This requires tuning some options. I think for now, it would be good if you can just try it as a direct solver, which is the default setting for now.

Pieter

Contributor

pghysels commented on a81eb2d Jun 11, 2017

Dear @mlstowell

Last week I moved the STRUMPACK code to a public github repository:
https://github.com/pghysels/STRUMPACK
For now, at least until we can release a new version of STRUMPACK, I think you best use the git version.
The STRUMPACK code has a script (example_build.sh) with examples on how to build the STRUMPACK code on the different machines I use. Please let me know if you have any issues with that, or have any other feedback. You can also look in the INSTALL.txt file or the user manual in doc/manual.tex.

And here are the steps I took in order to compile mfem with STRUMPACK. In the same folder as where I have the mfem/ source I run the following commands:
(note that I manually pass the parmetis, scotch and scalapack include folders and libs for now, we might have to make some changes to the build system for it to find those)

mkdir mfem-build
cd mfem-build
cmake ../mfem
-DCMAKE_BUILD_TYPE=Debug
-DMFEM_USE_MPI=YES
-DMFEM_USE_METIS_5=YES
-DMFEM_USE_OPENMP=YES
-DMFEM_THREAD_SAFE=ON
-DMFEM_USE_STRUMPACK=YES
-DMETIS_DIR=/home/pieterg/local/parmetis-4.0.3/build/Linux-x86_64/metis
-DMETIS_LIBRARIES="-L/home/pieterg/local/parmetis-4.0.3/build/Linux-x86_64/libparmetis/ -lparmetis
-L/home/pieterg/local/parmetis-4.0.3/build/Linux-x86_64/libmetis/ -lmetis"
-DSTRUMPACK_INCLUDE_DIRS="/home/pieterg/LBL/STRUMPACK/STRUMPACK-install/include;/home/pieterg/local/parmetis-4.0.3/metis/include;/home/pieterg/local/parmetis-4.0.3/include;/home/pieterg/local/scotch_6.0.4/include"
-DSTRUMPACK_LIBRARIES="-L/home/pieterg/LBL/STRUMPACK/STRUMPACK-install/lib/ -lstrumpack
-L/usr/lib/x86_64-linux-gnu/openmpi/lib/ -lmpi -lmpi_cxx -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lopenblas -lscalapack-openmpi -lblacs-openmpi
-L/home/pieterg/local/scotch_6.0.4/lib/ -lscotch -lscotcherr -lptscotch -lptscotcherr -lc -lstdc++ -lm -lc -lm -lquadmath -lm -lc -lgfortran"

make -j 4 VERBOSE=1
make ex18p VERBOSE=1

cd examples
mpirun -np 4 ex18p -m ../data/square-disc.mesh --strumpack

I will run some benchmarks next week with STRUMPACK as a preconditioner. This requires tuning some options. I think for now, it would be good if you can just try it as a direct solver, which is the default setting for now.

Pieter

@pghysels pghysels requested review from tzanio and mlstowell Jun 14, 2017

@tzanio tzanio requested a review from v-dobrev Jun 14, 2017

@tzanio tzanio changed the title from Strumpack dev to Support for the STRUMPACK sparse direct solver and preconditioner [strumpack-dev] Jun 14, 2017

tzanio added some commits Jun 14, 2017

Show outdated Hide outdated examples/ex18p.cpp

@tzanio tzanio referenced this pull request Jun 15, 2017

Closed

Developer's guide? #223

@tzanio

tzanio approved these changes Jun 28, 2017

@tzanio

This comment has been minimized.

Show comment
Hide comment
@tzanio

tzanio Jun 28, 2017

Member

@pghysels, can you check the edits in the INSTALL and CHANGELOG files as well as this PR for updating the documentation? (Feel free to change anything as you see fit.)

I'd like to try to merge this today if possible. @mlstowell, @v-dobrev -- Any objections?

Member

tzanio commented Jun 28, 2017

@pghysels, can you check the edits in the INSTALL and CHANGELOG files as well as this PR for updating the documentation? (Feel free to change anything as you see fit.)

I'd like to try to merge this today if possible. @mlstowell, @v-dobrev -- Any objections?

@pghysels

This comment has been minimized.

Show comment
Hide comment
@pghysels

pghysels Jun 28, 2017

Contributor

STRUMPACK also depends on ParMetis and Metis.
In defaults.mk STRUMPACK_LIB includes SCOTCH_LIB and SCALAPACK_LIB, but not METIS_LIB. In defaults.mk, SUPERLU_LIB also does not depend include the ParMetis library. But in defaults.cmake, ParMETIS is listed as a requirement for SuperLUDist.

In INSTALL, it says that "STRUMPACK requires the PT-Scotch and Scalapack libraries." But STRUMPACK also requires ParMetis and Metis, just like SuperLUDist.
I just want to make sure you are aware of the dependency of STRUMPACK on (Par)METIS.

Contributor

pghysels commented Jun 28, 2017

STRUMPACK also depends on ParMetis and Metis.
In defaults.mk STRUMPACK_LIB includes SCOTCH_LIB and SCALAPACK_LIB, but not METIS_LIB. In defaults.mk, SUPERLU_LIB also does not depend include the ParMetis library. But in defaults.cmake, ParMETIS is listed as a requirement for SuperLUDist.

In INSTALL, it says that "STRUMPACK requires the PT-Scotch and Scalapack libraries." But STRUMPACK also requires ParMetis and Metis, just like SuperLUDist.
I just want to make sure you are aware of the dependency of STRUMPACK on (Par)METIS.

@tzanio

This comment has been minimized.

Show comment
Hide comment
@tzanio

tzanio Jun 28, 2017

Member

@pghysels: ParMetis issues fixed in db368dd?

Member

tzanio commented Jun 28, 2017

@pghysels: ParMetis issues fixed in db368dd?

@mlstowell

Everything looks great to me. Thanks for the new solver!

tzanio added some commits Jun 28, 2017

@tzanio tzanio added the in-next label Jun 28, 2017

@tzanio

This comment has been minimized.

Show comment
Hide comment
@tzanio

tzanio Jun 28, 2017

Member

Merged in the next branch for testing. Thanks @pghysels and @xiaoyeli!

Member

tzanio commented Jun 28, 2017

Merged in the next branch for testing. Thanks @pghysels and @xiaoyeli!

@tzanio tzanio merged commit 5b0ab29 into master Jul 7, 2017

3 checks passed

continuous-integration/appveyor/branch AppVeyor build succeeded
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@tzanio tzanio deleted the strumpack-dev branch Jul 7, 2017

@tzanio tzanio added this to the Next release milestone Oct 22, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment