trilinos warning and memory consumption

[tjhei]

@gassmoeller reported large memory consumption when compiling trilinos and he saw the following:

CMake Warning at packages/tpetra/CMakeLists.txt:919 (MESSAGE):
  Tpetra_INST_COMPLEX_DOUBLE=ON, but Trilinos_ENABLE_COMPLEX_DOUBLE=OFF.
  This will work, but it will make builds more expensive, because it turns
  off explicit instantiation for complex<double> in KokkosKernels.  If you
  want to use Scalar=complex<double> in Tpetra, just set
  Trilinos_ENABLE_COMPLEX_DOUBLE=ON.  You do not need to set
  Tpetra_INST_COMPLEX_DOUBLE explicitly any more.

I need to test this.

[gassmoeller]

I have tested this a bit further. What I can already say is that setting Trilinos_ENABLE_COMPLEX_DOUBLE=ON removes the warning message. And that setting Trilinos_ENABLE_COMPLEX_DOUBLE=OFF and all the COMPLEX INST variables as well significantly reduces the memory usage (I am now able to compile -j6 with 32 GB RAM, which I could not do before).

I have not yet checked if/how much Trilinos_ENABLE_COMPLEX_DOUBLE=ON reduces the memory needs.

[gassmoeller]

Setting Trilinos_ENABLE_COMPLEX_DOUBLE=ON removes the warning, but I am again not able to compile with 6 processes and 32 GB RAM :-(

This is of course not an issue with candi or deal.II, but I think many users will be surprised if their compilation crashes even if they have 4GB/job on a laptop or desktop (and it can literally freeze the whole system if it starts swapping data). For large clusters and workstations it is less of an issue, because the total number of compilation units that need that much memory may not be large, but at least with the current default branch @jdannberg had the same problems with 16 jobs and 64 GB RAM.

[koecher]

@gassmoeller Thanks for reporting that. Have you tried using one build process only?

On laptops with 8GiB of RAM I do so, it takes longer time but usually doesn't break due to memory issues/swapping freeze while linking with more than one build process.

[gassmoeller]

@koecher, yes I can successfully compile with 4 processes and 32 GB RAM. This was not so much a question for help, more a question of if we are ok with this large memory requirement. We use candi as suggested installation procedure for deal.II in geodynamics/aspect and I am considering modifying our installation instructions to disable all the complex instantiations for our users, because they never need it and many are students with less powerful laptops (and waiting for several hours for compiling trilinos is problematic for short-term user workshops). I guess enabling the complex instantiations by default in candi is necessary, because deal.II by default enables the complex instantiations? If so it is of course fine to leave it enabled. But maybe deal.II should document somewhere that to install it using candi one needs 8GB RAM per job?

[koecher]

In the Readme of candi we have section on that already (multiple build jobs) with a warning for low memory systems.

Most packages are fine with 2 or more build jobs with 8GiB RAM, but not all. It is hard to say what is the recommendation.

I would be fine to disable complex support by default in Trilinos but I don't know if somebody needs it. As far as I know, there is no deal.II step tutorial using this feature of trilinos.

[gassmoeller]

In the Readme of candi we have section on that already (multiple build jobs) with a warning for low memory systems.

Ah, I did not see that, thanks. I guess I was mostly surprised because the memory requirements seem to have increased compared to earlier versions of trilinos.

I would be fine to disable complex support by default in Trilinos but I don't know if somebody needs it. As far as I know, there is no deal.II step tutorial using this feature of trilinos.

I am not sure how widely it is used. Maybe @tjhei has some insight?

[bangerth]

On 6/30/21 10:19 AM, Rene Gassmoeller wrote: I would be fine to disable complex support by default in Trilinos but I don't know if somebody needs it. As far as I know, there is no deal.II step tutorial using this feature of trilinos. I am not sure how widely it is used. Maybe @tjhei <https://github.com/tjhei> has some insight?

That is true, but there are tutorial programs that use complex-valued deal.II vectors. So we shouldn't disable complex deal.II vectors. I don't know whether we can disable complex Trilinos vectors without also disabling complex deal.II vectors.