deal.ii on Perlmutter

[pghysels]

Issues seem to come from TBB.

spack-build-out_dealii.txt

packages.yaml.txt

[pghysels]

@masterleinad

[masterleinad]

It looks like multiple TBB versions are clashing:

In file included from /usr/include/tbb/tbb_machine.h:113,
                 from /usr/include/tbb/atomic.h:42,
                 from /usr/include/tbb/pipeline.h:23,
                 from /tmp/pghysels/spack-stage/spack-stage-dealii-9.5.1-htmimkp74l7umnhzzeml57ivd5b5p2aj/spack-src/include/deal.II/base/work_stream.h:34,
                 from /tmp/pghysels/spack-stage/spack-stage-dealii-9.5.1-htmimkp74l7umnhzzeml57ivd5b5p2aj/spack-src/source/numerics/matrix_tools_once.cc:19:
/usr/include/tbb/tbb_stddef.h:434:14: error: expected type-specifier before 'split'
  434 |     operator split() const { return split(); }
      |              ^~~~~
In file included from /usr/include/tbb/pipeline.h:24,
                 from /tmp/pghysels/spack-stage/spack-stage-dealii-9.5.1-htmimkp74l7umnhzzeml57ivd5b5p2aj/spack-src/include/deal.II/base/work_stream.h:34,
                 from /tmp/pghysels/spack-stage/spack-stage-dealii-9.5.1-htmimkp74l7umnhzzeml57ivd5b5p2aj/spack-src/source/numerics/matrix_tools_once.cc:19:
/usr/include/tbb/task.h:145:9: error: reference to 'task_group_context' is ambiguous
  145 |         task_group_context& my_context;
      |         ^~~~~~~~~~~~~~~~~~
In file included from /pscratch/sd/p/pghysels/spack-xsdk/opt/spack/linux-sles15-zen3/gcc-11.2.0/intel-tbb-2021.9.0-hscma6sgogq6pt4etdwhsqfg3tmlknwy/include/tbb/task_group.h:17,
                 from /tmp/pghysels/spack-stage/spack-stage-dealii-9.5.1-htmimkp74l7umnhzzeml57ivd5b5p2aj/spack-src/include/deal.II/base/thread_management.h:43,
                 from /tmp/pghysels/spack-stage/spack-stage-dealii-9.5.1-htmimkp74l7umnhzzeml57ivd5b5p2aj/spack-src/include/deal.II/base/graph_coloring.h:23,
                 from /tmp/pghysels/spack-stage/spack-stage-dealii-9.5.1-htmimkp74l7umnhzzeml57ivd5b5p2aj/spack-src/include/deal.II/base/work_stream.h:22,
                 from /tmp/pghysels/spack-stage/spack-stage-dealii-9.5.1-htmimkp74l7umnhzzeml57ivd5b5p2aj/spack-src/source/numerics/matrix_tools_once.cc:19:
/pscratch/sd/p/pghysels/spack-xsdk/opt/spack/linux-sles15-zen3/gcc-11.2.0/intel-tbb-2021.9.0-hscma6sgogq6pt4etdwhsqfg3tmlknwy/include/oneapi/tbb/task_group.h:164:7: note: candidates are: 'class tbb::detail::d1::task_group_context'

The compiler both sees the one in /usr/inlude/tbb and the one in /pscratch/sd/p/pghysels/spack-xsdk/opt/spack/linux-sles15-zen3/gcc-11.2.0/intel-tbb-2021.9.0-hscma6sgogq6pt4etdwhsqfg3tmlknwy

[masterleinad]

Note that deal.II finds the latter:

-- Include /tmp/pghysels/spack-stage/spack-stage-dealii-9.5.1-htmimkp74l7umnhzzeml57ivd5b5p2aj/spack-src/cmake/configure/configure_10_tbb.cmake
-- Found TBB_LIBRARY
-- TBB_DEBUG_LIBRARY not found! The call was:
--     find_library(TBB_DEBUG_LIBRARY NAMES tbb_debug HINTS /pscratch/sd/p/pghysels/spack-xsdk/opt/spack/linux-sles15-zen3/gcc-11.2.0/intel-tbb-2021.9.0-hscma6sgogq6pt4etdwhsqfg3tmlknwy PATH_SUFFIXES lib lib64 lib)
-- Found TBB_INCLUDE_DIR
-- Processing TBB variables and targets
-- Processing TBB variables and targets - Done
-- 
-- Defining target: interface_tbb
--     VERSION:             2019.9
--     LINK_LIBRARIES:      /pscratch/sd/p/pghysels/spack-xsdk/opt/spack/linux-sles15-zen3/gcc-11.2.0/intel-tbb-2021.9.0-hscma6sgogq6pt4etdwhsqfg3tmlknwy/lib64/libtbb.so
--     INCLUDE_DIRECTORIES: /usr/include
-- 
-- DEAL_II_WITH_TBB successfully set up with external dependencies.

[pghysels]

Thanks.
Let me try to add the system TBB to my packages.yaml file.

[pghysels]

I haven't figured out how to just use the system TBB yet.
But it builds with ~threads.

[masterleinad]

But it builds with ~threads.

That's probably good enough for the short term.

[v-dobrev]

I'm seeing the same issue on Tioga: even though TBB_DIR is set to point to the spack-installed intel-tbb, dealii finds the TBB headers in /usr/include (the library location is found correctly, inside the given TBB_DIR).

One option to address this may be to update the dealii/package.py to always use the settings inside this if section: https://github.com/spack/spack/blob/9792625d1f2adcf0f6f8398c16edf412e4f76e4a/var/spack/repos/builtin/packages/dealii/package.py#L564-L576 -- I have not tested this but it looks like it should work.

[v-dobrev]

Just to add to my last comment: for now, I'm just using ^dealii~threads on Tioga.