Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Trilinos nightly failure, Cuda+UVM build, ifpack2/stokhos/sacado interaction in Ifpack2_LocalSparseTriangularSolver_def #2114

Closed
ndellingwood opened this issue Feb 19, 2024 · 0 comments
Closed

Trilinos nightly failure, Cuda+UVM build, ifpack2/stokhos/sacado interaction in Ifpack2_LocalSparseTriangularSolver_def #2114

ndellingwood opened this issue Feb 19, 2024 · 0 comments
Assignees
@jgfouca
Labels
bug develop-only bug InDevelop

Comments

@ndellingwood
Copy link
Contributor

Nightly Trilinos builds with Cuda+UVM backend fail to compile with errors tracing to Stokhos-Ifpack2 interaction with the Ifpack2_LocalSparseTriangularSolver_def.cpp code:

23:45:11 /home/jenkins/weaver/workspace/KokkosEco_Trilinos_Weaver_CUDA112_opt-uvm/Trilinos/packages/stokhos/src/sacado/kokkos/vector/KokkosExp_View_MP_Vector_Contiguous.hpp(1510): error: no operator "=" matches these operands
23:45:11             operand types are: Kokkos::Impl::CudaLDGFetch<std::add_const_t<const double>, std::conditional_t<false, int, std::conditional_t<true, int2, std::conditional_t<false, int4, void>>>> = const double *const
23:45:11           detected during:
23:45:11             instantiation of "void Kokkos::Impl::ViewMapping<DstTraits, SrcTraits, std::enable_if<<expression>, DstTraits::specialize>::type>::assign(Kokkos::Impl::ViewMapping<DstTraits, SrcTraits, std::enable_if<<expression>, DstTraits::specialize>::type>::DstType &, const Kokkos::Impl::ViewMapping<DstTraits, SrcTraits, std::enable_if<<expression>, DstTraits::specialize>::type>::SrcType &, const Kokkos::Impl::ViewMapping<DstTraits, SrcTraits, std::enable_if<<expression>, DstTraits::specialize>::type>::TrackType &) [with DstTraits=Kokkos::ViewTraits<const double **, std::conditional_t<true, Kokkos::LayoutLeft, Kokkos::Serial::array_layout>, Kokkos::Device<Kokkos::HostSpace::execution_space, Kokkos::Cuda::memory_space>, Kokkos::MemoryTraits<3U>>, SrcTraits=Kokkos::ViewTraits<std::add_const_t<Sacado::MP::Vector<SFS_int_double_16_DFN_CPU_default_local_ordinal_type_default_global_ordinal_type_Tpetra_KokkosCompat_KokkosCudaWrapperNode>> **, std::conditional_t<true, Kokkos::LayoutLeft, Kokkos::Serial::array_layout>, Kokkos::Device<Kokkos::HostSpace::execution_space, Kokkos::Cuda::memory_space>, Kokkos::MemoryTraits<3U>>]" 
23:45:11 /home/jenkins/weaver/workspace/KokkosEco_Trilinos_Weaver_CUDA112_opt-uvm/Trilinos/kokkos/core/src/Kokkos_View.hpp(1355): here
23:45:11             instantiation of "Kokkos::View<DataType, Properties...>::View(const Kokkos::View<RT, RP...> &, std::enable_if_t<Kokkos::Impl::ViewMapping<Kokkos::View<DataType, Properties...>::traits, Kokkos::View<RT, RP...>::traits, Kokkos::ViewTraits<DataType, Properties...>::specialize>::is_assignable_data_type, void> *) [with DataType=const double **, Properties=<std::conditional_t<true, Kokkos::LayoutLeft, Kokkos::Serial::array_layout>, Kokkos::Device<Kokkos::HostSpace::execution_space, Kokkos::Cuda::memory_space>, Kokkos::MemoryTraits<3U>>, RT=std::add_const_t<Sacado::MP::Vector<SFS_int_double_16_DFN_CPU_default_local_ordinal_type_default_global_ordinal_type_Tpetra_KokkosCompat_KokkosCudaWrapperNode>> **, RP=<std::conditional_t<true, Kokkos::LayoutLeft, Kokkos::Serial::array_layout>, Kokkos::Device<Kokkos::HostSpace::execution_space, Kokkos::Cuda::memory_space>, Kokkos::MemoryTraits<3U>>]" 
23:45:11 (440): here
23:45:11             instantiation of "void Kokkos::deep_copy(const ExecSpace &, const Kokkos::View<DT, DP...> &, const Kokkos::View<ST, SP...> &, std::enable_if<<expression>, void>::type *) [with ExecSpace=std::conditional_t<true, Kokkos::DefaultHostExecutionSpace, Kokkos::CudaSpace::execution_space>, DT=Sacado::MP::Vector<SFS_int_double_16_DFN_CPU_default_local_ordinal_type_default_global_ordinal_type_Tpetra_KokkosCompat_KokkosCudaWrapperNode> **, DP=<std::conditional_t<true, Kokkos::LayoutLeft, Kokkos::Serial::array_layout>, Kokkos::Device<Kokkos::HostSpace::execution_space, Kokkos::Cuda::memory_space>, Kokkos::MemoryUnmanaged>, ST=std::add_const_t<Sacado::MP::Vector<SFS_int_double_16_DFN_CPU_default_local_ordinal_type_default_global_ordinal_type_Tpetra_KokkosCompat_KokkosCudaWrapperNode>> **, SP=<std::conditional_t<true, Kokkos::LayoutLeft, Kokkos::Serial::array_layout>, Kokkos::Device<Kokkos::HostSpace::execution_space, Kokkos::Cuda::memory_space>, Kokkos::MemoryTraits<3U>>]" 
23:45:11 (458): here
23:45:11             instantiation of "void Kokkos::deep_copy(const Kokkos::View<DT, DP...> &, const Kokkos::View<ST, SP...> &, std::enable_if<<expression>, void>::type *) [with DT=Sacado::MP::Vector<SFS_int_double_16_DFN_CPU_default_local_ordinal_type_default_global_ordinal_type_Tpetra_KokkosCompat_KokkosCudaWrapperNode> **, DP=<std::conditional_t<true, Kokkos::LayoutLeft, Kokkos::Serial::array_layout>, Kokkos::Device<Kokkos::HostSpace::execution_space, Kokkos::Cuda::memory_space>, Kokkos::MemoryUnmanaged>, ST=std::add_const_t<Sacado::MP::Vector<SFS_int_double_16_DFN_CPU_default_local_ordinal_type_default_global_ordinal_type_Tpetra_KokkosCompat_KokkosCudaWrapperNode>> **, SP=<std::conditional_t<true, Kokkos::LayoutLeft, Kokkos::Serial::array_layout>, Kokkos::Device<Kokkos::HostSpace::execution_space, Kokkos::Cuda::memory_space>, Kokkos::MemoryTraits<3U>>]" 
23:45:11 /home/jenkins/weaver/workspace/KokkosEco_Trilinos_Weaver_CUDA112_opt-uvm/Trilinos/kokkos-kernels/sparse/impl/KokkosSparse_trsv_impl.hpp(186): here
23:45:11             instantiation of "void KokkosSparse::Impl::Sequential::TrsvWrap<CrsMatrixType, DomainMultiVectorType, RangeMultiVectorType>::lowerTriSolveCsrUnitDiag(RangeMultiVectorType, const CrsMatrixType &, DomainMultiVectorType) [with CrsMatrixType=std::conditional_t<true, KokkosSparse::CrsMatrix<std::add_const_t<Sacado::MP::Vector<SFS_int_double_16_DFN_CPU_default_local_ordinal_type_default_global_ordinal_type_Tpetra_KokkosCompat_KokkosCudaWrapperNode>>, std::add_const_t<int>, Kokkos::Device<Kokkos::CudaUVMSpace::execution_space, Kokkos::CudaUVMSpace::memory_space>, Kokkos::MemoryTraits<1U>, std::add_const_t<const size_t>>, KokkosSparse::Experimental::BsrMatrix<std::add_const_t<Sacado::MP::Vector<SFS_int_double_16_DFN_CPU_default_local_ordinal_type_default_global_ordinal_type_Tpetra_KokkosCompat_KokkosCudaWrapperNode>>, std::add_const_t<int>, Kokkos::Device<Kokkos::CudaUVMSpace::execution_space, Kokkos::CudaUVMSpace::memory_space>, Kokkos::MemoryTraits<1U>, std::add_const_t<const size_t>>>, DomainMultiVectorType=Kokkos::View<std::add_const_t<Sacado::MP::Vector<SFS_int_double_16_DFN_CPU_default_local_ordinal_type_default_global_ordinal_type_Tpetra_KokkosCompat_KokkosCudaWrapperNode>> **, std::conditional_t<true, Kokkos::LayoutLeft, Kokkos::Serial::array_layout>, Kokkos::Device<Kokkos::HostSpace::execution_space, Kokkos::Cuda::memory_space>, Kokkos::MemoryTraits<3U>>, RangeMultiVectorType=Kokkos::View<Sacado::MP::Vector<SFS_int_double_16_DFN_CPU_default_local_ordinal_type_default_global_ordinal_type_Tpetra_KokkosCompat_KokkosCudaWrapperNode> **, std::conditional_t<true, Kokkos::LayoutLeft, Kokkos::Serial::array_layout>, Kokkos::Device<Kokkos::HostSpace::execution_space, Kokkos::Cuda::memory_space>, Kokkos::MemoryUnmanaged>]" 
23:45:11 /home/jenkins/weaver/workspace/KokkosEco_Trilinos_Weaver_CUDA112_opt-uvm/Trilinos/kokkos-kernels/sparse/impl/KokkosSparse_trsv_spec.hpp(118): here
23:45:11             instantiation of "void KokkosSparse::Impl::TRSV<CrsMatrixType, DomainMultiVectorType, RangeMultiVectorType, false, false>::trsv(const char *, const char *, const char *, const CrsMatrixType &, DomainMultiVectorType, RangeMultiVectorType) [with CrsMatrixType=std::conditional_t<true, KokkosSparse::CrsMatrix<std::add_const_t<Sacado::MP::Vector<SFS_int_double_16_DFN_CPU_default_local_ordinal_type_default_global_ordinal_type_Tpetra_KokkosCompat_KokkosCudaWrapperNode>>, std::add_const_t<int>, Kokkos::Device<Kokkos::CudaUVMSpace::execution_space, Kokkos::CudaUVMSpace::memory_space>, Kokkos::MemoryTraits<1U>, std::add_const_t<const size_t>>, KokkosSparse::Experimental::BsrMatrix<std::add_const_t<Sacado::MP::Vector<SFS_int_double_16_DFN_CPU_default_local_ordinal_type_default_global_ordinal_type_Tpetra_KokkosCompat_KokkosCudaWrapperNode>>, std::add_const_t<int>, Kokkos::Device<Kokkos::CudaUVMSpace::execution_space, Kokkos::CudaUVMSpace::memory_space>, Kokkos::MemoryTraits<1U>, std::add_const_t<const size_t>>>, DomainMultiVectorType=Kokkos::View<std::add_const_t<Sacado::MP::Vector<SFS_int_double_16_DFN_CPU_default_local_ordinal_type_default_global_ordinal_type_Tpetra_KokkosCompat_KokkosCudaWrapperNode>> **, std::conditional_t<true, Kokkos::LayoutLeft, Kokkos::Serial::array_layout>, Kokkos::Device<Kokkos::HostSpace::execution_space, Kokkos::Cuda::memory_space>, Kokkos::MemoryTraits<3U>>, RangeMultiVectorType=Kokkos::View<Sacado::MP::Vector<SFS_int_double_16_DFN_CPU_default_local_ordinal_type_default_global_ordinal_type_Tpetra_KokkosCompat_KokkosCudaWrapperNode> **, std::conditional_t<true, Kokkos::LayoutLeft, Kokkos::Serial::array_layout>, Kokkos::Device<Kokkos::HostSpace::execution_space, Kokkos::Cuda::memory_space>, Kokkos::MemoryUnmanaged>]" 
23:45:11 /home/jenkins/weaver/workspace/KokkosEco_Trilinos_Weaver_CUDA112_opt-uvm/Trilinos/kokkos-kernels/sparse/src/KokkosSparse_trsv.hpp(154): here
23:45:11             instantiation of "void KokkosSparse::trsv(const char *, const char *, const char *, const AMatrix &, const BMV &, const XMV &) [with AMatrix=KokkosSparse::CrsMatrix<Sacado::MP::Vector<SFS_int_double_16_DFN_CPU_default_local_ordinal_type_default_global_ordinal_type_Tpetra_KokkosCompat_KokkosCudaWrapperNode>, default_local_ordinal_type, std::conditional_t<true, Kokkos::Cuda::memory_space, std::conditional_t<false, std::conditional_t<true, Kokkos::CudaSpace::execution_space, Kokkos::DefaultExecutionSpace>, std::conditional_t<true, Kokkos::Device<Kokkos::HostSpace::execution_space, Kokkos::Cuda::memory_space>, Kokkos::HostSpace>>>, void, size_t>, BMV=Kokkos::View<std::add_const_t<Sacado::MP::Vector<SFS_int_double_16_DFN_CPU_default_local_ordinal_type_default_global_ordinal_type_Tpetra_KokkosCompat_KokkosCudaWrapperNode>> **, std::conditional_t<true, Kokkos::LayoutLeft, Kokkos::Serial::array_layout>, Kokkos::Device<Kokkos::HostSpace::execution_space, Kokkos::Cuda::memory_space>, std::conditional_t<true, std::conditional_t<false, void, Kokkos::Experimental::DefaultViewHooks>, Kokkos::Experimental::DefaultViewHooks>, std::conditional_t<false, void, Kokkos::MemoryManaged>>, XMV=Kokkos::View<Sacado::MP::Vector<SFS_int_double_16_DFN_CPU_default_local_ordinal_type_default_global_ordinal_type_Tpetra_KokkosCompat_KokkosCudaWrapperNode> **, std::conditional_t<true, Kokkos::Cuda::array_layout, Kokkos::Cuda::array_layout>, Kokkos::Device<Kokkos::HostSpace::execution_space, Kokkos::Cuda::memory_space>, std::conditional_t<false, void, Kokkos::Experimental::DefaultViewHooks>>]" 
23:45:11 /home/jenkins/weaver/workspace/KokkosEco_Trilinos_Weaver_CUDA112_opt-uvm/Trilinos/packages/ifpack2/src/Ifpack2_LocalSparseTriangularSolver_def.hpp(989): here
23:45:11             instantiation of "void Ifpack2::LocalSparseTriangularSolver<MatrixType>::localTriangularSolve(const Ifpack2::LocalSparseTriangularSolver<MatrixType>::MV &, Ifpack2::LocalSparseTriangularSolver<MatrixType>::MV &, Teuchos::ETransp) const [with MatrixType=Tpetra::RowMatrix<Sacado::MP::Vector<SFS_int_double_16_DFN_CPU_default_local_ordinal_type_default_global_ordinal_type_Tpetra_KokkosCompat_KokkosCudaWrapperNode>, default_local_ordinal_type, default_global_ordinal_type, Tpetra_KokkosCompat_KokkosCudaWrapperNode>]" 
23:45:11 /home/jenkins/weaver/workspace/KokkosEco_Trilinos_Weaver_CUDA112_opt-uvm/Trilinos/packages/ifpack2/src/Ifpack2_LocalSparseTriangularSolver_def.hpp(1028): here
23:45:11 
23:45:11 1 error detected in the compilation of "/home/jenkins/weaver/workspace/KokkosEco_Trilinos_Weaver_CUDA112_opt-uvm/Build/packages/stokhos/src/Ifpack2_LocalSparseTriangularSolver_MP_Vector_16_Cuda.cpp".
23:45:11 make[2]: *** [packages/stokhos/src/CMakeFiles/stokhos_ifpack2_mp_16_cuda.dir/build.make:314: packages/stokhos/src/CMakeFiles/stokhos_ifpack2_mp_16_cuda.dir/Ifpack2_LocalSparseTriangularSolver_MP_Vector_16_Cuda.cpp.o] Error 1

These errors surfaced after #2104 and were not resolved by the updates in #2111 . Based on looking at the output, were there changes with the trsv updates that may implicitly require non-specialized (i.e. non-FAD) type Views which was supported before, and could be tripping the static_assert when trying to assign a View of Fad type? Adding @jgfouca

Reproducer (weaver rhel8 queue):

# Repos
git clone -b develop https://github.com/trilinos/Trilinos.git
TRILINOS_DIR=$PWD/Trilinos
git clone -b develop https://github.com/kokkos/kokkos.git
KOKKOS_DIR=$PWD/kokkos
git clone -b develop https://github.com/kokkos/kokkos-kernels.git
KOKKOSKERNELS_DIR=$PWD/kokkos-kernels

# Grab a node
bsub -Is -n 1 -q rhel8 -gpu "num=4" bash

# Env and Configure
source /projects/ppc64le-pwr9-rhel8/legacy-env.sh
module purge
module load git  cmake/3.23.1 cuda/11.2.2/gcc/8.3.1 openmpi/4.1.1/gcc/8.3.1/cuda/11.2.2 openblas/0.3.18/gcc/8.3.1
module load metis/5.1.0/gcc/8.3.1 hdf5/1.10.7/gcc/8.3.1/openmpi/4.1.1 parmetis/4.0.3/gcc/8.3.1/openmpi/4.1.1 zlib/1.2.11/gcc/8.3.1 boost/1.70.0/gcc/8.3.1
export OMPI_CXX=$KOKKOS_DIR/bin/nvcc_wrapper

cmake \
 -D CMAKE_INSTALL_PREFIX=${TRILINOS_INSTALL_DIR} \
 -D CMAKE_CXX_COMPILER="`which mpicxx`" \
 -D CMAKE_CXX_FLAGS="-g" \
 -D CMAKE_C_COMPILER="`which mpicc`" \
 -D CMAKE_C_FLAGS="-g" \
 -D CMAKE_Fortran_COMPILER="`which mpifort`" \
 -D CMAKE_Fortran_FLAGS="-g" \
 -D CMAKE_BUILD_TYPE:STRING=RELEASE \
 -D CMAKE_CXX_STANDARD=17 \
 -D CMAKE_INSTALL_PREFIX=$PWD/install \
 -D TPL_ENABLE_CUDA:STRING=ON \
 -D TPL_ENABLE_MPI:STRING=ON \
 -D MPI_BASE_DIR:PATH="$OPENMPI_ROOT" \
 -D MPI_BIN_DIR:PATH="$OPENMPI_BIN" \
 -D MPI_EXEC_POST_NUMPROCS_FLAGS:STRING="-map-by;socket:PE=4" \
 -D TPL_ENABLE_BLAS:STRING=ON \
  -D BLAS_LIBRARY_DIRS:FILEPATH="$OPENBLAS_ROOT/lib" \
  -D BLAS_LIBRARY_NAMES:STRING="openblas" \
 -D TPL_ENABLE_LAPACK:STRING=ON \
  -D LAPACK_INCLUDE_DIRS:FILEPATH="$OPENBLAS_ROOT/include" \
  -D LAPACK_LIBRARY_DIRS:FILEPATH="$OPENBLAS_ROOT/lib" \
  -D LAPACK_LIBRARY_NAMES:STRING="openblas" \
 -D TPL_ENABLE_Boost:BOOL=ON \
  -D Boost_INCLUDE_DIRS:FILEPATH="$BOOST_ROOT/include" \
  -D Boost_LIBRARY_DIRS:FILEPATH="$BOOST_ROOT/lib" \
 -D TPL_ENABLE_BoostLib:BOOL=ON \
  -D BoostLib_INCLUDE_DIRS:FILEPATH="$BOOST_ROOT/include" \
  -D BoostLib_LIBRARY_DIRS:FILEPATH="$BOOST_ROOT/lib" \
 -D TPL_ENABLE_Netcdf:BOOL=OFF \
 -D Trilinos_ENABLE_TESTS=OFF \
 -D Trilinos_ENABLE_EXAMPLES=OFF \
 -D Trilinos_ENABLE_Kokkos=ON \
 -D Kokkos_ENABLE_TESTS=ON \
  -D Kokkos_ENABLE_CUDA=ON \
  -D Kokkos_ENABLE_CUDA_LAMBDA=ON \
  -D Kokkos_ENABLE_CUDA_UVM=ON \
  -D Kokkos_ARCH_VOLTA70=ON \
  -D Kokkos_ARCH_POWER9=ON \
 -D Trilinos_ENABLE_KokkosKernels=ON \
 -D KokkosKernels_ENABLE_TESTS=ON \
  -D KokkosKernels_INST_MEMSPACE_CUDAUVMSPACE=ON \
  -D KokkosKernels_ENABLE_TPL_CUSOLVER=OFF \
 -D Trilinos_ENABLE_Tpetra=ON \
 -D Tpetra_ENABLE_TESTS=ON \
  -D Tpetra_ALLOCATE_IN_SHARED_SPACE=ON \
  -D Tpetra_ENABLE_CUDA=ON \
  -D Tpetra_ENABLE_SERIAL=ON \
 -D Trilinos_ENABLE_Sacado=ON \
 -D Sacado_ENABLE_TESTS=ON \
 -D Trilinos_ENABLE_Amesos2=ON \
 -D Amesos2_ENABLE_TESTS=ON \
 -D Trilinos_ENABLE_Teuchos=ON \
 -D Teuchos_ENABLE_TESTS=ON \
 -D Trilinos_ENABLE_Ifpack2=ON \
 -D Ifpack2_ENABLE_TESTS=ON \
 -D Trilinos_ENABLE_Belos=ON \
 -D Belos_ENABLE_TESTS=ON \
 -D Trilinos_ENABLE_Stokhos=ON \
 -D Stokhos_ENABLE_TESTS=ON \
 -D Trilinos_ENABLE_Phalanx=ON \
 -D Phalanx_ENABLE_TESTS=ON \
 -D Trilinos_ENABLE_Zoltan2=ON \
 -D Zoltan2_ENABLE_TESTS=ON \
 -D Trilinos_ENABLE_Anasazi=ON \
 -D Trilinos_ENABLE_MueLu=ON \
 -D MueLu_ENABLE_TESTS=ON \
 -D Trilinos_ENABLE_Panzer=ON \
 -D Panzer_ENABLE_TESTS=ON \
 -D Trilinos_ENABLE_Intrepid2=ON \
 -D Intrepid2_ENABLE_TESTS=ON \
 -D Trilinos_ENABLE_SEACAS=OFF \
 -D Trilinos_ENABLE_Zoltan2Sphynx=OFF \
 -D Trilinos_ENABLE_ShyLU_NodeTacho=OFF \
 -D Stokhos_KokkosCrsMatrixUQPCEUnitTest_Cuda_MPI_1_SET_RUN_SERIAL=ON \
 -D Stokhos_TpetraCrsMatrixUQPCEUnitTest_Cuda_MPI_4_SET_RUN_SERIAL=ON \
 -D Stokhos_KokkosCrsMatrixMPVectorUnitTest_Cuda_MPI_1_SET_RUN_SERIAL=ON \
 -D Intrepid2_unit-test_MonolithicExecutable_Intrepid2_Tests_MPI_1_SET_RUN_SERIAL=ON \
 -D Kokkos_CoreUnitTest_CudaTimingBased_MPI_1_SET_RUN_SERIAL=ON \
 -D Kokkos_CoreUnitTest_Default_MPI_1_SET_RUN_SERIAL=ON \
  -DKokkos_SOURCE_DIR_OVERRIDE:STRING=kokkos \
  -DKokkosKernels_SOURCE_DIR_OVERRIDE:STRING=kokkos-kernels \
  -DTrilinos_ENABLE_INSTALLATION_TESTING=OFF \
$TRILINOS_DIR

# Build - should only need to build the stokhos unit tests to reproduce
make -j16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jgfouca jgfouca

Labels
bug develop-only bug InDevelop
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jgfouca @ndellingwood