Failure with Tpetra vectors: nonlocal_vector is not available.

[bangerth]

One of the failures I see with #16611 is that we run into trouble with the Tpetra vectors because nonlocal_vector is a nullptr. Specifically in this function:

   template <typename Number, typename MemorySpace>
    inline void
    Vector<Number, MemorySpace>::add(const size_type  n_elements,
                                     const size_type *indices,
                                     const Number    *values)
    {
#  if DEAL_II_TRILINOS_VERSION_GTE(13, 2, 0)
      auto vector_2d_local = vector->template getLocalView<Kokkos::HostSpace>(
        Tpetra::Access::ReadWrite);
      auto vector_2d_nonlocal =
        nonlocal_vector->template getLocalView<Kokkos::HostSpace>(
          Tpetra::Access::ReadWrite);
#  else
      vector->template sync<Kokkos::HostSpace>();
      auto vector_2d_local = vector->template getLocalView<Kokkos::HostSpace>();
      auto vector_2d_nonlocal =
        nonlocal_vector->template getLocalView<Kokkos::HostSpace>();
#  endif

      auto vector_1d_local = Kokkos::subview(vector_2d_local, Kokkos::ALL(), 0);
      auto vector_1d_nonlocal =
        Kokkos::subview(vector_2d_nonlocal, Kokkos::ALL(), 0);
#  if !DEAL_II_TRILINOS_VERSION_GTE(13, 2, 0)
      vector->template modify<Kokkos::HostSpace>();
      nonlocal_vector->template modify<Kokkos::HostSpace>();
#  endif

      for (size_type i = 0; i < n_elements; ++i)
        {
          const size_type                   row = indices[i];
          TrilinosWrappers::types::int_type local_row =
            vector->getMap()->getLocalElement(row);

          // check if the index is in the non local set
          bool nonlocal = false;
          if (local_row == Teuchos::OrdinalTraits<int>::invalid())
            {
              local_row  = nonlocal_vector->getMap()->getLocalElement(row);
              nonlocal   = true;
              compressed = false;
            }

#  if DEAL_II_TRILINOS_VERSION_GTE(14, 0, 0)
          Assert(
            local_row != Teuchos::OrdinalTraits<int>::invalid(),
            ExcAccessToNonLocalElement(row,
                                       vector->getMap()->getLocalNumElements(),
                                       vector->getMap()->getMinLocalIndex(),
                                       vector->getMap()->getMaxLocalIndex()));
#  else
          Assert(
            local_row != Teuchos::OrdinalTraits<int>::invalid(),
            ExcAccessToNonLocalElement(row,
                                       vector->getMap()->getNodeNumElements(),
                                       vector->getMap()->getMinLocalIndex(),
                                       vector->getMap()->getMaxLocalIndex()));

#  endif

          if (local_row != Teuchos::OrdinalTraits<int>::invalid())
            {
              if (nonlocal)
                vector_1d_nonlocal(local_row) += values[i];
              else
                vector_1d_local(local_row) += values[i];
            }
        }

#  if !DEAL_II_TRILINOS_VERSION_GTE(13, 2, 0)
      vector->template sync<
        typename Tpetra::Vector<Number, int, types::signed_global_dof_index>::
          device_type::memory_space>();
      nonlocal_vector->template sync<
        typename Tpetra::Vector<Number, int, types::signed_global_dof_index>::
          device_type::memory_space>();
#  endif
    }

If nonlocal_vector == nullptr, extracting the view at the very top already fails.

@kinnewig @jpthiele @masterleinad FYI.

[bangerth]

For reference, we create these vectors as follows:

        LinearAlgebra::TpetraWrappers::Vector<double> v;
        v.reinit(complete_index_set(100), MPI_COMM_WORLD);

So the vector is in essence sequential and owns all elements. We end up with no nonlocal_vector because of this implementation:

    template <typename Number, typename MemorySpace>
    void
    Vector<Number, MemorySpace>::reinit(const IndexSet &parallel_partitioner,
                                        const MPI_Comm  communicator,
                                        const bool /*omit_zeroing_entries*/)
    {
      vector.reset();
      nonlocal_vector.reset();

      compressed = true;
      has_ghost  = false;
      vector     = Utilities::Trilinos::internal::make_rcp<VectorType>(
        parallel_partitioner.make_tpetra_map_rcp(communicator, true));
    }

[jpthiele]

If it is sequential and therefore has no ghost elements wouldn't it make sense to catch that with an if,
such that the nonlocal view is only created if has_ghost==true?

[bangerth]

Yes. I'm working on a patch. In fact, we only need to create the view if we actually write into any of these entries.