Tpetra: Add nonblocking MPI-global GEMV

[mhoemmen]

@trilinos/tpetra

A collaborator would like to implement Pipelined GMRES (see #748 for comparison). This calls for a nonblocking MPI-global GEMV in order to implement the classical Gram-Schmidt projection step. The implementation of Tpetra::idot (see #1013) currently supports this use case, but does not optimize the intraprocess dense matrix-vector product (GEMV). There are two issues:

1. Tpetra::idot does not call KokkosBlas::gemv for that case. Instead, it rolls its own kernel. This is likely to be slow for multivectors with many columns, and it may even run out of scratch memory on the GPU (which uses scratch for intermediate reduction results).
2. ( KokkosBlas: Add GEMV that wraps BLAS (and cuBLAS) kokkos/kokkos-kernels#16 ) KokkosBlas::gemv does not call the BLAS or cuBLAS (if appropriate). Instead, it rolls its own kernel, which has the same issues as the kernel that Tpetra::idot uses.

[mhoemmen]

Note that this GEMV might be a little different than a BLAS GEMV, if Tpetra::MultiVector::dot_type != Tpetra::MultiVector::impl_scalar_type (in the Stokhos case, for example). We're really computing a bunch of dot products at once. This means that the intermediate terms in the sum have type dot_type, not impl_scalar_type.

This may be relevant to @etphipp 's interests, though please don't feel obligated to comment; I just want to make sure you know what's going on.

[mhoemmen]

My recent commit db0e10c adds a process-local GEMV wrapper. I still need to use this in dot products. Note that Tpetra::idot actually implements what we want, but it might not be efficient on GPUs yet; it should call the new GEMV wrappers instead of dispatching to a custom kernel in that case.

[github-actions]

This issue has had no activity for 365 days and is marked for closure. It will be closed after an additional 30 days of inactivity.
If you would like to keep this issue open please add a comment and remove the MARKED_FOR_CLOSURE label.
If this issue should be kept open even with no activity beyond the time limits you can add the label DO_NOT_AUTOCLOSE.

[github-actions]

This issue was closed due to inactivity for 395 days.