Handling of auxiliary vectors in matrix-free loops

[peterrum]

I am a bit unhappy to see code like this:

aux_vector.update_ghost_values();

matrix_free.template cell_loop<VectorType, VectorType>(
  [&](const auto &, auto & , const auto &, auto macro_cells) {
    for (unsigned int cell = macro_cells.first; cell < macro_cells.second; ++cell)
      fe_evalution.read_dof_values_plain(aux_vector);
  }, dst, src);

aux_vector.zero_out_ghosts();

where an auxiliary vector is used within a matrix-free loop (note that aux_vector might be used before/later as the dst/src vector of a matrix-free loop). In particular, this prevents the usage of asynchronous communication and the new shmem features.
I would like to propose something like this:

matrix_free.attach_vector_for_ghost_values_update(aux_vector, flags);

matrix_free.template cell_loop<VectorType, VectorType>(
  [&](const auto &, auto & , const auto &, auto macro_cells) {
    for (unsigned int cell = macro_cells.first; cell < macro_cells.second; ++cell)
      fe_evalution.read_dof_values_plain(aux_vector);
  }, dst, src);

where the ghost-value-update is done by MatrixFree in the same way as for src.

We still need to think about the case that we want to "update the ghost values" not before/after each matrix-free loop, but only once a time step. Any ideas here?

FYI @mschreter

[peterrum]

Let's close this one. It is hard o handle auxiliary vectors, since we don't want to update/zero out vectors too often (e.g. in an iterative solver the values might not change).